A breast tumor image feature fusion method, system and computer storage medium

Abstract

The application discloses a breast tumor image feature fusion method and system and a computer storage medium, and the method comprises the following steps: acquiring DCE-MRI and DWI sequence images of a breast tumor; performing missing detection on the DCE-MRI and DWI sequence images to form an MRI data set; determining a mapping relationship between the DCE-MRI and DWI double sequence images in the MRI data set; generating a missing DCE-MRI or DWI image based on the mapping relationship, and adding the generated DCE-MRI or DWI image to the MRI data set; extracting features of the DCE-MRI and DWI double sequence images in the MRI data set; and fusing the features of the DCE-MRI and DWI double sequence images. The application uses the mapping relationship between the MRI multiple sequences to generate missing features, thereby reducing the influence of sequence missing on the downstream network and improving the performance of breast tumor image fusion.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical image processing, in particular to a breast tumor image feature fusion method and system and a computer storage medium. BACKGROUND

[0002] Breast MRI is a commonly used imaging method in breast disease screening, among which the roles of dynamic contrast-enhanced magnetic resonance imaging (DCE-MRI) and diffusion weighted imaging (DWI) sequences are particularly obvious. DCE-MRI sequence can obtain the hemodynamic characteristics of the lesion by dynamic information of the contrast agent in the capillary and interstitial space, DWI is an imaging method that relies on the irregular Brownian motion of water molecules, which can non-invasively detect the diffusion motion of water molecules in living tissues and quantitatively detect the degree of water molecule diffusion restriction. The complementary information obtained by the joint analysis of DCE-MRI and DWI provides more information about the microstructure and vascular distribution of tumors.

[0003] At present, researchers have proposed various breast tumor image fusion methods based on advanced image processing, pattern recognition and artificial intelligence technologies. Compared with the complexity and tediousness of the feature design and extraction process in traditional machine learning methods, deep learning technology based on data-driven can automatically learn features from raw data. This kind of method has achieved better performance in the task of breast tumor image fusion, but most deep network models are based on a single MRI sequence, ignoring the rich information contained in multi-sequence MRI.

[0004] The transverse combination and complementary advantages among MRI sequences can often improve the sensitivity and specificity of benign and malignant lesion diagnosis. The fusion technology among MRI sequences can be divided into pixel-level fusion, feature set fusion and decision-level fusion according to different information representation levels from low to high. Based on a large number of previous studies, researchers found that pixel-level fusion excessively depends on registration, and decision-level fusion does not consider the potential relationship between features, so it is generally believed that feature-level fusion is the most effective among the three fusion methods. Through the feature fusion of multi-sequence MRI, the performance of the network for image fusion can be improved. The current mainstream fusion method such as splicing is relatively simple, and there is still a lot of room for performance improvement. In addition, most experiments are based on complete feature sequence combination. However, due to the limitations of factors such as scan damage, artifacts, incorrect machine settings, patient allergy to certain contrast agents and limited scan time, the sequence information of MRI may be unavailable or even missing. The current mainstream deep learning network often has the problem of relying on the effectiveness of the input sequence, and the missing sequence information will cause the failure of the multi-sequence image fusion task. Therefore, the processing of the sequence missing phenomenon and the deep fusion of multi-sequence are the keys to improving the multi-sequence MRI breast tumor image fusion. SUMMARY

[0005] The breast tumor image feature fusion method, system and computer storage medium provided by the embodiments of the present application can solve the problem that the fusion task fails when the sequence information of MRI in the prior art is unavailable or missing.

[0006] In one aspect, the breast tumor image feature fusion method provided by the embodiments of the present application comprises:

[0007] obtaining breast tumor DCE-MRI and DWI sequence images;

[0008] detecting the missing of the breast tumor DCE-MRI and DWI sequence images to form an MRI data set, wherein the MRI data set comprises paired breast tumor DCE-MRI and DWI dual sequence images and single sequence images with missing;

[0009] determining the mapping relationship between the paired breast tumor DCE-MRI and DWI dual sequence images in the MRI data set;

[0010] generating the missing breast tumor DCE-MRI or DWI image based on the mapping relationship, and adding the generated breast tumor DCE-MRI or DWI image to the MRI data set;

[0011] extracting the features of the breast tumor DCE-MRI and DWI dual sequence images in the MRI data set;

[0012] fusing the features of the breast tumor DCE-MRI and DWI dual sequence images.

[0013] In another aspect, the embodiment of the present application provides a breast tumor image feature fusion system, comprising:

[0014] An image acquisition module is configured to acquire breast tumor DCE-MRI and DWI sequence images.

[0015] A missing detection module is configured to perform missing detection on the breast tumor DCE-MRI and DWI sequence images to form an MRI data set, wherein the MRI data set comprises paired breast tumor DCE-MRI and DWI dual sequence images and single sequence images with missing data.

[0016] A mapping determination module is configured to determine a mapping relationship between the paired breast tumor DCE-MRI and DWI dual sequence images in the MRI data set.

[0017] An image generation module is configured to generate missing breast tumor DCE-MRI or DWI images based on the mapping relationship and add the generated breast tumor DCE-MRI or DWI images to the MRI data set.

[0018] A feature extraction module is configured to extract features of the breast tumor DCE-MRI and DWI dual sequence images in the MRI data set.

[0019] A feature fusion module is configured to fuse the features of the breast tumor DCE-MRI and DWI dual sequence images.

[0020] In another aspect, the embodiment of the present application provides a computer storage medium, which stores a plurality of computer instructions for causing a computer to execute the above method.

[0021] The breast tumor image feature fusion method, system and computer storage medium provided in the present application have the following advantages:

[0022] 1. The present application makes full use of the rich image information in the MRI multi-sequence, and based on the fact that there is a strong correlation between the MRI sequences of the same patient, the mapping relationship between the MRI multi-sequences is used to generate the missing features. In the feature generation process, the method of the present application can better model the correlation between the DCE-MRI and DWI sequence features, generate the missing features, thereby reducing the influence of sequence missing on the downstream network and improving the performance of breast tumor image fusion.

[0023] 2. Input the sequence with a feature extraction block that has spatial attention and channel attention. This part has the same structure as the feature extraction part of the generator and shares parameters. It makes full use of the feature relationships learned in the cross-sequence feature transformation process. Since feature fusion and feature extraction at different levels are performed in parallel, the representativeness of the extracted features is fully guaranteed.

[0024] 3. Taking full advantage of MRI multi-sequence imaging, an extended sequence fusion block (ESFB) was designed to adaptively weight and fuse features of different sequences at different levels, effectively utilizing the complex relationships between sequence features to improve the network's performance in fusing breast tumor images. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 A flowchart of a breast tumor image feature fusion method provided in an embodiment of the present invention;

[0027] Figure 2 The DCE-MRI and DWI images of breast tumors provided in the embodiments of the present invention;

[0028] Figure 3 This is a schematic diagram illustrating the image processing by the generator and discriminator provided in an embodiment of the present invention;

[0029] Figure 4 This is a schematic diagram of the image processing flow provided by the generator in an embodiment of the present invention;

[0030] Figure 5 This is an overall flowchart of a breast tumor image feature fusion method provided in an embodiment of the present invention;

[0031] Figure 6 This is a schematic diagram of the processing flow of the ESFB module provided in an embodiment of the present invention;

[0032] Figure 7 This diagram illustrates a performance comparison between the method of the present invention and existing methods, as provided in an embodiment of the present invention. Detailed Implementation

[0033] With reference to the accompanying drawings, the technical solutions in the embodiments of the present application will be described clearly and completely. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative efforts belong to the scope of the present application.

[0034] Before introducing the technical solutions of the present application, the related technologies are first described.

[0035] DCE-MRI: dynamic contrast-enhanced magnetic resonance imaging. DCE-MRI imaging mainly relies on the microcirculation change of lesions and tissues. Through continuous and rapid imaging sequences, a series of continuous dynamic enhancement process images reflecting the tissue enhancement in each period before, during and after the injection of contrast agent are obtained. Through corresponding computer software processing and analysis of the obtained image information, various parameters reflecting the microcirculation function of the tissue can be obtained.

[0036] DWI: diffusion weighted imaging. DWI is an imaging method obtained by means of irregular Brownian motion of water molecules. It can non-invasively detect the diffusion motion of water molecules in living tissues and quantitatively detect the degree of diffusion restriction of water molecules.

[0037] Figure 1 A flowchart of a breast tumor image feature fusion method provided by an embodiment of the present application. The breast tumor image feature fusion method provided by the embodiment of the present application comprises:

[0038] S100, acquiring breast tumor DCE-MRI and DWI sequence images.

[0039] Exemplarily, the acquired image data contains MRI images from 98 high-risk female patients with breast cancer. They simultaneously underwent DCE-MRI and DWI sequence image acquisition and the image data were all effective. Part of the images are as shown in Figure 2

[0040] S110, performing missing detection on the breast tumor DCE-MRI and DWI sequence images to form an MRI data set. The MRI data set contains paired breast tumor DCE-MRI and DWI dual sequence images and single sequence images with missing data.

[0041] S120, determining the mapping relationship between the paired breast tumor DCE-MRI and DWI dual sequence images in the MRI data set.

[0042] Exemplarily, a generative adversarial network can be used to learn the mapping relationship between the paired breast tumor DCE-MRI and DWI dual sequence images in the MRI data set. ​

[0043] The S120 specifically comprises: S121, inputting the paired breast tumor DCE-MRI and DWI dual sequence images in the MRI data set into a generator, the generator converts the breast tumor DCE-MRI and DWI dual sequence images into corresponding DWI sequence images and DCE-MRI sequence images respectively; S122, judging whether the sequence images output by the generator are original images by using a discriminator, and promoting the training of the generator according to the judgment result; S123, after the generator is well trained, training the discriminator, and iterating the generator and the discriminator in the training process until the generative adversarial network converges, and the trained generative adversarial network contains the mapping relationship of the two types of images.

[0044] In S121, the generative adversarial network mainly designed based on CycleGAN network, wherein, the generator G dce and G dwi are formed on the basis of ResNet by fusing channel attention and spatial attention. G dce represents the DCE generator, which can convert the input DWI sequence image into a DCE-MRI sequence image, G dwi represents the DWI generator, which can convert the input DCE-MRI sequence image into a DWI sequence image, as shown in Figure 3 . The specific operation of the generator is that the sequence image is input into four feature extraction layers in the generator, and each feature extraction layer is composed of two feature extraction blocks, a total of eight feature extraction blocks. The feature extraction blocks use channel attention mechanism and spatial attention mechanism to extract the features of the breast tumor DCE-MRI and DWI dual sequence images. The channel attention mechanism and the spatial attention mechanism both make the network pay attention to the region of interest by using global maximum pooling and average pooling, the difference is that the spatial attention mechanism is facing the above operation of width and height, while the channel attention is facing the channel. After the feature extraction block outputs the features, the converted sequence image is obtained by upsampling the block, as shown in Figure 4 .

[0045] In S122, the discriminator includes a plurality of discriminator feature extraction layers, each discriminator feature extraction layer includes a 2D convolution block, a batch normalization block and a ReLu activation function block. Among them, the discriminators D dce and D dwi each include four discriminator feature extraction layers, and the convolution kernel size in the 2D convolution block is 4.

[0046] In S123, the training of the generative adversarial network can be supervised by the adversarial loss and the cycle consistency loss. In the process of adversarial learning between the generator and the discriminator, the generator wants to generate samples as realistic as possible and better learn the mapping relationship between the sequence images, while the discriminator wants to accurately distinguish the type of input sequence image as much as possible.

[0047] In an embodiment of the present application, the adversarial loss function is defined as:

[0048] L a (G dwi ,D dwi )=E dwi [D dwi (DWI) 2 ]+E dce [(1-D dwi (G dwi (DCE)) 2 )]

[0049] L a (G dce ,D dce )=E dce [D dce (DCE) 2 ]+E dwi [(1-D dce (G dce (DWI)) 2 )]

[0050] Wherein, E represents the expected value of the corresponding sample. The cycle consistency loss is used to supervise the similarity between G dce G dwi (DCE) and DCE, G dwi G dce (DWI) and DWI, and is defined as:

[0051] L cyc =E dce [||G dce G dwi (DCE)||]+E dwi [||G dwi G dce (DWI)||]

[0052] The overall loss L of network training is defined as:

[0053]

[0054] Wherein, λ represents the weight of the two types of loss, used to balance the adversarial loss and the cycle consistency loss.

[0055] S130, generating the missing breast tumor DCE-MRI or DWI image based on the mapping relationship, and adding the generated breast tumor DCE-MRI or DWI image to the MRI data set.

[0056] Illustratively, since the trained generative adversarial network contains the mapping relationship, the generator G dce and G dwi The missing sequence image can be predicted to generate the missing sequence image.

[0057] S140, extracting the features of the breast tumor DCE-MRI and DWI dual sequence images in the MRI data set.

[0058] Illustratively, four feature extraction layers connected in sequence can be used to extract the features of the breast tumor DCE-MRI and DWI dual sequence images.

[0059] S150, fusing the features of the breast tumor DCE-MRI and DWI dual sequence images.

[0060] Illustratively, S150 specifically includes: S151, inputting the features of the breast tumor DCE-MRI and DWI dual sequence images extracted by the first feature extraction layer into a first SFB (sequence fusion block) module; S152, inputting the features of the breast tumor DCE-MRI and DWI dual sequence images extracted by the second feature extraction layer and the fusion result of the first SFB module into a first SFB_extend (SFB with added atrous convolution) module; S153, inputting the features of the breast tumor DCE-MRI and DWI dual sequence images extracted by the third feature extraction layer and the fusion result of the first SFB_extend module into a second SFB module; S154, inputting the features of the breast tumor DCE-MRI and DWI dual sequence images extracted by the fourth feature extraction layer and the fusion result of the second SFB module into a second SFB_extend module, and the second SFB_extend module outputs the fused features of the breast tumor DCE-MRI and DWI dual sequence images. As shown in Figure 5 .

[0061] In the fusion process of each SFB module and SFB_extend module, first, the two input sequence images are subjected to element weighted average processing f1 and element multiplication processing f2, the former operation can increase the signal-to-noise ratio of the sequence image, and the latter can improve the spatial resolution of the image while preserving the features of the sequence image. Then, the two input sequence images are subjected to maximum value processing f3 after being merged by channel, so that the gray scale of the fused pixels is enhanced. Finally, the three are weighted and fused, and the weights are λ1, λ2, and λ3, respectively. The output of the fusion is the output SFBn-1 The output of the nth SFB module or SFB_extend module is obtained by inputting the convolution layer together. n As shown in FIG. 1. Figure 6 It is noted that when n = 1, the input of the previous sequence image does not exist, and in this case, the output result of the convolution is directly used for subsequent processing.

[0062] The output of the final ESFB module is defined as:

[0063]

[0064] Wherein F n -1 represents the fusion operation through the SFB_extend module, F n-1 represents the fusion operation through the SFB module, n represents the number of feature extraction layers, represents the output obtained after the sequence image i passes through the nth-1 feature extraction layer.

[0065] In order to verify the effectiveness of the method of the present application, the present application selects MRI data sets from 98 patients for experimental verification:

[0066] In the experiment, the effectiveness of the missing feature information generation algorithm is verified using only DCE-MRI sequence images, only DWI sequence images and data combined with both. In order to increase the persuasiveness, under the premise of ensuring the same network calculation amount, the present application also verifies on the basis of VGG, ResNet, DenseNet and other networks.

[0067] As shown in Table 1, the method provided by the present application is compared with other networks for breast tumor image fusion. Through comparative analysis, it can be obtained that when a single DCE-MRI sequence is used for training on different networks, the network has high accuracy, and when a single DWI sequence is used for training, the network has high sensitivity, and when the two are combined, the network has improved indicators. After generating the missing features, the network performance is improved compared with using only a single sequence for training. It can be seen that generating missing features is effective and necessary when different networks are used for breast tumor image fusion. The data in Table 1 shows that the method provided by the present application has achieved relatively good results compared with other methods.

[0068] Figure 7 The fusion results of the method provided by the present application and other different fusion methods are shown, and the compared fusion methods include channel fusion, wavelet fusion and pixel fusion. The present application is performed in a multi-sequence data set containing missing features. As can be seen from the figure, different fusion methods perform differently on different networks. In any network, the network performance of the method provided by the present application is always optimal.

[0069] As shown in Table 2, the classification performance of the method provided by the application and other popular methods is compared. The application uses the two-dimensional slice of the fourth phase in the DCE-MRI sequence to represent the tumor lesion characteristic information of the DCE-MRI sequence reaction, and the characteristic supplement mode adopts the generative network generation. In the training process, the same data enhancement method is adopted, and four-fold cross-validation is used to comprehensively evaluate the network performance. As can be seen from the data in Table 2, the accuracy, precision, recall and F1-score of the application at the patient level are 89.66%, 90.11%, 95.40% and 92.56% respectively, which are better than those of other comparison methods. The application can effectively fuse different sequence characteristics, and well improve the feature learning ability and classification performance of the model.

[0070] Table 1 Comparison results of the method provided by the application and other networks for breast tumor image fusion

[0071]

[0072] Table 2 Comparison of classification performance of the method provided by the application and other popular methods

[0073]

[0074] The embodiment of the application also provides a breast tumor image feature fusion system, comprising:

[0075] An image acquisition module is configured to acquire breast tumor DCE-MRI and DWI sequence images.

[0076] A missing detection module is configured to perform missing detection on the breast tumor DCE-MRI and DWI sequence images to form an MRI data set, wherein the MRI data set includes paired breast tumor DCE-MRI and DWI dual sequence images and single sequence images with missing data.

[0077] A mapping determination module is configured to determine a mapping relationship between the paired breast tumor DCE-MRI and DWI dual sequence images in the MRI data set.

[0078] An image generation module is configured to generate missing breast tumor DCE-MRI or DWI images based on the mapping relationship, and add the generated breast tumor DCE-MRI or DWI images to the MRI data set.

[0079] A feature extraction module is configured to extract features of the breast tumor DCE-MRI and DWI dual sequence images in the MRI data set.

[0080] A feature fusion module is configured to fuse the features of the breast tumor DCE-MRI and DWI dual sequence images.

[0081] The embodiment of the present application further provides a computer storage medium, which stores a plurality of computer instructions for causing a computer to execute the method.

[0082] Although preferred embodiments of the application have been described herein, additional changes and modifications can be suggested to one skilled in the art, particularly in light of the essential novel teachings herein. The disclosures herein are intended to embrace all such changes and modifications. The phrase "patent application" should be understood to include both provisional and non-provisional patent applications.

[0083] Obviously, numerous modifications and variations of the present application are possible in light of the above teachings. It is therefore to be understood that within the scope of the appended claims and their equivalents, the application can be practiced otherwise than as specifically described.

Claims

1. A breast tumor image feature fusion method, characterized in that, The method comprises the following steps: acquiring breast tumor DCE-MRI and DWI sequence images; performing missing detection on the breast tumor DCE-MRI and DWI sequence images to form an MRI data set, wherein the MRI data set comprises paired breast tumor DCE-MRI and DWI dual sequence images and single sequence images with missing data; determining a mapping relationship between the paired breast tumor DCE-MRI and DWI dual sequence images in the MRI data set; generating a missing breast tumor DCE-MRI or DWI image based on the mapping relationship, and adding the generated breast tumor DCE-MRI or DWI image to the MRI data set; extracting features of the breast tumor DCE-MRI and DWI dual sequence images in the MRI data set; fusing the features of the breast tumor DCE-MRI and DWI dual sequence images; wherein the step of extracting features of the breast tumor DCE-MRI and DWI dual sequence images in the MRI data set comprises: extracting features of the breast tumor DCE-MRI and DWI dual sequence images by using four feature extraction layers connected in sequence; the step of fusing the features of the breast tumor DCE-MRI and DWI dual sequence images comprises: inputting the features of the breast tumor DCE-MRI and DWI dual sequence images extracted by the first feature extraction layer into a first SFB module; inputting the features of the breast tumor DCE-MRI and DWI dual sequence images extracted by the second feature extraction layer and the fusion result of the first SFB module into a first SFB_extend module; inputting the features of the breast tumor DCE-MRI and DWI dual sequence images extracted by the third feature extraction layer and the fusion result of the first SFB_extend module into a second SFB module; inputting the features of the breast tumor DCE-MRI and DWI dual sequence images extracted by the fourth feature extraction layer and the fusion result of the second SFB module into a second SFB_extend module, and the second SFB_extend module outputs the fusion features of the breast tumor DCE-MRI and DWI dual sequence images.

2. The breast tumor image feature fusion method of claim 1, wherein, the step of determining a mapping relationship between the paired breast tumor DCE-MRI and DWI dual sequence images in the MRI data set comprises: learning the mapping relationship between the paired breast tumor DCE-MRI and DWI dual sequence images in the MRI data set by using a generative adversarial network.

3. The breast tumor image feature fusion method of claim 2, wherein, the step of learning the mapping relationship between the paired breast tumor DCE-MRI and DWI dual sequence images in the MRI data set by using a generative adversarial network comprises: inputting the paired breast tumor DCE-MRI and DWI dual sequence images in the MRI data set into a generator, and the generator converts the breast tumor DCE-MRI and DWI dual sequence images into corresponding DWI sequence images and DCE-MRI sequence images, respectively; The discriminator is used to determine whether the sequence image output by the generator is the original image, and the training of the generator is promoted according to the determination result; After the generator is trained, the discriminator is trained, and the generator and the discriminator are iterated in the training process until the generative adversarial network converges, and the trained generative adversarial network comprises the mapping relationship.

4. The breast tumor image feature fusion method of claim 3, wherein, The generator comprises a feature extraction block and an up-sampling block, the feature extraction block extracts features of the breast tumor DCE-MRI and DWI double sequence images by using a channel attention mechanism and a spatial attention mechanism, and the up-sampling block up-samples the extracted features to obtain corresponding DWI sequence images and DCE-MRI sequence images.

5. The breast tumor image feature fusion method of claim 3, wherein, The discriminator comprises a plurality of discriminator feature extraction layers, each of which comprises a 2D convolution block, a batch normalization block and a ReLu activation function block.

6. The breast tumor image feature fusion method of claim 3, wherein, When the generative adversarial network is trained, the training of the generative adversarial network is supervised by using an adversarial loss and a cycle consistency loss.

7. A system for applying the breast tumor image feature fusion method according to any one of claims 1 to 6, characterized in that, It comprises: An image acquisition module is configured to acquire breast tumor DCE-MRI and DWI sequence images; A missing detection module is configured to detect missing of the breast tumor DCE-MRI and DWI sequence images to form an MRI data set, wherein the MRI data set comprises paired breast tumor DCE-MRI and DWI double sequence images and single sequence images with missing; A mapping determination module is configured to determine a mapping relationship between the paired breast tumor DCE-MRI and DWI double sequence images in the MRI data set; An image generation module is configured to generate missing breast tumor DCE-MRI or DWI images based on the mapping relationship, and add the generated breast tumor DCE-MRI or DWI images to the MRI data set; A feature extraction module is configured to extract features of the breast tumor DCE-MRI and DWI double sequence images in the MRI data set; A feature fusion module is configured to fuse the features of the breast tumor DCE-MRI and DWI double sequence images.

8. A computer storage medium, characterized in that, The computer storage medium stores a plurality of computer instructions, and the computer instructions are used to make the computer execute the method of any one of claims 1-6.

Images