Breast cancer tissue image classification method based on multi-scale feature fusion and attention mechanism

Abstract

The invention relates to a breast cancer tissue image classification method based on multi-scale feature fusion and an attention mechanism, and belongs to the technical field of medical image classification. According to the method, discriminative multi-scale features of images of different dimensions are extracted, and more comprehensive target feature representation is obtained. Based on this, the method combines an attention mechanism, adaptively adjusts the channel weight and the space weight of the target feature, strengthens the difference between the target feature and the redundant feature, and reduces the noise interference. And finally, the accuracy of an image classification result is effectively improved by aggregating and strengthening discriminative multi-scale features on which breast cancer tissue image classification depends.

Description

Technical Field

[0001] The present invention belongs to the technical field of medical image classification, and in particular relates to a breast cancer tissue image classification method based on multi-scale feature fusion and attention mechanism. Background Art

[0002] Medical image classification is an important means to achieve accurate classification of medical images and is also a core link in the field of smart medicine. For example, in the treatment of breast cancer, pathological image classification is a key link. By accurately classifying breast cancer tissue images, it is possible to identify different types and stages of cancer, help doctors assess the malignancy of the tumor, and thus develop targeted treatment plans and effectively monitor the progress and effects of treatment.

[0003] Convolutional neural network (CNN) has become an important tool for deep learning in the field of medical image classification due to its excellent feature extraction ability, and has promoted significant progress in this field. However, in the task of breast cancer tissue image classification, problems such as irregular cell shape, chaotic distribution of cell nuclei, and large differences in tissue morphology have brought many challenges to medical image classification based on convolutional neural network. At present, existing methods usually only focus on the extraction of image features, but ignore the adjustment of feature weights, and fail to effectively distinguish target features from redundant features. This limitation weakens the accuracy and reliability of feature extraction, thereby affecting the accuracy of breast cancer tissue image classification. Therefore, according to actual needs, designing and implementing a reliable, intelligent, accurate and efficient breast cancer tissue image classification algorithm can not only improve the efficiency of clinical diagnosis and reduce the deviation caused by pathologists' subjective judgment, but also provide an objective basis for the formulation of patient treatment and prognosis plans. Summary of the invention

[0004] The present invention designs and develops a breast cancer tissue image classification method based on multi-scale feature fusion and attention mechanism. By extracting multi-scale features of images of different dimensions, a more comprehensive feature representation is obtained. In addition, an integrated attention module is used to adaptively adjust the channel weight and spatial weight of the target feature, strengthen the difference between the target feature and the redundant feature, reduce noise interference, thereby improving the classification performance of the model, and overcoming the above challenges and deficiencies in the prior art.

[0005] The above-mentioned object of the present invention is achieved by the following technical solutions:

[0006] The present invention provides a breast cancer tissue image classification method based on multi-scale feature fusion and attention mechanism, comprising the following steps:

[0007] Step 1: Select a breast cancer tissue image dataset (such as BCI), divide the breast cancer tissue image dataset into a training set, a validation set, and a test set, and preprocess the image data to obtain the processed image X org , further for image X org Perform data enhancement (including color jitter, random rotation, etc.), the formula is as follows:

[0008] X=DA(X org )

[0009] Among them, X org is the image processed for breast cancer tissue, DA(·) represents the data augmentation strategy, and X is the image after data augmentation for breast cancer tissue;

[0010] Step 2: Sample breast cancer tissue data augmented images X and corresponding classification labels from the training set to form a batch, that is, batch = Cat(X, Y), where Cat is a tensor concatenation operation and Y is the category label of the image;

[0011] Step 3: Pre-convolute the batch data to reduce the spatial dimension of the data and obtain the feature map P 1 , that is, P 1 =pconv(X), where pconv(·) represents the pre-convolution operation;

[0012] Step 4: Use the channel attention module to adjust the feature map P 1 The channel weights are obtained by 2 , further, through the spatial attention module 2 The spatial weight is adjusted to obtain the final result of weight distribution P 3 The channel attention module and the spatial attention module together form an integrated attention module, which is expressed as follows:

[0013] P 3 =IAM(P 1 )

[0014] Among them, IAM(·) represents the integrated attention module;

[0015] Step 5: Extract multi-scale features of high-dimensional feature maps. First, use the multi-scale feature extraction module to process the high-dimensional feature map to obtain high-dimensional deep features E 5 , and then use the information preservation module to adjust the size of the high-dimensional deep features to obtain E' 5 , to ensure that the size of the feature map is consistent during the subsequent feature fusion process. This process can be expressed by the formula:

[0016] E 5 =MFEM(P 3 )

[0017] E' 5 =IPM(E 5 )

[0018] Among them, MFEM(·) represents the multi-scale feature extraction module, and IPM(·) represents the information preservation module;

[0019] Step 6: Extract multi-scale features of low-dimensional feature maps. First, use the information retention module to adjust the feature map dimension to obtain a low-dimensional feature map P' 3 ,Furthermore, the low-dimensional feature map is processed using the multi-scale feature extraction module to obtain the low-dimensional deep feature B 5 , which can be expressed in the form of:

[0020] P' 3 =IPM(P 3 )

[0021] B 5 =MFEM(P' 3 )

[0022] Where IPM(·) represents the information retention module, P' 3 represents the feature map after dimensionality reduction, MFEM(·) represents the multi-scale feature extraction module;

[0023] Step 7: Fusion of deep features of multi-dimensional images. 5 Multi-scale features E' with high-dimensional feature maps 5 The deep fusion feature R is obtained by fusion. This step can be expressed by the formula: R = Cat (E' 5 ,B 5 );

[0024] Step 8: Get the final classification result. First, use the integrated attention module to assign feature weights to the deep fusion feature R, and then output the final classification result R' through the linear layer Linear. The formula is as follows:

[0025] R'=Linear(IAM(R))

[0026] Step 9: Average the loss of all samples to get the overall cross entropy loss L total , the loss is defined as follows:

[0027]

[0028] Among them, L ce represents the cross entropy loss function, y i is the true category (label) of the i-th sample, C is the number of categories, and N is the number of samples;

[0029] Step 10, repeating steps 3 to 9 until the current number of iterations reaches a predefined maximum number of iterations;

[0030] Step 11: Send the breast cancer tissue image to be classified into the trained model for forward propagation to obtain the classification result.

[0031] The calculation formula of the integrated attention module described in step 4 is as follows:

[0032] ω 1 =σ(C1D k (GAP(P 1 )))

[0033] P 2 =P 1 ·ω 1

[0034] ω 2 =σ(f 7×7 ([F avg (P 2 );F max (P 2 )]))

[0035] P 3 =P 2 ·ω 2

[0036] Among them, GAP(·) represents global average pooling, C1D k (·) represents the one-dimensional convolution that maps the input feature map to a k-dimensional vector, σ(·) represents the Sigmoid function, and P 1 is the shallow semantic feature map, ω 1 is the channel attention weight, P 2 is the feature map after channel weight assignment, f 7×7 (·) represents a 7×7 convolution operation, F avg (·) represents the average pooling operation, F max (·) represents the maximum pooling operation, ω 2 is the spatial attention weight, P 3 It is the feature map after spatial weight allocation.

[0037] The multi-scale feature extraction module described in step 5 and step 6 includes five convolution modules, namely: MCBA, MCBB, MCBC, MCBD and MCBE. The specific steps of the above modules are as follows:

[0038] First, P 3 and P' 3 Input three cascaded MCBA to get E 1 and B 1, the specific formula is as follows:

[0039] E 1 =MCBA(MCBA(MCBA(P 3 )))

[0040] B 1 =MCBA(MCBA(MCBA(P' 3 )))

[0041] Where MCBA(·) represents the MCBA convolution operation. Then, E 1 and B 1 Enter a MCBB to get E 2 and B 2 , the formula is as follows:

[0042] E 2 =MCBB(E 1 )

[0043] B 2 =MCBB(B 1 )

[0044] Where MCBB(·) represents the MCBB convolution operation. 2 and B 2 Input three cascaded MCBCs to get E 3 and B 3 :

[0045] E 3 =MCBC(MCBC(MCBC(E 2 )))

[0046] B 3 =MCBC(MCBC(MCBC(B 2 )))

[0047] Where MCBC(·) represents the MCBC convolution operation, and then E 3 With B 3 Enter a MCBD to get E 4 With B 4 :

[0048] E 4 =MCBD(E 3 )

[0049] B 4 =MCBD(B 3 )

[0050] Where MCBD(·) represents the MCBD convolution operation, and finally E 4 With B 4Input two cascaded MCBEs to obtain the final multi-scale feature E 5 With B 5 :

[0051] E 5 =MCBE(MCBE(E 4 ))

[0052] B 5 =MCBE(MCBE(B 4 ))

[0053] Where MCBE(·) represents the MCBE convolution operation, and all convolution blocks are integrated into the multi-scale feature extraction module MFEM(·).

[0054] The beneficial effect of the present invention is that the present invention proposes a breast cancer tissue image classification method based on multi-scale feature fusion and attention mechanism. The method extracts the discriminative multi-scale features of the multidimensional image, enables the model to obtain a more comprehensive target feature representation, and adjusts the feature weights through the attention mechanism to strengthen the target feature. This algorithm significantly improves the accuracy of breast cancer tissue image classification, and effectively addresses the challenges brought about by the large scale variation of breast cancer tissue image features and susceptibility to noise interference. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] The drawings described herein are used to provide further understanding of the present invention and constitute a part of this application. The illustrative examples of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0056] Figure 1 It is a flow chart of the breast cancer tissue image classification method based on multi-scale feature fusion and attention mechanism of the present invention;

[0057] Figure 2 It is a schematic diagram of a breast cancer tissue image classification model based on multi-scale feature fusion and attention mechanism of the present invention;

[0058] Figure 3 Schematic diagram of the integrated attention module of the present invention;

[0059] Figure 4 This is a schematic diagram of the information retention module of the present invention;

[0060] Figure 5 Schematic diagram of the multi-scale feature extraction module of the present invention. DETAILED DESCRIPTION

[0061] The technical solution will be described in detail below through the accompanying drawings in the embodiments of the present invention. It should be noted that the embodiments shown are only partial examples of the present invention, not all embodiments. On this basis, any other embodiments that can be obtained by any person skilled in the art without additional innovation should be regarded as the scope of protection of the present invention.

[0062] Example:

[0063] In view of the problems existing in the above-mentioned prior art, this embodiment provides a breast cancer tissue image classification method based on multi-scale feature fusion and attention mechanism, and the method comprises the following steps:

[0064] Step 1: obtaining breast cancer tissue images to be classified;

[0065] Step 2: Divide the breast cancer tissue image dataset into a training set, a validation set, and a test set according to a preset ratio;

[0066] Step 3: Preprocess and enhance the original data;

[0067] Step 4: generating a classification model, inputting the training set of the data set into the classification model, training the breast cancer tissue image classification model, and evaluating the model using the validation set;

[0068] Step 5: Input the test set into the trained breast cancer tissue image classification model to obtain the image classification results.

[0069] Specifically, the structure of the breast cancer tissue image classification model is as follows: Figure 2 As shown. The model structure is as follows:

[0070] ① Construct a pre-convolution module, which is composed of the pre-convolution blocks of the InceptionV3 network. It perceives and extracts low-level features in local areas, reduces the spatial dimension of the data, and obtains shallow semantic features.

[0071] ②Integrated attention module, see Figure 3 This module uses the channel attention mechanism to guide the model to selectively focus on the feature channels that are most helpful for the classification task, and ignore irrelevant or redundant channels. The spatial attention mechanism is used to adjust the weights of different areas of the image, so that the model can focus more on the image areas that are important for the classification task. The specific calculation process of the integrated attention module is described as follows:

[0072] ω 1 =σ(C1D k (GAP(P 1 )))

[0073] P 2 =P 1 ·ω 1

[0074] ω 2 =σ(f 7×7 ([F avg (P 2 );F max (P 2 )]))

[0075] P 3 =P 2 ·ω 2

[0076] Among them, GAP(·) represents global average pooling, C1D k (·) represents the one-dimensional convolution that maps the input feature map to a k-dimensional vector, σ(·) represents the Sigmoid function, and P 1 is the shallow semantic feature map, ω 1 is the channel attention weight, P 2 is the feature map after channel weight assignment, f 7×7 (·) represents a 7×7 convolution operation, F avg (·) represents the average pooling operation, F max (·) represents the maximum pooling operation, ω 2 is the spatial attention weight, P 3 It is the feature map after spatial weight allocation.

[0077] ③Information retention module, see Figure 4 The information preservation module performs multi-scale processing on the feature map during the feature map dimensionality reduction process, fuses the extracted spatial detail features and global statistical features, and adjusts the weight of the feature map through the channel weight to alleviate the loss of detail information. This process can be formulated as:

[0078] P' 3 =σ'((f 3×3 (P 3 )+f 1×1 (F avg (P 3 )))σ(C1D k (GAP(P 3 ))))

[0079] where f 3×3 (·) represents 3×3 convolution, f 1×1 (·) represents 1×1 convolution, σ'(·) represents SiLU function, P 3 represents the input feature map, P' 3 Represents the feature map after dimensionality reduction.

[0080] ④The multi-scale feature extraction module includes five different convolution blocks, see Figure 5This module uses a multi-scale convolution strategy and adopts convolution kernels of different sizes for convolution operations. Smaller convolution kernels can better extract detail information, while larger convolution kernels can extract a wide range of spatial features, so that image information at different levels can be effectively processed.

[0081] The following is an example of actual application, which includes the following steps:

[0082] Step 1, obtain a breast cancer tissue image dataset, wherein the dataset is selected from the breast cancer tissue image human epidermal growth factor receptor 2 (HER2) classification of BCI as an example;

[0083] Step 2: Divide the image dataset into training set, validation set, and test set according to the preset ratio, including 3117, 779, and 520 images and corresponding category labels respectively;

[0084] Step 3, preprocessing the original data. To ensure that the images have the same size and resolution, this implementation case uniformly adjusts the resolution of the original images to 299×299.

[0085] Step 4, generating a breast cancer tissue image classification model, inputting the training set of the data set into the breast cancer tissue image classification model to train the breast cancer tissue image classification model, and evaluating the model using the validation set; the network uses the Adam optimizer, the batch size is set to 64, the epoch is 250, and the initial learning rate is set to 0.0005;

[0086] Step 5: Input the test set of the dataset into the trained image classification model to obtain the classification results. The classification results measure the model performance through Accuracy, Precision and F1-score.

[0087] After multiple rounds of experiments, the results of the HER2 classification experiment of breast cancer tissue images showed that the accuracy, precision and F1-score of the present invention reached 93.65%, 94.07% and 93.22%. Compared with the suboptimal method Vitb_16, the accuracy increased by 3.25%, the precision increased by about 1.87%, and the F1-score increased by about 5.62%. By combining multi-scale feature fusion with the attention mechanism, the present invention enables the model to have stronger feature representation ability and target feature discrimination ability, and achieves more accurate classification.

[0088] Table 1 Experimental results

[0089]

[0090] In summary, the present invention provides a new breast cancer tissue image classification method. This method improves the feature representation ability of the model by fusing multi-scale features, and uses an attention mechanism to enhance the model's sensitivity to target features, thereby improving image classification performance.

[0091] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made to the present invention shall be included in the protection scope of the present invention.

Claims

1. A breast cancer tissue image classification method based on multi-scale feature fusion and attention mechanism, characterized in that: The following steps are involved: Step 1: Select a breast cancer tissue image dataset, divide the breast cancer tissue image dataset into a training set, a validation set, and a test set, and preprocess the image data to obtain the processed image X org , further for image X org Perform data enhancement, including color jitter and random rotation. The formula is as follows: X=DA(X org ) Among them, X org is the image processed for breast cancer tissue, DA(·) represents the data augmentation strategy, and X is the image after data augmentation for breast cancer tissue; Step 2: Sample breast cancer tissue data enhanced images X and corresponding classification labels from the training set to form a batch, that is, batch = Cat (X, Y), where Cat is a tensor concatenation operation and Y is the category label of the image; Step 3: Perform pre-convolution on the batch data to reduce the spatial dimension of the data and obtain the feature map P1, that is, P1 = pconv(X), where pconv(·) represents the pre-convolution operation; Step 4: Use the channel attention module to adjust the channel weight of the feature map P1 to obtain P2. Further, adjust the spatial weight of P2 through the spatial attention module to obtain the final result of weight distribution P3; the channel attention module and the spatial attention module together constitute an integrated attention module, which is expressed as follows: P3=IAM(P1) Among them, IAM(·) represents the integrated attention module; Step 5: Extract multi-scale features of high-dimensional feature maps. First, use the multi-scale feature extraction module to process the high-dimensional feature map to obtain the high-dimensional deep feature E5, and then use the information retention module to adjust the size of the high-dimensional deep feature to obtain E′5, ensuring that the size of the feature map is consistent in the subsequent feature fusion process. This process can be expressed by the formula: E5=MFEM(P3) E′5=IPM(E5) Among them, MFEM(·) represents the multi-scale feature extraction module, and IPM(·) represents the information preservation module; Step 6: Extract multi-scale features of low-dimensional feature maps. First, use the information retention module to adjust the feature map dimension to obtain a low-dimensional feature map P′3. Further, use the multi-scale feature extraction module to process the low-dimensional feature map to obtain a low-dimensional deep feature B5, which can be expressed as: P′3=IPM(P3) B5=MFEM(P′3) Among them, IPM(·) represents the information preservation module, P′3 represents the feature map after dimensionality reduction, and MFEM(·) represents the multi-scale feature extraction module; Step 7: Fuse the deep features of the multi-dimensional image; fuse the multi-scale feature B5 of the low-dimensional feature map with the multi-scale feature E′5 of the high-dimensional feature map to obtain the deep fusion feature R. This step can be expressed by the formula: R=Cat(E′5, B5); Step 8: Obtain the final classification result. First, use the integrated attention module to assign feature weights to the deep fusion feature R, and then output the final classification result R′ through the linear layer. The formula is as follows: R′=Linear(IAM(R)) Step 9: Average the loss of all samples to get the overall cross entropy loss L total , the loss is defined as follows: Among them, L ce represents the cross entropy loss function, y i is the true category (label) of the i-th sample, C is the number of categories, and NNN is the number of samples; Step 10, repeating steps 3 to 9 until the current number of iterations reaches a predefined maximum number of iterations; Step 11: Send the breast cancer tissue image to be classified into the trained model for forward propagation to obtain the classification result.

2. The breast cancer tissue image classification method based on multi-scale feature fusion and attention mechanism according to claim 1, characterized in that: The calculation formula of the integrated attention module described in step 4 is as follows: ω1=σ(C1D k (GAP(P1))) P2=P1·ω1 ω2=σ(f 7×7 ([F avg (P2);F max (P2)])) P3=P2·ω2 Among them, GAP(·) represents global average pooling, C1D k (·) represents a one-dimensional convolution that maps the input feature map to a k-dimensional vector, σ(·) represents the Sigmoid function, P1 is the shallow semantic feature map, ω1 is the channel attention weight, P2 is the feature map after the channel weight is assigned, and f 7×7 (·) represents a 7×7 convolution operation, F avg (·) represents the average pooling operation, F max (·) represents the maximum pooling operation, ω2 is the spatial attention weight, and P3 is the feature map after spatial weight assignment.

3. The breast cancer tissue image classification method based on multi-scale feature fusion and attention mechanism according to claim 1, characterized in that: The multi-scale feature extraction module described in step 5 and step 6 includes five convolution modules, namely: MCBA, MCBB, MCBC, MCBD and MCBE. The specific steps of the above modules are as follows: First, P3 and P′3 are input into three cascaded MCBAs to obtain E1 and B1. The specific formula is as follows: E1=MCBA(MCBA(MCBA(P3))) B1=MCBA(MCBA(MCBA(P3))) Where MCBA(·) represents the MCBA convolution operation; E1 and B1 are input into an MCBB to obtain E2 and B2, the formula is as follows: E2=MCBB(E1) B2=MCBB(B1) Where MCBB(·) represents the MCBB convolution operation; the obtained E2 and B2 are input into three cascaded MCBCs to obtain E3 and B3: E3=MCBC(MCBC(MCBC(E2))) B3=MCBC(MCBC(MCBC(B2))) Where MCBC(·) represents the MCBC convolution operation, and then E3 and B3 are input into an MCBD to obtain E4 and B4: E4=MCBD(E3) B4=MCBD(B3) Among them, MCBD(·) represents the MCBD convolution operation. Finally, E4 and B4 are input into two cascaded MCBEs to obtain the final multi-scale features E5 and B5: E5=MCBE(MCBE(E4)) B5=MCBE(MCBE(B4)) Where MCBE(·) represents the MCBE convolution operation, and all convolution blocks are integrated into a multi-scale feature extraction module MFEM(·).

Images