A breast cancer pathology image classification device and method based on deep learning

Abstract

The present invention provides a deep learning-based breast cancer pathology image classification device and method, belonging to the field of image recognition technology. It solves the problem that clinical breast cancer diagnosis is time-consuming and labor-intensive due to the complexity and information interweaving of breast tissue pathology images. The method comprises the following steps: obtaining a breast tissue pathology image and preprocessing the image; utilizing a dilated convolution and a parallel convolution branch structure, adding an activation factor with adjustable parameters to each branch and combining it with a residual connection to obtain a receptive field scale attention module; building a first-level neural network based on the receptive field scale attention module, extracting features based on image patches, and training a feature extractor; fusing patch features through a tiled feature fusion module; building a second-level neural network based on a residual hybrid attention module, extracting global features based on the feature map after feature fusion to achieve image-level prediction and classification, training the network, and saving the final model. The present invention is applied to the classification of breast cancer pathology images.

Description

Technical Field

[0001] The present invention provides a deep learning-based breast cancer pathology image classification device and method, belonging to the field of image recognition technology. Background Art

[0002] According to data released by the World Health Organization's International Agency for Research on Cancer (IARC) in 2021, breast cancer has become the world's number one cancer. The most reliable way to diagnose cancer clinically is through a breast biopsy. During a biopsy, a tissue sample is surgically removed for analysis. The microscopic imaging data of biopsy samples is large in volume and complex in nature, requiring professional pathologists to perform intensive work and prone to diagnostic errors. In addition, the current distribution of medical resources in my country is relatively uneven, with relatively backward technology and equipment in remote areas and insufficient experience for doctors. Therefore, an efficient classification and recognition technology for breast cancer pathology images is necessary to assist pathologists in clinical diagnosis, improve the speed of diagnosis, and reduce the misdiagnosis rate.

[0003] Numerous researchers have continuously experimented with various algorithms and optimized them to achieve optimal results for computer-based diagnosis of breast cancer pathology images. However, image classification remains challenging, especially for pathology images, which are inherently challenging. Cancer cells often exhibit significant variation, with structural and morphological diversity resulting in complex geometric structures and textures. Furthermore, the complexity of image diagnosis is exacerbated by factors such as staining during the pathology slide production process, as well as by differences in the equipment used to acquire the images. Furthermore, training deep convolutional neural networks requires a vast number of samples, while medical pathology images are relatively limited due to difficulties in labeling and concerns about patient privacy. Existing methods struggle to capture these crucial details, resulting in poor detection performance. Furthermore, current models for breast cancer pathology image classification consume significant computational resources, making them difficult to train and deploy on limited hardware resources. Summary of the Invention

[0004] In order to solve the problems in the prior art of complex breast cancer pathology image structure, interwoven information, and difficulty in distinguishing tissue categories, which make clinical diagnosis difficult, time-consuming, and labor-intensive, the present invention provides a breast cancer pathology image classification device and method based on deep learning.

[0005] In order to solve the above technical problems, the present invention adopts a technical solution: a breast cancer pathology image classification device based on deep learning, comprising:

[0006] Pathology slice scanner, used to scan breast tissue slices to obtain breast pathology images;

[0007] Processor: The processor is connected to the pathology slice scanner and includes a preprocessing module, a detection module, and an analysis module. The preprocessing module is used to perform image normalization and standardization preprocessing on the breast pathology images acquired by the pathology slice scanner;

[0008] The detection module is used to input the preprocessed breast pathology image into a lightweight deep learning model, first extract image block features, perform feature fusion, and then extract global features, and classify the breast pathology image based on the extracted breast cancer pathology image features;

[0009] The lightweight deep learning model is based on a convolutional neural network structure, consisting of a two-stage neural network, integrating a receptive field scale attention module, a tiled feature fusion, and a residual hybrid attention module;

[0010] The analysis module is used to perform visual analysis on breast pathology images based on breast cancer pathology image features extracted by the lightweight deep learning model, and the analysis results are displayed on a display screen.

[0011] A deep learning-based breast cancer pathology image classification method comprises the following steps:

[0012] S1: Obtain a breast cancer pathology image dataset, normalize and standardize the images, and use data augmentation techniques to expand and balance the dataset, dividing it into training and test sets. Use a sliding window algorithm to extract image patches from the images. The patch size is determined based on the principle of containing complete tissue information without destroying cell distribution.

[0013] S2: Constructing a receptive field-scale attention module: We use parallel convolution to assemble a branch structure, add dilated convolutions with different dilation rates to each branch, and use a grid search algorithm to specify the activation factor of each branch. The receptive field-scale attention module is obtained by combining the three.

[0014] S3: Construct a residual hybrid attention module: implement channel attention and spatial attention in series, and combine them with residual connections to obtain a residual hybrid attention module;

[0015] S4: Feature fusion: Fusing patch features from the same image by tiling.

[0016] S5: Build the first-level neural network: The first-level neural network is the Receptive Field Scale Activation Module Network (RFSAM-Net): Design the backbone network and embed the receptive field scale attention module in step S2 into the backbone network. Use a four-layer module framework and pass the extracted features through a global average pooling layer to perform structural regularization on the entire network. The feature vector obtained by the global average pooling layer is connected to the fully connected layer, and the predicted category probability of the patch is obtained through calculation and transformation.

[0017] S6: Build the second-level neural network: The second-level neural network is the Residual Convolutional Block Attention Module Network (ResCBAM-Net): Design the backbone network and embed the Residual Convolutional Block Attention Module from step S3 into the backbone network. Pass the extracted features through a global average pooling layer to perform structural regularization on the entire network. The feature vector obtained by the global average pooling layer is connected to the fully connected layer, and the predicted category probability of the entire image is obtained through calculation and transformation.

[0018] S7: After the overall network structure is built, the parameters in the network are trained and updated: after the first-level neural network training is completed, the fully connected layer and the subsequent parts are discarded and used as feature extractors to assist the second-level neural network training. After the two-stage network training is completed, the final breast cancer pathology image classification model is obtained.

[0019] The steps for constructing the receptive field scale attention module in step S2 are as follows:

[0020] First, 1×1 convolution is used to reduce the dimension, and a dilated convolution with an adjustable ratio parameter is added to the branch structure to expand the receptive field and capture multi-scale contextual information. The subsequent feature map is then channel-merged with the original input map. Then, a grid search algorithm is used to specify activation factors for different branches to control the contribution of different branches to the module output and capture highly correlated features.

[0021] The structure of the receptive field scale attention module includes:

[0022] Branch 1: Convolutional layer RFSAM1_conv1, using 1×1 convolution kernel, stride 1; convolutional layer RFSAM1_conv2, using 3×3 convolution kernel, stride 1; convolutional layer RFSAM1_conv3, using 3×3 convolution kernel, stride 1, padding 2; convolutional layer RFSAM1_conv4, using 3×3 convolution kernel, stride 1, padding 3, dilation 4; BN and ReLU are used in all the above layers;

[0023] Branch 2: Convolutional layer RFSAM2_conv1, using 1×1 convolution kernel, stride 1; convolutional layer RFSAM2_conv2, using 3×3 convolution kernel, stride 1, padding 2; convolutional layer RFSAM2_conv3, using 3×3 convolution kernel, stride 1, padding 2, dilation 3; BN and ReLU are used in all the above layers;

[0024] Branch 3: Convolutional layer RFSAM3_conv1, using 1×1 convolution kernel with a step size of 1; convolutional layer RFSAM3_conv2, using 3×3 convolution kernel with a step size of 1; convolutional layer RFSAM3_conv3, using 3×3 convolution kernel with a step size of 1 and a padding of 3; BN and ReLU are used in all the above layers;

[0025] Branch 4: Use a 3×3 convolution kernel with a stride of 1, a padding of 3, and use BN and ReLU;

[0026] Branch 5: residual connection;

[0027] Feature fusion: The outputs of the five branches are fused through the concat operation, and the activation factor values ​​are determined by the grid search algorithm on the first four branches.

[0028] The structure of the residual hybrid attention module in step S3 includes:

[0029] Branch 1: Convolutional layer ResCBAM1_conv1, using a 1×1 convolution kernel, a stride of 1, and using BN and ReLU;

[0030] The ResCBAM1_channel attention module performs channel compression and recovery operations on the two outputs through two parallel maximum pooling layers and average pooling layers. The two outputs are added and activated by the sigmoid activation function to obtain the final output channel mask consistent with the number of input channels. The mask is multiplied by the input of the ResCBAM1_channel attention module to obtain the final output of the ResCBAM1_channel attention module;

[0031] ResCBAM1_Spatial Attention Module: The input passes through two parallel maximum pooling layers and average pooling layers to obtain two feature maps. A concat operation is performed to obtain a feature map, which is then reduced to 1D through a convolution operation and nonlinearly transformed by a sigmoid activation function to obtain a mask on the spatial area. The mask is multiplied by the input of the ResCBAM1_Spatial Attention Module to obtain the final output of the ResCBAM1_Spatial Attention Module.

[0032] The convolutional layer ResCBAM1_conv2 uses a 3×3 convolution kernel with a stride of 1, a padding of 1, and uses BN and ReLU;

[0033] Branch 2: residual connection;

[0034] Feature fusion: The outputs of the two branches are fused through the concat operation.

[0035] The feature fusion in step S4 is to restore the patch feature map according to the patch generation order, so that the second-level neural network can extract the global features of the image.

[0036] The overall structure of the first-level neural network includes:

[0037] The first convolutional layer, conv1, uses a 3×3 convolution kernel with a stride of 2 and uses BN and ReLU;

[0038] First receptive field scale attention module;

[0039] The first pooling layer, pool1, uses maximum pooling with a window size of 3 and a stride of 2;

[0040] Second receptive field scale attention module;

[0041] The second convolutional layer, conv2, uses a 3×3 convolution kernel with a stride of 2 and uses BN and ReLU;

[0042] The third receptive field scale attention module;

[0043] The third convolutional layer, conv3, uses a 3×3 convolution kernel with a stride of 2 and uses BN and ReLU;

[0044] Fourth receptive field scale attention module;

[0045] Then, global average pooling is used to take the feature map output of each channel as a neuron, and finally classification is performed through a fully connected layer and a Softmax function.

[0046] The overall structure of the second-level neural network includes:

[0047] The first convolutional layer, conv1, uses a 1×1 convolution kernel with a stride of 1 and uses BN and ReLU.

[0048] The second convolutional layer conv2 uses a 3×3 convolution kernel with a stride of 1 and uses BN and ReLU;

[0049] The first residual hybrid attention module;

[0050] The third convolutional layer, conv3, uses a 3×3 convolution kernel with a stride of 2 and uses BN and ReLU;

[0051] The fourth convolutional layer conv4 uses a 3×3 convolution kernel with a stride of 1 and uses BN and ReLU;

[0052] The second residual hybrid attention module;

[0053] Then, global average pooling is used to take the feature map output of each channel as a neuron, and finally classification is performed through a fully connected layer and a Softmax function.

[0054] The step S7 specifically includes:

[0055] First, train the first-level neural network: use a grid search algorithm to screen and determine the optimal activation factor; input the preprocessed image data in step S1 into the first-level neural network in batches, and use a sliding window strategy to intercept image patches in the image, with overlap between image patches; construct a loss function based on the predicted value and the true label and calculate the loss; use an optimization algorithm to perform backpropagation to update the network parameters until the loss no longer decreases, at which point the network parameters are saved as the final model; after the first-level neural network training is completed, discard the global average pooling and the subsequent part, and use it as a feature extractor to provide local features for the subsequent training of the second-level neural network;

[0056] Then, the second-level neural network is trained: the image data preprocessed in step S1 is input into the second-level neural network in batches, and the image patches are intercepted in the image using a sliding window strategy, so that the image patches no longer overlap; after the image features are extracted by the first-level neural network, feature fusion is performed using a tiled feature fusion method to obtain a feature map corresponding to the original image. The above feature map is used as the input of the second-level neural network, and a loss function is constructed and the loss is calculated based on the final prediction value of the feature map by the second-level neural network and the true label of the image. The optimization algorithm is used for backpropagation to update the network parameters until the loss no longer decreases. At this time, the network parameters are saved as the final model.

[0057] The beneficial effects of the present invention over the prior art are as follows: the deep learning-based breast cancer pathology image classification device and method provided by the present invention are implemented through a two-level neural convolutional network, which reduces the computational complexity of the model while having a stronger feature extraction capability for pathological images. In the first-level neural network, a receptive field scale attention module is proposed by utilizing multiple parallel convolutions and combining the different contributions of different branches to the module output. In the second-level convolutional neural network, a residual mixed attention module is proposed by combining channel attention, spatial attention and residual connection. The network extracts global features from the patch feature map after feature fusion to achieve image-level classification. The model designed by the present invention not only has a higher accuracy, but also has a small number of parameters and computational complexity, and is relatively easier to deploy in practice. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] The present invention will be further described below with reference to the accompanying drawings.

[0059] Figure 1 This is a system block diagram of the breast cancer pathology image classification device of the present invention.

[0060] Figure 2 Flowchart of the breast cancer pathology image classification method of the present invention.

[0061] Figure 3 Schematic diagram of the structure of the receptive field scale attention module constructed in the present invention.

[0062] Figure 4 Structural diagram of the residual hybrid attention module constructed for this invention.

[0063] Figure 5 A structural diagram of the tiled feature fusion method designed for the present invention.

[0064] Figure 6 The overall structure diagram of the first-level neural network designed for this invention.

[0065] Figure 7 The overall structure diagram of the second-level neural network designed for this invention.

[0066] Figure 8 This is a comparison chart of the network model proposed in this invention and other network methods in terms of parameter quantity, computational complexity, and model classification accuracy. DETAILED DESCRIPTION

[0067] Due to the complex background and disordered cell distribution of breast cancer pathology images, as well as the interweaving of benign and malignant tissues, and the fact that models with a huge number of parameters and computational complexity undoubtedly increase the difficulty of actual deployment, the classification and discrimination of breast cancer pathology images is difficult and unique. Therefore, the present invention proposes a deep learning-based breast cancer pathology image classification device, whose structure is as follows: Figure 1 As shown, a pathology slice scanner is connected to a processor equipped with a computer program of the breast cancer pathology image classification method based on deep learning of the present invention via a communication line, and the processor is connected to a display screen to display the analysis results. The pathology slice scanner is used to scan breast tissue slices to obtain breast pathology images; the processor includes a preprocessing module, a detection module, and an analysis module. The preprocessing module is used to perform image normalization and standardization preprocessing on the patient's breast pathology images obtained by the pathology slice scanner; the detection module is used to input the preprocessed breast pathology images into a lightweight deep learning model, first extract image block features, and perform feature fusion, and then extract global features through a second-level neural network, effectively utilizing the features to achieve image classification; the lightweight deep learning model is based on a convolutional neural network structure, consisting of a two-level neural network, integrating a receptive field scale attention module, a tiled feature fusion, and a residual mixed attention module; the analysis module is used to perform visual analysis of the breast pathology images based on the features of the breast cancer pathology images extracted by the lightweight deep learning model, displaying the focused key tissue areas in the form of a heat map to assist clinical diagnosis, and the results are displayed on the display screen.

[0068] The steps of the breast cancer pathology image classification method based on deep learning proposed in this invention are as follows: Figure 2 As shown, specifically including:

[0069] Step 1: Dataset Preparation

[0070] 1.1 Data Collection

[0071] Since the quality of acquired pathological images varies depending on the settings of digital pathology slide scanners, this paper uses public breast cancer pathology image data to verify the performance of the model. These are the BreakHis breast cancer pathology image dataset and the ICIAR2018 breast cancer pathology image dataset, which can be downloaded from the following two websites: https: / / iciar2018-challenge.grand-challenge.org / Dataset / 368 and https: / / web.inf.ufpr.br / vri / databases / breast-cancer-histopathological-database-breakhis / .

[0072] 1.2 Data Preprocessing

[0073] First, the Macenko method based on staining deconvolution was used to standardize the image staining. Second, image geometric transformation and color transformation were used to perform data expansion and category balance. Image geometric transformation involves horizontal flipping, vertical flipping, mirror flipping, and rotation of the image. Color transformation involves converting the image to the HSV color space to change and enhance the color values ​​of the image. A sliding window algorithm was used to extract image patches from the image. The patch size was determined based on the principle of containing complete tissue information in the patch without destroying cell distribution. Different sliding window sizes were selected for different datasets. The window sizes for the BreakHis dataset and ICIAR2018 dataset were set to 224×224 and 512×512, respectively.

[0074] Step 2: Network Architecture Design

[0075] 1.1 Feature Enhancement Module Design

[0076] Receptive Field Scale Attention Module (RFSAM): Figure 3 As shown, it is composed of four branch convolutions and residual connections;

[0077] Branch 1: Convolutional layer RFSAM1_conv1, using 1×1 convolution kernel, stride 1; convolutional layer RFSAM1_conv2, using 3×3 convolution kernel, stride 1; convolutional layer RFSAM1_conv3, using 3×3 convolution kernel, stride 1, padding 2; convolutional layer RFSAM1_conv4, using 3×3 convolution kernel, stride 1, padding 3, dilation 4; BN and ReLU are used in all the above layers;

[0078] Branch 2: Convolutional layer RFSAM2_conv1, using 1×1 convolution kernel, stride 1; convolutional layer RFSAM2_conv2, using 3×3 convolution kernel, stride 1, padding 2; convolutional layer RFSAM2_conv3, using 3×3 convolution kernel, stride 1, padding 2, dilation 3; BN and ReLU are used in all the above layers;

[0079] Branch 3: Convolutional layer RFSAM3_conv1, using 1×1 convolution kernel with a step size of 1; convolutional layer RFSAM3_conv2, using 3×3 convolution kernel with a step size of 1; convolutional layer RFSAM3_conv3, using 3×3 convolution kernel with a step size of 1 and a padding of 3; BN and ReLU are used in all the above layers;

[0080] Branch 4: Use a 3×3 convolution kernel with a stride of 1, a padding of 3, and use BN and ReLU;

[0081] Branch 5: residual connection;

[0082] Feature fusion: The outputs of the five branches are fused through the concat operation, and the grid search algorithm is used on the first four branches to optimize the contribution of each branch to the output, so that the relevant features of the image are highly concentrated.

[0083] Residual Convolutional Block Attention Module (ResCBAM): Figure 4 As shown, it is composed of two branches;

[0084] Branch 1: Convolution layer ResCBAM1_conv1, using 1×1 convolution kernel, stride 1, and using BN and ReLU; ResCBAM1_Channel Attention Module, through two parallel maximum pooling layers (MaxPool) and average pooling layers (AvgPool), the two outputs (C×1×1) are compressed (C / 4×1×1) and restored (C×1×1), the two outputs are added and activated by the sigmoid activation function to obtain the final output channel mask (C×1×1) with the same number of input channels. The mask is multiplied by the input of the module to obtain the final output of the module; R The esCBAM1_spatial attention module passes its input through two parallel max pooling layers (MaxPool) and average pooling layers (AvgPool) to obtain two feature maps (1×H×W). A concat operation is performed to obtain a feature map of size 2×H×W. This map is then reduced to 1D through a convolution operation and nonlinearly transformed by a sigmoid activation function to obtain a mask (1×H×W) on the spatial region. The mask is multiplied by the input of the module to obtain the final output of the module. The convolution layer ResCBAM1_conv2 uses a 3×3 convolution kernel with a stride of 1 and a padding of 1. It also uses batch normalization and ReLU.

[0085] Branch 2: residual connection;

[0086] Feature fusion: The outputs of the two branches are fused through the concat operation.

[0087] 1.2 Feature fusion method design: Flat feature fusion, such as Figure 5 As shown;

[0088] The edge parts of different patches contain key information after integration. Therefore, the present invention proposes a tiled feature fusion method to reorganize the patch feature map according to the position of the corresponding patch, to maximize the correspondence between the features and the positions of the original input image, so that the fused feature map contains more discriminative features.

[0089] 1.3 Overall Network Design

[0090] The first level neural network structure, such as Figure 6 As shown in the figure: the first convolution layer conv1 adopts 3×3 convolution kernel, step size 2, and uses BN and ReLU; add a layer of receptive field scale attention module; the first pooling layer pool1 adopts maximum pooling, window size 3, and step size 2; add a layer of receptive field scale attention module; the second convolution layer conv2 adopts 3×3 convolution kernel, step size 2, and uses BN and ReLU; add a layer of receptive field scale attention module; the third convolution layer conv3 adopts 3×3 convolution kernel, step size 2, and uses BN and ReLU; add a layer of receptive field scale attention module; then use global average pooling to take the feature map output mean result of each channel as a neuron, and finally classify through a fully connected layer and Softmax function.

[0091] The second level neural network structure, such as Figure 7 As shown in the figure: the first convolution layer conv1 adopts 1×1 convolution kernel with a step size of 1, and uses BN and ReLU; the second convolution layer conv2 adopts 3×3 convolution kernel with a step size of 1, and uses BN and ReLU; a layer of residual mixed attention module is added; the third convolution layer conv3 adopts 3×3 convolution kernel with a step size of 2, and uses BN and ReLU; the fourth convolution layer conv4 adopts 3×3 convolution kernel with a step size of 1, and uses BN and ReLU; a layer of residual mixed attention module is added; then global average pooling is used to output the mean result of the feature map of each channel as a neuron, and finally classification is performed through a fully connected layer and a Softmax function.

[0092] Step 3: Network structure parameter design

[0093] The present invention determines the hyperparameters in the final network structure by conducting a grid search experiment on the activation factors on different branches in the receptive field scale attention module. In the breast cancer pathology image classification task, it is necessary to pay attention not only to the distribution information and texture characteristics of the cell nucleus, but also to the cell morphology, cell nuclear level and different interstitial components. Different activation factor combination values ​​correspond to different network receptive fields, thereby capturing information at different scales. Finally, the present invention experimentally determined the optimal combination values ​​of activation factors in the four-layer receptive field scale attention module used in the first-level neural network (1.0, 1.0, 0.1, 0.2; 0.7, 0.9, 0.1, 0.9; 0.4, 0.6, 0.4, 0.8; 0.6, 0.6, 0.3, 0.3).

[0094] Step 4: Experimental process and result analysis

[0095] 1.1 Network Training

[0096] Based on two data sets, the lightweight deep learning model is trained and the performance of the lightweight deep learning model is verified. The initialization parameters of the lightweight deep learning model during training are set as follows:

[0097] The batch size is 16, and the L2 regularization parameter is set to 4×10 -4 , the optimizer uses the Adam algorithm to optimize, parameters β 1 Set to 0.9, β 2 The initial learning rate and epochs for the two-stage neural network training were (0.002, 20) and (0.001, 80), respectively. A lambda learning strategy was used, with the learning rate gradually decreasing over the training process, reaching 0.1 times its original value after training. The target loss function used the cross-entropy loss function, and L2 regularization was used to add a parameter penalty term to the loss function to reduce model complexity, improve generalization, and address overfitting.

[0098] The Adam optimizer formula is:

[0099] ;

[0100] ;

[0101] ;

[0102] ;

[0103] ;

[0104] ;

[0105] In the above formula: θ t is the parameter vector at time t; g t is the loss function about θ t gradient; m t is the first-order moment gradient at time t; v t is the second-order moment gradient at time t; β 1 is the first-order moment decay rate; β 2 is the second-order moment decay rate; is the first-order moment bias correction; is the second-order moment bias correction, is a constant added to maintain numerical stability, is the learning rate, yes β 1 The tth power, yes β 2 t to the power of .

[0106] The cross entropy loss function formula is:

[0107] ;

[0108] In the above formula: y i It is a sample x i The corresponding one-hot encoding (0 or 1) after the real label is converted; Predict samples for the model x i is the prediction score for a certain category.

[0109] The loss function formula with L2 regularization is:

[0110] ;

[0111] In the above formula: ∈[0,∞) is used to weigh the parameter penalty term The relationship between the loss function and A larger value represents a greater regularization strength, and the value is 1 during the training process of the present invention.

[0112] The L2 regularization formula is as follows:

[0113] ;

[0114] in w is the parameter vector.

[0115] 1.2 Experimental Results

[0116] As shown in Table 1, the lightweight deep learning model of the present invention has good classification performance on the BreakHis (40x, 100x, 200x, 400x) dataset and the ICIAR2018 dataset (200x), especially the best classification effect at 200x and 400x magnifications (classification accuracy rates of 99.50%, 97.03% and 97.99% respectively). Compared with Inception, Inception-ResNet and VGG networks, the lightweight deep learning model trained by the present invention has the best performance in all indicators. In addition, Figure 8 As shown, the lightweight deep learning model of the present invention has low parameter and computational complexity, and is easier to deploy and use in a limited resource environment.

[0117]

[0118] Table 1 Comparison results of the accuracy, precision, recall and F1 value of the method of the present invention and other methods on different data sets.

[0119] The receptive field-scale attention module, proposed in the first-level neural network of this invention, consists of two main components: multiple parallel convolution branches containing dilated convolutions, responsible for extracting multi-scale information and features; and a receptive field-scale attention mechanism, which modifies the distribution of the effective receptive field, clustering highly correlated features. This module employs dilated convolutions of varying ratios (1, 3, and 4) added to the three parallel branches, ensuring that the output of each parallel branch contains a wide range of semantic information and captures multi-scale information about the image. Furthermore, the present invention controls the influence of each branch on the output features by assigning different activation factors to different branches in the module. This modifies the range and degree of influence of input pixels on the output, thereby altering the distribution of the network's effective receptive field. While the influence of pixels within a typical network's receptive field follows a Gaussian distribution, typically decaying rapidly from the center. The receptive field-scale attention mechanism proposed in this invention enhances the influence of non-central portions of the image on the output feature map, improving the network's ability to extract features from the entire input region. A grid search is performed to determine the optimal combination of activation factors to achieve the optimal information extraction capability.

[0120] Since the first-level neural network is trained and learned based on patches, achieving patch-level prediction and classification, to verify the performance of the first-level neural network, three different voting strategies were used to fuse the patch prediction results to obtain the predicted image category: probability summation (sum), majority voting (maj), and maximum probability (max). As shown in Table 2, the image classification accuracy achieved by the first-level neural network (composed of a four-layer receptive field-scale attention module) is significantly superior to other methods, especially compared to the Inception series of networks with the same parallel convolutional structure, with classification accuracy improved by over 4%.

[0121]

[0122] Table 2 Comparison results of the accuracy of the first-level neural network proposed in this invention and other methods on different data sets.

[0123] The second-level neural network proposed in this paper is primarily composed of a residual hybrid attention module. This module consists of three main components: residual connections, spatial attention, and channel attention. Based on feature fusion, this network extracts global image features through a serial connection of spatial attention and channel attention to achieve image-level prediction and classification. This allows the network model to focus on information that is beneficial to classification and ignore irrelevant or even interfering information.

[0124] The feature fusion method proposed in this paper is a tiled feature fusion method. This method reorganizes the patch feature maps in the order in which the sliding window intercepts the patches, maximizing their positional correspondence with the original input image. Compared to the overlay fusion method, the tiled method proposed in this paper better considers the edge combination information of different patches, making it easier for the network to extract global image features and improving network performance.

[0125] Regarding the specific structure of the present invention, it should be noted that the connection relationship between the various component modules adopted in the present invention is definite and feasible. Except for those specifically described in the embodiments, the specific connection relationship can bring about corresponding technical effects and solve the technical problems raised by the present invention without relying on the execution of corresponding software programs. The components, modules, models of specific components appearing in the present invention, the connection methods between each other, and the conventional usage methods and expected technical effects brought about by the above-mentioned technical features, except for those specifically described, all belong to the disclosed contents in patents, journal articles, technical manuals, technical dictionaries, and textbooks that can be obtained by technical personnel in this field before the application date, or belong to the existing technologies such as conventional technology and common knowledge in this field. There is no need to elaborate, so that the technical solution provided in this case is clear, complete, and feasible, and the corresponding physical products can be reproduced or obtained based on this technical means.

[0126] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A deep learning-based breast cancer pathology image classification device, characterized by: include: Pathology slice scanner, used to scan breast tissue slices to obtain breast pathology images; Processor: The processor is connected to the pathology slice scanner and includes a preprocessing module, a detection module, and an analysis module. The preprocessing module is used to perform image normalization and standardization preprocessing on the breast pathology images acquired by the pathology slice scanner; The detection module is used to input the preprocessed breast pathology image into a lightweight deep learning model, first extract image block features, perform feature fusion, and then extract global features, and classify the breast pathology image based on the extracted breast cancer pathology image features; The lightweight deep learning model is based on a convolutional neural network structure, consisting of a two-stage neural network, integrating a receptive field scale attention module, a tiled feature fusion, and a residual hybrid attention module; The first-level neural network structure includes: the first convolution layer conv1, using 3×3 convolution kernel, stride 2, and using BN and ReLU; adding a receptive field scale attention module; the first pooling layer pool1, using maximum pooling, window size 3, stride 2; adding a receptive field scale attention module; the second convolution layer conv2, using 3×3 convolution kernel, stride 2, and using BN and ReLU; adding a receptive field scale attention module; the third convolution layer conv3, using 3×3 convolution kernel, stride 2, and using BN and ReLU; adding a receptive field scale attention module; then using global average pooling to take the feature map output mean result of each channel as a neuron, and finally through a fully connected layer and Softmax function for classification; The second-level neural network structure includes: the first convolution layer conv1, which uses a 1×1 convolution kernel with a step size of 1, and uses BN and ReLU; the second convolution layer conv2, which uses a 3×3 convolution kernel with a step size of 1, and uses BN and ReLU; adding a layer of residual mixed attention module; the third convolution layer conv3, which uses a 3×3 convolution kernel with a step size of 2, and uses BN and ReLU; the fourth convolution layer conv4, which uses a 3×3 convolution kernel with a step size of 1, and uses BN and ReLU; adding a layer of residual mixed attention module; then using global average pooling to take the mean result of the feature map output of each channel as a neuron, and finally through a fully connected layer and a Softmax function for classification; The analysis module is used to perform visual analysis on breast pathology images based on breast cancer pathology image features extracted by the lightweight deep learning model, and the analysis results are displayed on a display screen.

2. A deep learning-based breast cancer pathology image classification method, characterized by: The steps include: S1: Obtain a breast cancer pathology image dataset, normalize and standardize the images, and use data augmentation techniques to expand and balance the dataset, dividing it into training and test sets. Use a sliding window algorithm to extract image patches from the images. The patch size is determined based on the principle of containing complete tissue information without destroying cell distribution. S2: Constructing a receptive field-scale attention module: We use parallel convolution to assemble a branch structure, add dilated convolutions with different dilation rates to each branch, and use a grid search algorithm to specify the activation factor of each branch. The receptive field-scale attention module is obtained by combining the three. S3: Construct a residual hybrid attention module: implement channel attention and spatial attention in series, and combine them with residual connections to obtain a residual hybrid attention module; S4: Feature fusion: Fusing patch features from the same image by tiling. S5: Build the first-level neural network: The first-level neural network is the Receptive Field Scale Activation Module Network (RFSAM-Net): Design the backbone network and embed the receptive field scale attention module in step S2 into the backbone network. Use a four-layer module framework and pass the extracted features through a global average pooling layer to perform structural regularization on the entire network. The feature vector obtained by the global average pooling layer is connected to the fully connected layer, and the predicted category probability of the patch is obtained through calculation and transformation. S6: Build the second-level neural network: The second-level neural network is the Residual Convolutional Block Attention Module Network (ResCBAM-Net): Design the backbone network and embed the Residual Convolutional Block Attention Module from step S3 into the backbone network. Pass the extracted features through a global average pooling layer to perform structural regularization on the entire network. The feature vector obtained by the global average pooling layer is connected to the fully connected layer, and the predicted category probability of the entire image is obtained through calculation and transformation. S7: After the overall network structure is built, the parameters in the network are trained and updated: after the first-level neural network training is completed, the fully connected layer and the subsequent parts are discarded and used as feature extractors to assist the second-level neural network training. After the two-stage network training is completed, the final breast cancer pathology image classification model is obtained.

3. The method for breast cancer pathology image classification based on deep learning according to claim 2, characterized in that: The steps for constructing the receptive field scale attention module in step S2 are as follows: First, 1×1 convolution is used to reduce the dimension, and a dilated convolution with an adjustable ratio parameter is added to the branch structure to expand the receptive field and capture multi-scale contextual information. The subsequent feature map is then channel-merged with the original input map. Then, a grid search algorithm is used to specify activation factors for different branches to control the contribution of different branches to the module output and capture highly correlated features.

4. The method for breast cancer pathology image classification based on deep learning according to claim 3, characterized in that: The structure of the receptive field scale attention module includes: Branch 1: Convolutional layer RFSAM1_conv1, using 1×1 convolution kernel, stride 1; convolutional layer RFSAM1_conv2, using 3×3 convolution kernel, stride 1; convolutional layer RFSAM1_conv3, using 3×3 convolution kernel, stride 1, padding 2; convolutional layer RFSAM1_conv4, using 3×3 convolution kernel, stride 1, padding 3, dilation 4; BN and ReLU are used in all the above layers; Branch 2: Convolutional layer RFSAM2_conv1, using 1×1 convolution kernel, stride 1; convolutional layer RFSAM2_conv2, using 3×3 convolution kernel, stride 1, padding 2; convolutional layer RFSAM2_conv3, using 3×3 convolution kernel, stride 1, padding 2, dilation 3; BN and ReLU are used in all the above layers; Branch 3: Convolutional layer RFSAM3_conv1, using 1×1 convolution kernel with a step size of 1; convolutional layer RFSAM3_conv2, using 3×3 convolution kernel with a step size of 1; convolutional layer RFSAM3_conv3, using 3×3 convolution kernel with a step size of 1 and a padding of 3; BN and ReLU are used in all the above layers; Branch 4: Use a 3×3 convolution kernel with a stride of 1, a padding of 3, and use BN and ReLU; Branch 5: residual connection; Feature fusion: The outputs of the five branches are fused through the concat operation, and the activation factor values ​​are determined by the grid search algorithm on the first four branches.

5. The method for breast cancer pathology image classification based on deep learning according to claim 2, characterized in that: The structure of the residual hybrid attention module in step S3 includes: Branch 1: Convolutional layer ResCBAM1_conv1, using a 1×1 convolution kernel, a stride of 1, and using BN and ReLU; The ResCBAM1_channel attention module performs channel compression and recovery operations on the two outputs through two parallel maximum pooling layers and average pooling layers. The two outputs are added and activated by the sigmoid activation function to obtain the final output channel mask consistent with the number of input channels. The mask is multiplied by the input of the ResCBAM1_channel attention module to obtain the final output of the ResCBAM1_channel attention module; ResCBAM1_Spatial Attention Module: The input passes through two parallel maximum pooling layers and average pooling layers to obtain two feature maps. A concat operation is performed to obtain a feature map, which is then reduced to 1D through a convolution operation and nonlinearly transformed by a sigmoid activation function to obtain a mask on the spatial area. The mask is multiplied by the input of the ResCBAM1_Spatial Attention Module to obtain the final output of the ResCBAM1_Spatial Attention Module. The convolutional layer ResCBAM1_conv2 uses a 3×3 convolution kernel with a stride of 1, a padding of 1, and uses BN and ReLU; Branch 2: residual connection; Feature fusion: The outputs of the two branches are fused through the concat operation.

6. The method for breast cancer pathology image classification based on deep learning according to claim 2, characterized in that: The feature fusion in step S4 is to restore the patch feature map according to the patch generation order, so that the second-level neural network can extract the global features of the image.

7. The method for breast cancer pathology image classification based on deep learning according to claim 4, characterized in that: The overall structure of the first-level neural network includes: The first convolutional layer, conv1, uses a 3×3 convolution kernel with a stride of 2 and uses BN and ReLU; First receptive field scale attention module; The first pooling layer, pool1, uses maximum pooling with a window size of 3 and a stride of 2; Second receptive field scale attention module; The second convolutional layer, conv2, uses a 3×3 convolution kernel with a stride of 2 and uses BN and ReLU; The third receptive field scale attention module; The third convolutional layer, conv3, uses a 3×3 convolution kernel with a stride of 2 and uses BN and ReLU; Fourth receptive field scale attention module; Then, global average pooling is used to take the feature map output of each channel as a neuron, and finally classification is performed through a fully connected layer and a Softmax function.

8. The method for breast cancer pathology image classification based on deep learning according to claim 5, characterized in that: The overall structure of the second-level neural network includes: The first convolutional layer, conv1, uses a 1×1 convolution kernel with a stride of 1 and uses BN and ReLU. The second convolutional layer conv2 uses a 3×3 convolution kernel with a stride of 1 and uses BN and ReLU; The first residual hybrid attention module; The third convolutional layer, conv3, uses a 3×3 convolution kernel with a stride of 2 and uses BN and ReLU; The fourth convolutional layer conv4 uses a 3×3 convolution kernel with a stride of 1 and uses BN and ReLU; The second residual hybrid attention module; Then, global average pooling is used to take the feature map output of each channel as a neuron, and finally classification is performed through a fully connected layer and a Softmax function.

9. The method for breast cancer pathology image classification based on deep learning according to claim 2, characterized in that: The step S7 specifically includes: First, train the first-level neural network: use a grid search algorithm to screen and determine the optimal activation factor; input the preprocessed image data in step S1 into the first-level neural network in batches, and use a sliding window strategy to intercept image patches in the image, with overlap between image patches; construct a loss function based on the predicted value and the true label and calculate the loss; use an optimization algorithm to perform backpropagation to update the network parameters until the loss no longer decreases, at which point the network parameters are saved as the final model; after the first-level neural network training is completed, discard the global average pooling and the subsequent part, and use it as a feature extractor to provide local features for the subsequent training of the second-level neural network; Then, the second-level neural network is trained: the image data preprocessed in step S1 is input into the second-level neural network in batches, and the image patches are intercepted in the image using a sliding window strategy, so that the image patches no longer overlap; after the image features are extracted by the first-level neural network, feature fusion is performed using a tiled feature fusion method to obtain a feature map corresponding to the original image. The above feature map is used as the input of the second-level neural network, and a loss function is constructed and the loss is calculated based on the final prediction value of the feature map by the second-level neural network and the true label of the image. The optimization algorithm is used for backpropagation to update the network parameters until the loss no longer decreases. At this time, the network parameters are saved as the final model.

Images