Breast Pathology Image Recognition System Based on Context-Aware Multi-Scale Feature Fusion

Abstract

The present invention discloses a breast pathology image recognition system based on context-aware multi-scale feature fusion, comprising: extracting multi-scale features of breast tissue pathology images; using a cross-attention mechanism to extract extensive context information from the multi-scale feature map; performing upsampling operations on the multi-scale context views from top to bottom and cascading them with the adjacent lower-level context views, reducing the dimension of the output feature maps of the branches corresponding to each scale, unifying the size and splicing and fusing them through global average pooling to obtain multi-scale fusion features; and obtaining image category prediction results based on the multi-scale fusion features. By fusing multi-scale context information to enhance the correlation between features and categories, the context information of different levels obtained by different branches is fused twice to represent the comprehensive features of the image, thereby achieving the aggregation of multi-scale context-aware information and improving classification accuracy.

Description

Technical Field

[0001] The present invention relates to the technical field of breast pathology image recognition, and in particular to a breast pathology image recognition system based on context-aware multi-scale feature fusion. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] Early diagnosis of breast cancer is crucial for the treatment and prevention of the disease. The gold standard for diagnosing breast cancer is histopathological image analysis of a breast biopsy. During the diagnostic process, the operator stains the obtained tissue sections with hematoxylin and eosin (H&E) and uses a microscope to evaluate the morphology of the tissue cells to determine whether the breast lesion is benign or malignant.

[0004] Microscope-based breast pathology image recognition faces three challenges. First, microscopic images typically have extremely high resolution and therefore contain rich structural information and details, making them difficult to effectively characterize at a single scale. Second, microscopic images from different categories may exhibit some overlapping features, which can interfere with cancer detection. For example, images acquired at lower microscope magnifications almost always contain normal breast tissue areas, which makes feature extraction and classification of breast lesions difficult. Third, there is little difference between different categories of breast lesions, while lesions of the same category have small differences in the morphological appearance of pathology images.

[0005] Pathologists have to review a huge number of ultra-high-resolution histopathology images every day. When they are visually fatigued, they are prone to overlooking small lesions, leading to misjudgment. In addition, the spatial differences between different breast tissue pathology sections are diverse, and there is no clear boundary between the histological characteristics of benign and malignant tumors. It is difficult for pathology experts to fully impart their diagnostic experience to junior physicians in a short period of time.

[0006] With the continuous development of machine learning, traditional machine learning methods have become the most commonly used auxiliary means of image analysis. Previous breast pathology image-assisted analysis has mostly relied on manually designed features to characterize the diversity of breast cells. However, traditional methods often involve a large and complex feature engineering phase, and their data features are primarily derived through expert experience, making it difficult to capture underlying abstract features. Furthermore, feature extraction methods are often not universal across different tasks.

[0007] Deep learning-based methods have, to a certain extent, overcome the limitations of traditional machine learning methods. Capturing contextual information is essential for understanding the relationship between tissue cells and their surrounding background in pathology images. Existing models for contextual information use a single CNN network, which cannot guarantee the consistent extraction of high-quality features. While integrating features or results from multiple networks can improve network classification performance, these designs often overlook the importance of multi-scale information in various detection tasks. In pathology diagnosis, experts often use different magnifications when viewing microscopic images, so multi-scale information in pathology images inevitably plays a key role in classification.

[0008] While existing multi-input branch networks or multi-feature extraction networks have shown good performance to a certain extent, their high model complexity places a heavy strain on computer performance. Due to limited data, most of these models exhibit some degree of overfitting. Existing methods do not explore the multi-scale features of different feature layers within a single model, or fail to fully utilize these effective multi-scale features, which can only lead to limited performance improvements. Furthermore, due to the diversity of lesion sizes, global detection is meaningful in cases where there are a large number of irrelevant features in the image that are detrimental to class prediction. However, general convolutional neural networks do not inherently possess the ability to detect such extensive contextual information. Summary of the Invention

[0009] In order to solve the above problems, the present invention proposes a breast pathology image recognition system based on context-aware multi-scale feature fusion. It enhances the correlation between features and categories by fusing multi-scale context information, and fuses the context information of different levels obtained by different branches twice to represent the comprehensive features of the image, realizing the aggregation of multi-scale context-aware information and improving the classification accuracy.

[0010] In order to achieve the above object, the present invention adopts the following technical solutions:

[0011] In a first aspect, the present invention provides a breast pathology image recognition system based on context-aware multi-scale feature fusion, comprising:

[0012] an image acquisition module configured to acquire breast tissue pathology images;

[0013] a multi-scale feature extraction module configured to extract multi-scale features of breast tissue pathology images;

[0014] The context-aware module is configured to extract extensive contextual information using a cross-attention mechanism on the feature maps at two scales to obtain a contextual view;

[0015] The multi-scale feature fusion module is configured to sequentially upsample the context views of the two scales from top to bottom and concatenate them with the context views of the adjacent lower layer. After dimensionality reduction, the output feature maps of the branches corresponding to each scale are resized and concatenated through global average pooling to obtain multi-scale fused features.

[0016] The image recognition module is configured to obtain image category prediction results based on multi-scale fusion features.

[0017] As an optional implementation, in the context perception module, the feature maps obtained at the top layer of the multi-scale feature extraction module and the feature maps of other scales are both subjected to dimensionality reduction processing; after the feature maps of other scales are subjected to dimensionality reduction processing, a cross-attention mechanism is used to extract extensive contextual information to obtain a context view.

[0018] As an optional implementation, in the context perception module, 1*1 convolution is used to perform dimensionality reduction processing on multi-scale features.

[0019] As an optional implementation, the cross attention mechanism includes:

[0020] Use 1*1 convolution on the input feature map to obtain feature maps Q, K and V;

[0021] Get the feature vector of feature map Q at any position in the spatial dimension;

[0022] Obtain the feature vector on the cross path extracted at any position in the K-space dimension of the feature map;

[0023] Obtain the correlation matrix of the two eigenvectors and obtain the attention feature map based on the correlation matrix;

[0024] The feature map V is aggregated with the attention feature map, and the context view is obtained based on the context information obtained by the aggregation operation and the input feature map.

[0025] As an optional implementation, the correlation matrix is: Among them, Q u is the feature vector of feature graph Q at any position u in the spatial dimension; Ω u is the feature vector on the cross path extracted at any position u in the feature map K space dimension, Ω i,u yes The i-th element of .

[0026] As an optional embodiment, the polymerization operation is: in, is the set of feature vectors on the cross path extracted at position u of the feature map V, Φ i,u is Φu The i-th element of H″ u is context information, A i,u is the attention feature map.

[0027] As an optional implementation, the context view acquisition process is: H′ u =H u +H″ u ; where H′ u is the context view, H u is the input feature map.

[0028] As an optional implementation, in the multi-scale feature fusion module, 1*1 convolution is used to reduce the dimension of the output feature map.

[0029] In a second aspect, the present invention provides an electronic device comprising a memory and a processor, and computer instructions stored in the memory and executed on the processor, wherein the computer instructions, when executed by the processor, perform a breast pathology image recognition method based on context-aware multi-scale feature fusion;

[0030] The breast pathology image recognition method comprises:

[0031] Acquire breast tissue pathology images;

[0032] Extract multi-scale features of breast tissue pathology images;

[0033] A cross-attention mechanism is used to extract extensive contextual information from the feature maps at two scales to obtain a contextual view.

[0034] After upsampling the context views of the two scales from top to bottom and concatenating them with the context views of the adjacent lower layer, the output feature maps of the branches corresponding to each scale are reduced in dimension, resized and concatenated through global average pooling to obtain multi-scale fused features.

[0035] The image category prediction result is obtained based on the multi-scale fusion features.

[0036] In a third aspect, the present invention provides a computer-readable storage medium for storing computer instructions, which, when executed by a processor, implements a breast pathology image recognition method based on context-aware multi-scale feature fusion;

[0037] The breast pathology image recognition method comprises:

[0038] Acquire breast tissue pathology images;

[0039] Extract multi-scale features of breast tissue pathology images;

[0040] A cross-attention mechanism is used to extract extensive contextual information from the feature maps at two scales to obtain a contextual view.

[0041] After upsampling the context views of the two scales from top to bottom and concatenating them with the context views of the adjacent lower layer, the output feature maps of the branches corresponding to each scale are reduced in dimension, resized and concatenated through global average pooling to obtain multi-scale fused features.

[0042] The image category prediction result is obtained based on the multi-scale fusion features.

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] This paper proposes a breast pathology image recognition system based on context-aware multi-scale feature fusion. It proposes a context-aware and multi-scale feature fusion module, introduces cross-attention to the input low-level feature map to capture the contextual relationship of tissue images, combines high-level local features to comprehensively describe the regularity of tissue images, upsamples the top-level feature map and fuses it with the adjacent large-scale feature map to maintain the spatial relationship. Finally, it integrates the feature maps of each branch to represent the characteristics of the breast tissue image, captures rich image information with less computational effort, realizes the aggregation of multi-scale context-aware information, and improves classification accuracy.

[0045] The present invention proposes a network model consisting of a multi-scale feature extraction and a context-aware multi-scale feature fusion module, introduces a self-attention mechanism to obtain a larger receptive field and context information, and obtains stronger feature representation capabilities, which is superior to general multi-scale feature fusion methods.

[0046] The present invention uses a multi-scale feature extraction module instead of an image pyramid to obtain multi-scale features of an image, reducing the input branches of the network. At the same time, the multi-scale feature fusion module is lightweight, effectively reducing the complexity of the model and improving the generalization ability of the network.

[0047] Advantages of additional aspects of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The accompanying drawings, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0049] Figure 1 This is a diagram showing the architecture of a breast pathology image recognition system based on context-aware multi-scale feature fusion provided in Example 1 of the present invention;

[0050] Figure 2A schematic diagram of the network layer processing flow of a breast pathology image recognition system based on context-aware multi-scale feature fusion provided in Example 1 of the present invention;

[0051] Figure 3 This is a diagram of the cross-attention mechanism architecture provided in Example 1 of the present invention;

[0052] Figure 4(a)-Figure 4(b) Loss function curves for the baseline network provided in Example 1 of the present invention and the network of this embodiment;

[0053] Figure 5(a)-Figure 5(c) The two-dimensional projection diagram of the original data, ResNet50 model and ResNet50+MSCA model GAP layer feature vector provided in Example 1 of the present invention. DETAILED DESCRIPTION

[0054] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0055] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used in the present invention have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0056] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0057] In the absence of conflict, the embodiments of the present invention and the features thereof may be combined with each other.

[0058] Example 1

[0059] Deep learning-based methods have, to a certain extent, overcome the limitations of traditional machine learning methods. The development of deep learning has brought opportunities for solving problems in the visual field, such as image classification, image segmentation, and object detection. These methods have been applied to various medical image analysis tasks, including ultrasound images, pathology images, and X-ray technology. Therefore, many researchers are using deep learning methods to address the challenges encountered in breast pathology image analysis.

[0060] Capturing contextual information is essential for understanding the association between tissue cells and their surrounding background in pathological images. Context-aware models divide high-resolution images into multiple regions to extract cellular-level features and then aggregate the features across all regions to extend the context to tissue-level regions. Similarly, a multi-level context- and uncertainty-aware model, MCUa, integrates contextual information into a DCNN learning framework to dynamically learn the spatial dependencies between regions in histological images. Given the self-attention mechanism's demonstrated promising global detection performance in image detection tasks, Chen et al. proposed the GasHis-Transformer model for gastric histopathology images. This model employed BoTNet-50 and Inception-V3 to extract global and local information from pathological images, respectively, and then performed classification based on the fused features. This model demonstrated excellent generalization on histopathology grading tasks. Fan et al. designed an attention-based deep learning architecture to capture the global structure of multiple objects in cell images, as well as the low-level details of each object. Merging these critical instance-level details with high-level features for final prediction yielded remarkable results. Experimental results from these studies demonstrate that contextual information plays a crucial role in identifying heterogeneous tissue structural abnormalities.

[0061] Equally crucial for accurate identification is the multi-scale nature of breast pathology images. This is also applicable to various detection tasks. During pathology analysis, experts often use different magnifications when examining microscope images. Elements in tissue images, from cells to tissue structures, serve as the basis for doctors' diagnoses. However, many studies focus solely on cellular-level features.

[0062] Early on, Araujo et al. emphasized that breast cancer histopathology image analysis must rely on extracting features related to the nucleus as well as features related to the entire tissue. They proposed a CNN architecture that integrates information from multiple histological scales. Because both partial and global features are crucial for classifying high-resolution histopathology images, Sitaula et al. fed the full image and a segmented local image into a pre-trained model to capture detailed information about the different structures and local areas in breast tissue microscopic images. Finally, a classifier was used to classify the fused features. To fully utilize the multi-scale information in images, Yang et al. first converted each image into three scales, cropped each of these different scales to the same size, and then applied three multi-scale spatial attention deep convolutional neural networks to adaptively transform the multi-scale pathology microscopic images. Finally, the three classification results were combined to generate image labels. This method exploited multi-scale and spatial attention-assisted contextual information to mimic human reading patterns of histopathology images. Similarly, Yang's team proposed another multi-scale convolutional neural network EMS-Net, which used images cropped and enhanced at three scales to fine-tune the pre-trained DenseNet-161, ResNet-152 and ResNet-101 respectively, and achieved the extraction of different and complementary visual features by selecting the optimal set of fine-tuned deep models.

[0063] As described in the background technology, the above-mentioned multi-input branch network or multi-feature extraction network has good performance to a certain extent, but the model complexity is high, which puts a great test on the performance of the computer. Due to limited data, most of these models will show some degree of overfitting. Existing methods do not study the multi-scale features of different feature layers of a single model itself, or do not fully utilize these effective multi-scale features, which can only lead to limited performance improvements. In addition, due to the diversity of lesion sizes, global detection is meaningful for situations where there are a large number of irrelevant features in the image that are not conducive to category prediction, while general convolutional neural networks themselves do not have such extensive contextual information detection capabilities.

[0064] Based on this, this embodiment provides a breast pathology image recognition system based on context-aware multi-scale feature fusion, which enhances the correlation between features and categories by fusing multi-scale context information, and fuses the context information of different levels obtained by different branches twice to represent the comprehensive features of the image, thereby predicting the classification results.

[0065] Specifically include:

[0066] an image acquisition module configured to acquire breast tissue pathology images;

[0067] a multi-scale feature extraction module configured to extract multi-scale features of breast tissue pathology images;

[0068] The context-aware module is configured to extract extensive contextual information using a cross-attention mechanism on the feature maps at two scales to obtain a contextual view;

[0069] The multi-scale feature fusion module is configured to sequentially upsample the context views of the two scales from top to bottom and concatenate them with the context views of the adjacent lower layer. After dimensionality reduction, the output feature maps of the branches corresponding to each scale are resized and concatenated through global average pooling to obtain multi-scale fused features.

[0070] The image recognition module is configured to obtain image category prediction results based on multi-scale fusion features.

[0071] In this embodiment, if Figure 1 As shown, the breast pathology image recognition system mainly includes: an image preprocessing module, a multi-scale feature extraction module and a context-aware multi-scale feature fusion (MSCA) module; among them, the context-aware multi-scale feature fusion module predicts the image category by obtaining the multi-scale context information of the breast tissue pathology image.

[0072] The image preprocessing module is configured to perform operations such as scaling, flipping, and color dithering on the breast tissue pathology image acquired by the image acquisition module, and then complete image enhancement for subsequent operations.

[0073] In this embodiment, if Figure 2 As shown in the figure, the multi-scale feature extraction module uses ResNet50 as the backbone network, and tests the effect of loading the proposed context-aware multi-scale feature fusion module on popular feature extraction networks such as pre-trained Vgg16, ResNet18 and DenseNet121.

[0074] Judging from the structure of the backbone network itself, these networks lack the ability to combine multi-scale features. To address this problem, this embodiment draws on the feature pyramid that has achieved significant results in the field of object detection and introduces a context-aware multi-scale feature fusion module, so that these networks can better play a role in this classification problem.

[0075] The multi-scale feature extraction module utilizes the multi-scale feature maps inherent in deep convolutional neural networks. This type of hierarchical feature extraction is designed to retain richer multi-layer features. It does not require multi-level scaling of the original image and replaces the feature pyramid, greatly reducing the amount of computation.

[0076] Compared with natural image datasets, the dataset of this embodiment is relatively small. Loading pre-trained parameters can help the model initialize and converge better. The multi-scale feature extraction module uses the corresponding three-layer feature maps of different networks as output. The specific convolutional layers are shown in Table 1.

[0077] Table 1. Number of convolutional layers in pre-trained networks

[0078]

[0079] In this embodiment, the context-aware multi-scale feature fusion module includes a context-aware module and a multi-scale feature fusion module; among them, the context-aware module adopts the Criss-Cross Attention mechanism (CCA) to extract extensive context information. The Criss-Cross Attention mechanism can capture the dependencies of all features on the cross path, replace dense connections with sparse iterative methods, save memory and improve computational efficiency, and achieve the goal of lightweight.

[0080] In this embodiment, the feature map obtained at the top layer of the multi-scale feature extraction module and the feature maps of other scales are All the feature maps are processed with dimensionality reduction, and the unified dimension is 64. After the feature maps of other scales are processed with dimensionality reduction, a cross attention mechanism is used to extract extensive context information to obtain the context view.

[0081] like Figure 3 As shown in Figure 2, the specific processing process of the cross attention mechanism includes:

[0082] Use 1*1 convolution on the input feature map to obtain feature maps Q, K and V;

[0083] The feature vector of any position u in the feature map Q space dimension is expressed as The feature vector on the cross path extracted at any position u in the feature map K space dimension is expressed as Where i is an element in [1,…,H+W-1];

[0084] Apply Affinity operation to two eigenvectors to obtain the correlation matrix; that is, obtain the correlation between each position u and its H+W-1 positions in the same row and column, and obtain a (H+W-1)*(W*H)-dimensional correlation matrix; Affinity operation is defined as:

[0085]

[0086] Among them, d i,u ∈D represents Q u and The correlation between them.

[0087] Perform softmax operation on the correlation matrix to obtain the attention feature map A;

[0088] The feature map Aggregation operation is performed on the attention feature map A to obtain the weighted cross region of each position on the feature map. Finally, the context information is added to the local feature H to enhance the pixel-by-pixel representation and obtain the context view H′.

[0089] Aggregation operations are expressed as:

[0090]

[0091] H′ u =H u +H″ u (3)

[0092] Among them, Φ i,u yes The i-th element of Φ u is the set of feature vectors on the cross path extracted at position u in the feature map V.

[0093] In this embodiment, the feature map obtained from the top layer and the context views of other scales are sequentially upsampled from high to low and cascaded with the context views of the adjacent lower layers;

[0094] Among them, compared with the downsampling operation, the upsampling operation retains the content of the feature map to the greatest extent and strengthens the relationship between adjacent features.

[0095] There are two main ways to fuse feature maps: superposition and splicing. Unlike splicing fusion, after superimposing feature maps, 3*3 convolution is required to eliminate the aliasing effect caused by upsampling. Cascading feature maps avoids this situation. While eliminating the aliasing effect, it reduces the required parameters and reduces the possibility of overfitting.

[0096] Therefore, the top-down multi-scale feature fusion method of this embodiment effectively combines the semantic and spatial information contained in each scale feature layer, thereby improving the classification performance of lesions.

[0097] In this embodiment, a second fusion is performed before classification, and the feature maps output by the three scale branches are reduced in dimension using 1*1 convolution to increase the interaction of cross-channel information and improve the expressive ability of the network; then the feature maps of the three branches are unified in size through global average pooling and spliced and fused on the channel as the multi-scale fusion feature vector that is finally input into the fully connected layer.

[0098] In the multi-scale feature fusion module, convolutional layers are combined with batch normalization (BN) to promote model convergence and improve network generalization performance. The multi-scale feature fusion module uses multi-level contextual information at different scales to enhance the comprehensive representation of breast tissue pathology images.

[0099] Since the trainable parameters in the network are related to the number of convolutional layers and the size of the convolution kernel in the network, the common 3*3 convolution kernel has nine times more parameters than the 1*1 convolution kernel. Therefore, in the multi-scale feature extraction of each branch, this embodiment chooses to use 1*1 convolution instead of downsampling operation to reduce the number of parameters. 1*1 convolution has significant advantages. It can fuse the spatial information between channels to achieve the effect of dimensionality reduction. At the same time, it deletes unnecessary redundant features in the process of dimensionality reduction and reduces the training parameters. In addition, the 1*1 convolution before the fully connected layer fixes the output dimension of each branch feature map to 4, so that after the feature vectors of the three branches are fused, it will not generate too many training parameters, thereby alleviating the overfitting phenomenon.

[0100] After obtaining the final multi-scale fusion features, this embodiment sets a fully connected layer to map the feature representation to the sample space and uses the softmax function to predict the image category. The softmax function is defined as:

[0101]

[0102]

[0103] The output of softmax is the probability distribution over K categories, and the final classification result y pred is the category with the largest probability, and the network is trained by minimizing the cross entropy loss obtained by the training set images with known labels, which is defined as follows:

[0104]

[0105] Among them, y i represents the true label of the image, Represents the Softmax probability of each of the K categories.

[0106] It is worth noting that an important reason for setting a fully connected layer for prediction of 4 neurons instead of using 1*1 convolution and global average pooling is that ImageNet and breast tissue pathology images are very different, and the dataset of this embodiment is small, so fine-tuning of the fully connected layer is required to ensure the transfer of model representation capabilities.

[0107] This example uses the 2018 ICIAR (International Conference on Image Analysis and Recognition) Breast Cancer Histopathology (BACH) dataset. This dataset contains 400 H&E-labeled breast histology microscopy images of four categories: normal, benign, carcinoma in situ, and invasive. Each category has 100 images of 2048 x 1536 pixels, all captured at 200x magnification.

[0108] Training convolutional neural networks on small datasets is prone to overfitting, so image augmentation is an essential step for training on small datasets. All original images were resized to 299 x 299 pixels. Data augmentation techniques such as 90°, 180°, and 270° rotations, as well as vertical and horizontal flips and shearing, were applied to generate 10 enhanced images for each image. Color jittering was applied to each image during training to increase dataset diversity and mitigate image bias caused by staining differences. This step replaces stain normalization. This data augmentation step conforms to the principles of histopathology image analysis.

[0109] This example trains a deep learning network in an end-to-end manner, using the AdamW optimizer to minimize cross-entropy loss. The batch size is set to 16, the learning rate is set to 0.00008, the learning rate decays by 10% every 10 training cycles, and the maximum epoch is set to 25. During the experiment, the dataset is evenly divided into five subsets, each containing 20 images from each of the four categories. One subset is taken at a time to form the test set, and the remaining 320 images are expanded tenfold through data augmentation as the training set.

[0110] The performance of distinguishing each type of image is measured by precision, sensitivity and F1-score, and the overall accuracy is calculated by the percentage of correctly classified images. In order to demonstrate the effectiveness of the method proposed in this embodiment, the performance of the baseline method and the improved method on the BACH dataset are compared. The experimental results of different methods are shown in Table 2. From the experimental results, it can be seen that the modification of each step has an effect on improving the performance of the model, and the final algorithm has achieved good results. First, a single ResNet50 is constructed to extract high-level local features. After fine-tuning the model, the average classification accuracy is 92%; then a multi-scale feature fusion module is added to the network to enable the network to have rich spatial information and high-level semantic information at different scales. This module is an MSCA module without the self-attention mechanism. It can be seen from Table 3 that the multi-scale feature fusion method performs better than the general multi-scale feature fusion method. Here, the general multi-scale feature fusion refers to the method of splicing the feature vectors obtained by performing 1*1 convolution and global average pooling operations on multiple feature layers. In another experiment, Criss-cross Attention is added to both branches of the multi-scale feature fusion module to capture richer contextual information. This improvement further improves the accuracy by 1.59%.

[0111] Table 2 Comparison of the impact of different steps on experimental results

[0112]

[0113] Table 3 Comparison of experimental results between the general feature fusion method and the multi-scale feature fusion method of this embodiment

[0114]

[0115] Transfer learning is an effective way to improve recognition accuracy when the dataset is small. However, the results obtained by directly fine-tuning some pre-trained networks often do not transfer well to different test sets, which often leads to a certain degree of overfitting. Using this dataset to train and fine-tune the baseline network, we can obtain the loss function curve shown in Figure 4(a). Obviously, the loss function curve of the test set fluctuates greatly, and the convergence is not ideal. By training the CNN designed in this embodiment, the loss function curve of the test set fluctuates relatively smoothly, and the loss gap between the training set and test set results is small, as shown in Figure 4(b).

[0116] Figure 5(a)-Figure 5(c)The two-dimensional projections of the GAP layer feature vectors of the original data, the ResNet50 model, and the ResNet50+MSCA model are shown, demonstrating the application of the t-Distributed Stochastic Neighbor Embedding (t-SNE) method for neural network feature visualization. After the model proposed in this embodiment extracts features from the data, the inter-class discrimination is significantly improved, demonstrating that the model proposed in this embodiment can effectively classify the data.

[0117] This embodiment explores the comparison of the effects before and after building the multi-scale feature fusion module on four networks to prove the portability of the MSCA module. The experimental results are shown in Table 4. First, these networks are fine-tuned to adapt to the four-category task of this embodiment. The Vgg network has three fully connected layers, which concentrates too many training parameters. The loading of the MSCA module abandons all the original fully connected layers of Vgg16, greatly reduces the degree of overfitting, and improves the classification ability of the network. Since directly fine-tuning Vgg16 using the data set of this embodiment will cause the network to be completely unable to converge, the results in Table 4 are obtained by replacing the last pooling layer and classification layer in the original Vgg16 with a global average pooling layer and a fully connected layer. Experiments have shown that the MSCA module can improve the classification performance of multiple baseline networks to a certain extent.

[0118] Table 4 Performance comparison of methods in similar studies

[0119]

[0120]

[0121] Using the same 400-image BACH dataset, with the same training and test set ratios, experimental results obtained using five-fold cross-validation were compared with similar studies. The experimental results are shown in Table 5. Compared with the network models used in these studies, the model of this embodiment uses only a single feature extraction network, eliminating the need for a complex experimental environment. Regarding the extraction of multi-scale information, this embodiment does not require the separate separation of images into multiple scales and multiple inputs. The proposed multi-scale feature fusion module can extract multi-level contextual information with minimal computational overhead.

[0122] Table 5 Performance comparison of methods in similar studies

[0123]

[0124] The goal of this embodiment is to develop an efficient classification model for the multi-scale features of tissues in pathological images. The results show that the model proposed in this embodiment is competitive. For general multi-scale feature fusion networks, the design only considers that feature maps of different scales have a certain effect on the classification results, while ignoring the problem that the top and bottom feature maps cannot have both spatial information and semantic information. Having rich spatial and semantic information at the same time can better describe the tissue morphological features at different scales of breast pathology images. There are also some studies that represent the multi-scale features of the same image by scaling the original image and cutting out areas of the same size from input images of different scales. These models usually require multiple feature extraction networks. Compared with these methods, the method of this embodiment reduces the complexity of the model and improves the training efficiency of the network.

[0125] This example proposes an MSCA module to extract multi-scale features from breast pathology images. This module captures contextual information at different levels of the image, focusing on tissue and cell nucleus features at different scales. This helps simulate the behavior of experts analyzing histopathology images, alleviates overfitting in the baseline network, and improves the generalization ability of the neural network. Results show that the attention-based multi-scale feature fusion method proposed in this example enriches the image information extracted by the model, thereby enhancing the neural network's ability to discern breast pathology images.

[0126] In further embodiments, there is also provided:

[0127] An electronic device comprising a memory and a processor, and computer instructions stored in the memory and executed on the processor, wherein when the computer instructions are executed by the processor, a method for recognizing breast pathology images based on context-aware multi-scale feature fusion is performed;

[0128] The breast pathology image recognition method comprises:

[0129] Acquire breast tissue pathology images;

[0130] Extract multi-scale features of breast tissue pathology images;

[0131] A cross-attention mechanism is used to extract extensive contextual information from the feature maps at two scales to obtain a contextual view.

[0132] After upsampling the context views of the two scales from top to bottom and concatenating them with the context views of the adjacent lower layer, the output feature maps of the branches corresponding to each scale are reduced in dimension, resized and concatenated through global average pooling to obtain multi-scale fused features.

[0133] The image category prediction result is obtained based on the multi-scale fusion features.

[0134] It should be understood that in this embodiment, the processor may be a central processing unit (CPU), or may be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), off-the-shelf field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0135] The memory may include a read-only memory and a random access memory, and provides instructions and data to the processor. A portion of the memory may also include a non-volatile random access memory. For example, the memory may also store information about the device type.

[0136] A computer-readable storage medium for storing computer instructions, wherein when the computer instructions are executed by a processor, a method for breast pathology image recognition based on context-aware multi-scale feature fusion is performed;

[0137] The breast pathology image recognition method comprises:

[0138] Acquire breast tissue pathology images;

[0139] Extract multi-scale features of breast tissue pathology images;

[0140] A cross-attention mechanism is used to extract extensive contextual information from the feature maps at two scales to obtain a contextual view.

[0141] After upsampling the context views of the two scales from top to bottom and concatenating them with the context views of the adjacent lower layer, the output feature maps of the branches corresponding to each scale are reduced in dimension, resized and concatenated through global average pooling to obtain multi-scale fused features.

[0142] The image category prediction result is obtained based on the multi-scale fusion features.

[0143] This method can be directly implemented by a hardware processor, or by a combination of hardware and software modules within the processor. The software module can be located in a storage medium well-known in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. The storage medium is located in the memory, and the processor reads the information in the memory and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, a detailed description is not given here.

[0144] Those skilled in the art will appreciate that the units, i.e., algorithm steps, of the various examples described in conjunction with this embodiment can be implemented using electronic hardware or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0145] Although the above describes the specific embodiments of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without any creative work are still within the scope of protection of the present invention.

Claims

1. A breast pathology image recognition system based on context-aware multi-scale feature fusion, characterized by: include: an image acquisition module configured to acquire breast tissue pathology images; a multi-scale feature extraction module configured to extract a multi-scale feature map of a breast tissue pathology image; The context-aware module is configured to extract context information using a cross-attention mechanism on multi-scale feature maps to obtain a context view; The multi-scale feature fusion module is configured to sequentially upsample the multi-scale context views from top to bottom and concatenate them with the adjacent lower-level context views. After dimensionality reduction, the output feature maps of the branches corresponding to each scale are resized and concatenated through global average pooling to obtain multi-scale fused features. The image recognition module is configured to obtain image category prediction results based on multi-scale fusion features.

2. The breast pathology image recognition system based on context-aware multi-scale feature fusion according to claim 1, characterized in that: In the context perception module, dimensionality reduction processing is performed on the feature maps obtained at the top layer of the multi-scale feature extraction module and the feature maps of other scales; After dimensionality reduction of feature maps at other scales, a cross-attention mechanism is used to extract contextual information to obtain a contextual view.

3. The breast pathology image recognition system based on context-aware multi-scale feature fusion according to claim 2, characterized in that: In the context perception module, 1*1 convolution is used to perform dimensionality reduction on the multi-scale feature map.

4. The breast pathology image recognition system based on context-aware multi-scale feature fusion according to claim 1, characterized in that: The cross attention mechanism includes: Use 1*1 convolution on the input feature map to obtain feature maps Q, K and V; Get the feature vector of feature map Q at any position in the spatial dimension; Obtain the feature vector on the cross path extracted at any position in the K-space dimension of the feature map; Obtain the correlation matrix of the two eigenvectors and obtain the attention feature map based on the correlation matrix; The feature map V is aggregated with the attention feature map, and the context view is obtained based on the context information obtained by the aggregation operation and the input feature map.

5. The breast pathology image recognition system based on context-aware multi-scale feature fusion according to claim 4, characterized in that: The correlation matrix is: Among them, Q u is the feature vector of feature graph Q at any position u in the spatial dimension; Ω u is the feature vector on the cross path extracted at any position u in the feature map K space dimension, Ω i,u is Ω u The i-th element of .

6. The breast pathology image recognition system based on context-aware multi-scale feature fusion according to claim 4, characterized in that: The polymerization operation is: Among them, Φ i,u yes The i-th element of Φ u is the set of feature vectors on the cross path extracted at position u of the feature map V, H″ u is context information, A i,u is the attention feature map.

7. The breast pathology image recognition system based on context-aware multi-scale feature fusion according to claim 6, characterized in that: The process of acquiring the context view is as follows: u =H u +H″ u ; where H′ u is the context view, H u is the input feature map.

8. The breast pathology image recognition system based on context-aware multi-scale feature fusion according to claim 1, characterized in that: In the multi-scale feature fusion module, 1*1 convolution is used to reduce the dimension of the output feature map.

9. An electronic device, characterized in that: The invention comprises a memory and a processor, and computer instructions stored in the memory and executed on the processor, wherein the computer instructions are executed by the processor to complete a breast pathology image recognition method based on context-aware multi-scale feature fusion; The breast pathology image recognition method comprises: Acquire breast tissue pathology images; Extract multi-scale feature maps from breast tissue pathology images; A cross-attention mechanism is used to extract context information from multi-scale feature maps to obtain a context view. After upsampling the multi-scale context views from top to bottom and concatenating them with the adjacent lower-level context views, the output feature maps of the branches corresponding to each scale are reduced in dimension, resized and concatenated through global average pooling to obtain multi-scale fused features. The image category prediction result is obtained based on the multi-scale fusion features.

10. A computer-readable storage medium, characterized in that used to store computer instructions, which, when executed by a processor, complete a breast pathology image recognition method based on context-aware multi-scale feature fusion; The breast pathology image recognition method comprises: Acquire breast tissue pathology images; Extract multi-scale feature maps from breast tissue pathology images; A cross-attention mechanism is used to extract context information from multi-scale feature maps to obtain a context view. After upsampling the multi-scale context views from top to bottom and concatenating them with the adjacent lower-level context views, the output feature maps of the branches corresponding to each scale are reduced in dimension, resized and concatenated through global average pooling to obtain multi-scale fused features. The image category prediction result is obtained based on the multi-scale fusion features.

Images