Mammary gland pathology picture detection method based on transfer learning
By adopting a multi-source field adaptive transfer learning network in breast pathological image detection, the existing CNN network has solved the problem of overfitting and data size in breast pathological image detection, and achieved better detection effect and feature extraction capabilities. .
Patent Information
- Application Number
- CN202410934799.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-12
- Publication Date
- 2024-11-08
AI Technical Summary
The existing CNN networks have problems of insufficient overfitting and generalization in breast pathological image detection, and are harsh in the size of the input data, resulting in unsatisfactory detection results.
Using a multi-source field adaptive transfer learning network based on transfer learning, the breast pathological pictures are mapped to the public feature space through a public feature extractor and a domain-specific feature extractor, and the composite loss function is used for classification to realize the heterogeneous data set. Processing and analysis.
It improves the detection effect of breast pathological pictures, enhances the representation ability and spatial correlation in the characteristic space, and improves the classification performance of breast pathological pictures.
Smart Images

Figure CN118918062A_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present invention relate to the field of image processing technology, and in particular to a breast pathology image detection method based on transfer learning. Background Art
[0002] Breast cancer, as the most common cancer after skin cancer, poses a huge threat to the health of women around the world. According to statistics from the World Health Organization, by the end of 2020, approximately 7.8 million women worldwide had been diagnosed with breast cancer. The increasing prevalence of breast cancer has led to an increase in mortality rates worldwide, and has become the second leading cause of threat to women's lives.
[0003] On the one hand, although CNN has achieved significant advantages in image processing, training CNN models is still challenging: since CNN is a data-intensive network, it means that a large amount of labeled (annotated) data is needed to train the network, otherwise the network will have problems such as overfitting and insufficient generalization. However, in most medical fields, annotated data sets are usually difficult to obtain and expensive to obtain. On the other hand, the existing CNN network has stringent requirements on the input data size: the input data size must be consistent. Therefore, taking medical images as an example, in order to obtain images of consistent size, the original images need to be scaled and other data preprocessing operations. These operations will introduce redundant information or lose useful information to a certain extent, which is not conducive to network learning. It can be seen that the existing CNN network is not ideal for breast pathology image detection. Summary of the invention
[0004] The embodiment of the present invention provides a breast pathology image detection method based on transfer learning, which can improve the detection effect of breast pathology images.
[0005] In a first aspect, an embodiment of the present invention provides a method for detecting breast pathology images based on transfer learning, comprising:
[0006] Obtain breast pathology images;
[0007] The breast pathology image is input into a multi-source domain adaptive transfer learning network for detection to obtain a detection result, wherein the backbone network of the multi-source domain adaptive transfer learning network is a ResNet101 network, and the multi-source domain adaptive transfer learning network includes a common feature extractor, a domain-specific feature extractor and a specific classifier, the common feature extractor is used to extract common features shared by all domains, the domain-specific feature extractor is used to map each pair of source domain and target domain to a specific feature space, and the specific classifier is used to classify the target sample according to the features extracted by the specific feature extractor.
[0008] In some embodiments, the common feature extractor includes a spatial transformer network and a ResNet model, wherein the spatial transformer network is responsible for performing spatial affine transformation and alignment operations of image features, and the ResNet model is used to extract and analyze image features adjusted by the spatial transformer network.
[0009] In some embodiments, the steps for implementing the multi-source domain adaptive transfer learning network are as follows:
[0010] The ResNet model is used to extract shared common features from source domain and target domain data;
[0011] Inputting the universal features into the domain-specific feature extractor to map them into a specific feature space, wherein the domain-specific feature extractor comprises a three-layer combined convolution module and a residual connection, the output of the combined convolution module comprises a layer of second-order convolution kernel, a layer of batch normalization, a layer of ReLU activation function and a spatial channel attention module, and the spatial channel attention module comprises a spatial attention module and a channel attention module;
[0012] A composite loss function is used to predict the target sample category to minimize the output difference of the specific classifier, wherein the composite loss function is obtained by combining smooth cross entropy loss, CORAL loss and feature difference loss between the specific classifiers.
[0013] In some embodiments, the spatial transformer network is implemented as follows:
[0014] The feature tensor corresponding to the breast pathology image is passed into the positioning network to predict the transformation parameters, wherein the transformation parameters include translation transformation parameters, scaling transformation parameters and rotation transformation parameters;
[0015] Constructing a deformation grid based on the transformation parameters of the positioning network to define the corresponding position of each pixel after the transformation;
[0016] Inputting the transformation parameters into a sampler, interpolating and extracting pixel values from the original input image using the deformed grid, and generating a new image that has undergone spatial transformation according to the pixel values;
[0017] The new image after spatial transformation is input into the next layer of the network for feature extraction.
[0018] In some embodiments, the domain-specific feature extractor learns multiple domain-invariant representations by mapping each pair of source domain and target domain data to multiple different feature spaces, and then uses the multiple domain-invariant representations to train the specific classifiers of multiple domains.
[0019] In some embodiments, the domain-specific feature extractor is implemented as follows:
[0020] The transformation parameters are input to the convolution layers conv1 and conv4 with 1*1 convolution kernels to obtain channel information;
[0021] The channel information is input into the convolution layer conv3 with a 1*1 convolution kernel, and the nonlinear features of the feature map are increased;
[0022] The nonlinear features are input into the convolution layer conv2 with a 3*3 convolution kernel to extract small features and fit complex functions;
[0023] The small features are input into the residual structure composed of the conv4 convolutional layer and the BatchNorm layer.
[0024] In some embodiments, the spatial attention module is implemented as follows:
[0025] The spatial attention module uses two consecutive convolutional layers to generate an attention weight matrix;
[0026] A 1×1 convolutional layer is used to perform channel compression and preliminarily screen out potential key features;
[0027] Perform nonlinear transformation on key features through ReLU activation function;
[0028] Restore it to a single channel through a 7×7 convolutional layer with appropriate padding to obtain a delicate spatial attention distribution;
[0029] The Sigmoid activation function is used to ensure that the attention weight range is between 0 and 1 to reflect each spatial position.
[0030] In some embodiments, the channel attention module is implemented as follows:
[0031] The channel attention module uses average pooling and maximum pooling operations to extract global average features and maximum response features from the input feature map respectively;
[0032] Inputting the global average feature and the maximum response feature into a double-layer shared multi-layer perceptron respectively;
[0033] The feature vectors of the global average feature and the maximum response feature processed by the two-layer shared multilayer perceptron are added element by element to obtain a merged feature;
[0034] The merged features are input into the Sigmoid activation function so that the output value is between 0 and 1, and a channel attention map is generated.
[0035] In some embodiments, the CORAL loss is represented as follows:
[0036]
[0037] Among them, let the source domain training sample D s ={x i},x∈R d , labelled L s ={y i},i∈{1,…,L}, target domain data D r ={u},u∈R d , d is the number of network outputs, the number of source domain samples and the number of target domain samples are n s and n r ,make Respectively represent the j-th dimension feature of the i-th source domain (target domain) sample, C s , C r represents the feature covariance matrix, represents the Froben i us norm.
[0038] In some embodiments, for this particular classifier, a classification loss is added using a smoothed cross entropy loss:
[0039]
[0040] Where p is the probability distribution predicted by the multi-source domain adaptive transfer learning network, p i is the predicted probability of the multi-source domain adaptive transfer learning network for the i-th category, q is the true probability distribution, and q i is the value corresponding to the i-th category in the true label, and K is the total number of categories. At the same time, label smoothing is introduced to replace the 1 in the original label with a slightly smaller probability value 1-ε, and the remaining ε will be evenly distributed to other categories, so that the probability of each non-true category becomes The following is obtained:
[0041]
[0042] Label smoothing is introduced to enable the multi-source domain adaptive transfer learning network to balance the probability of each classification category during the training process.
[0043] In a second aspect, an embodiment of the present invention further provides a breast pathology image detection device based on transfer learning, the device comprising:
[0044] An acquisition module is used to acquire breast pathology images;
[0045] The detection module is used to input the breast pathology image into a multi-source domain adaptive transfer learning network for detection to obtain a detection result, wherein the backbone network of the multi-source domain adaptive transfer learning network is a ResNet101 network, and the multi-source domain adaptive transfer learning network includes a common feature extractor, a domain-specific feature extractor and a specific classifier, the common feature extractor is used to extract common features shared by all domains, the domain-specific feature extractor is used to map each pair of source domain and target domain to a specific feature space, and the specific classifier is used to classify the target sample according to the features extracted by the specific feature extractor.
[0046] In a third aspect, an embodiment of the present invention further provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for detecting breast pathology images based on transfer learning as described in the first aspect is implemented.
[0047] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are used to execute the breast pathology image detection method based on transfer learning as described in the first aspect.
[0048] According to the method for detecting breast pathology images based on transfer learning provided by an embodiment of the present invention, a breast pathology image is obtained; the breast pathology image is input into a multi-source domain adaptive transfer learning network for detection to obtain a detection result, wherein the backbone network of the multi-source domain adaptive transfer learning network is a ResNet101 network, and the multi-source domain adaptive transfer learning network includes a common feature extractor, a domain-specific feature extractor, and a specific classifier, the common feature extractor is used to extract common features shared by all domains, the domain-specific feature extractor is used to map each pair of source domain and target domain to a specific feature space, and the specific classifier is used to classify the target sample according to the features extracted by the specific feature extractor. Based on the proposed transfer learning network, the present invention incorporates multi-source domain adaptive technology to realize the processing and analysis of heterogeneous data sets. First, a common feature extractor is designed, and the image is mapped to a common feature space by combining a spatial transformer network (STN) and ResNet101 to enhance the representation ability of pathological sections in the feature space and capture their spatial correlation. Secondly, a domain-specific feature extractor is designed and used to map the source domain and target domain data to multiple different feature spaces, learn multiple domain-invariant feature representations, and train multiple domain-specific classifiers to transfer prior knowledge between multiple source domains to improve the classification performance of the target domain. Based on this, the embodiment of the present invention can improve the detection effect of breast pathology images. BRIEF DESCRIPTION OF THE DRAWINGS
[0049] Figure 1A is a flow chart of a breast pathology image detection method based on transfer learning provided by an embodiment of the present invention;
[0050] Figure 1B is an architecture diagram of a multi-source domain adaptive transfer learning network provided by an embodiment of the present invention;
[0051] Figure 2 is a schematic diagram of a common feature extractor provided by an embodiment of the present invention;
[0052] Figure 3 is a schematic diagram of the operation process of a space transformer network provided by an embodiment of the present invention;
[0053] Figure 4 is a schematic diagram of a spatial attention module provided by an embodiment of the present invention;
[0054] Figure 5 is a schematic diagram of a channel attention module provided by an embodiment of the present invention;
[0055] Figure 6 It is a structural schematic diagram of a breast pathology image detection device based on transfer learning provided by an embodiment of the present invention;
[0056] Figure 7 is a schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0057] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.
[0058] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification and claims and the following drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.
[0059] In the embodiments of the present invention, the words "further", "exemplarily" or "optionally" are used to indicate examples, illustrations or descriptions, and should not be interpreted as being more preferred or more advantageous than other embodiments or designs. The use of the words "further", "exemplarily" or "optionally" is intended to present related concepts in a specific way.
[0060] In order to more conveniently describe the working principle of the embodiment of the present invention later, an introduction to relevant technical scenarios is first given below.
[0061] Breast cancer, as the most common cancer after skin cancer, poses a huge threat to the health of women around the world. According to statistics from the World Health Organization, by the end of 2020, approximately 7.8 million women worldwide had been diagnosed with breast cancer. The increasing prevalence of breast cancer has led to an increase in mortality rates worldwide, and has become the second leading cause of threat to women's lives.
[0062] On the one hand, although CNN has achieved significant advantages in image processing, training CNN models is still challenging: since CNN is a data-intensive network, it means that a large amount of labeled (annotated) data is needed to train the network, otherwise the network will have problems such as overfitting and insufficient generalization. However, in most medical fields, annotated data sets are usually difficult to obtain and expensive to obtain. On the other hand, the existing CNN network has stringent requirements on the input data size: the input data size must be consistent. Therefore, taking medical images as an example, in order to obtain images of consistent size, the original images need to be scaled and other data preprocessing operations. These operations will introduce redundant information or lose useful information to a certain extent, which is not conducive to network learning. It can be seen that the existing CNN network is not ideal for breast pathology image detection.
[0063] Based on this, the present invention provides a method for detecting breast pathology images based on transfer learning, by obtaining breast pathology images; inputting the breast pathology images into a multi-source domain adaptive transfer learning network for detection, and obtaining a detection result, wherein the backbone network of the multi-source domain adaptive transfer learning network is a ResNet101 network, and the multi-source domain adaptive transfer learning network includes a common feature extractor, a domain-specific feature extractor, and a specific classifier, the common feature extractor is used to extract common features shared by all domains, the domain-specific feature extractor is used to map each pair of source domain and target domain to a specific feature space, and the specific classifier is used to classify the target sample according to the features extracted by the specific feature extractor. Based on the proposed transfer learning network, the present invention incorporates multi-source domain adaptive technology to realize the processing and analysis of heterogeneous data sets. First, a common feature extractor is designed, and the image is mapped to a common feature space by combining a spatial transformer network (STN) and ResNet101 to enhance the representation ability of pathological sections in the feature space and capture their spatial correlation. Secondly, a domain-specific feature extractor is designed and used to map the source domain and target domain data to multiple different feature spaces, learn multiple domain-invariant feature representations, and train multiple domain-specific classifiers to transfer prior knowledge between multiple source domains to improve the classification performance of the target domain. Based on this, the embodiment of the present invention can improve the detection effect of breast pathology images.
[0064] The embodiments of the present invention are further described below in conjunction with the accompanying drawings.
[0065] like Figure 1A As shown, Figure 1A It is a flowchart of a breast pathology image detection method based on transfer learning provided by an embodiment of the present invention. The breast pathology image detection method based on transfer learning may include but is not limited to steps S101 to S102.
[0066] Step S101: Obtain breast pathology images;
[0067] Step S102: Input the breast pathology image into the multi-source domain adaptive transfer learning network for detection to obtain the detection result, wherein the backbone network of the multi-source domain adaptive transfer learning network is the ResNet101 network, and the multi-source domain adaptive transfer learning network includes a common feature extractor, a domain-specific feature extractor and a specific classifier. The common feature extractor is used to extract common features shared by all domains, the domain-specific feature extractor is used to map each pair of source domain and target domain to a specific feature space, and the specific classifier is used to classify the target sample according to the features extracted by the specific feature extractor.
[0068] It can be understood that the purpose of the present invention is to provide a breast cancer histological pathology slice transfer learning network based on multi-source domain adaptation, whose backbone network structure is the ResNet101 framework, and the main functional modules are the common feature extractor, the domain-specific feature extractor and the specific classifier. The common feature extractor is used to extract common features shared by all domains; the domain-specific feature extractor maps each pair of source domain and target domain to a specific feature space; the specific classifier classifies the target sample according to the features extracted by the specific feature extractor.
[0069] It can be understood that the present invention integrates multi-source domain adaptation technology on the basis of the transfer learning network to realize the processing and analysis of heterogeneous data sets. First, a common feature extractor is designed to map the image to a common feature space by combining the spatial transformer network (STN) and ResNet101 to enhance the representation ability of pathological sections in the feature space and capture their spatial correlation. Secondly, a domain-specific feature extractor is designed and used to map the source domain and target domain data to multiple different feature spaces respectively, learn multiple domain-invariant feature representations, and train multiple domain-specific classifiers to achieve the transfer of prior knowledge between multiple source domains to improve the classification performance of the target domain.
[0070] It is understandable that the overall architecture of the multi-source domain adaptive transfer learning network is as follows Figure 1BAs shown. The present invention adopts a common feature extractor, which aims to extract common feature representations from images of different fields and map these images from the original feature space to a shared feature space. This common feature extractor consists of two parts: first, the spatial transformer network (STN) is responsible for performing spatial affine transformation and alignment operations of the input image or feature to capture its spatial correlation; then, ResNet (a classic backbone network that is widely used and performs well in classification tasks) is used to deeply extract and analyze the features adjusted by STN. As shown in Figure 1, this process ensures that the present invention can effectively extract valuable common features from all domains, and the implementation steps are as follows:
[0071] Step 1: First, use the ResNet model to extract shared common features from the source domain and target domain data.
[0072] Step 2: These common features are input into a specially designed three-layer combined convolution module and a residual-connected domain-specific feature extractor to achieve mapping of a specific feature space. The output of each combined convolution core module consists of a layer of second-order convolution kernel, a layer of batch normalization (BatchNorm), a layer of ReLU activation function, and an SCA (spatial channel attention module).
[0073] Step 3: Use a composite loss function strategy to accurately predict the target sample category and minimize the classifier output difference. The total loss function is a combination of smooth cross entropy loss, CORAL_loss loss, and feature difference loss between classifiers.
[0074] It can be understood that the present invention divides slice preprocessing into two major steps: training data preprocessing and test data preprocessing.
[0075] In the preprocessing stage of training data, the present invention takes a series of steps to optimize the input data of the model. The steps of preprocessing training data are as follows:
[0076] Step 1: All images are resized to a uniform size of 256x256 pixels. This step aims to eliminate the differences caused by the inconsistency of the original image sizes.
[0077] Step 2: The image size is further adjusted to 224x224 pixels by random cropping, which ensures the consistency of data size and increases the diversity of the dataset, effectively reducing the model's dependence on specific image areas.
[0078] Step 3: The image is then randomly flipped horizontally to further enrich the variants of the dataset and force the model to learn symmetry and directional invariance.
[0079] Step 4: The pixel value range is scaled to between 0.0 and 1.0 to obtain the preprocessed image data.
[0080] To ensure the objectivity, accuracy and reliability of the evaluation results and truly reflect the performance of the model on unseen data, we deliberately avoided strategies such as data augmentation (such as rotation, flipping, etc.) in the test data preprocessing, in order to maintain the original state of the test set and prevent any bias introduced by artificially added transformations. The steps for test data preprocessing are as follows:
[0081] Step 1: All test images are uniformly scaled to 224x224 pixels, which matches the image size used after random cropping during the model training phase, ensuring the consistency of the test images with the training set in size.
[0082] Step 2: The pixel value range is scaled to between 0.0 and 1.0 to obtain the preprocessed image data.
[0083] It can be understood that the fundamental goal of the common feature extractor of the present invention is to extract common features across domains, which are crucial for revealing and bridging the differences between the source domain and the target domain. Figure 2 Flowchart of the common feature extractor. In order to achieve this goal and enhance its effectiveness, the present invention innovatively places STN at the front end to generate a more accurate initial feature map, providing an optimized starting point for subsequent deep network analysis. STN dynamically performs spatial adjustment of the input image by introducing a parameterized spatial transformation model, ensuring the invariance of the viewing angle, thereby optimizing the extraction quality of common features in the preprocessing stage. Following STN, the pre-trained ResNet101, as a powerful feature extraction engine, not only deepens the learning depth of the features, but also stabilizes the coherence of the information flow in the feature extraction process through its unique residual structure, overcoming the common training difficulties in deep networks. The combination of the two not only significantly improves the extraction effect of common features, but also greatly enhances the alignment accuracy of the source and target domain feature distributions, provides strong support for the effectiveness of transfer learning, and lays a solid foundation for subsequent in-depth research in this field.
[0084] It is understandable that the specific process of STN space converter is as follows Figure 3 As shown:
[0085] Step 1: Pass the feature tensor into the Localization Network to predict the transformation parameters, including translation, scaling, rotation, etc., to meet the subsequent processing requirements.
[0086] Step 2: Use the transformation parameters of the localization network to construct a deformation grid, defining the corresponding position of each pixel after the transformation.
[0087] Step 3: The transformation parameters are input to the sampler, which uses the deformed grid to interpolate pixel values from the original input image to generate a new image that has been spatially transformed.
[0088] Step 4: Input the transformed new image into the next layer of the network for further feature extraction.
[0089] It can be understood that the domain-specific feature extractor learns multiple domain-invariant representations by mapping each pair of source and target domain data to multiple different feature spaces. These domain-invariant representations are then used to train multiple domain-specific classifiers. This allows knowledge to be transferred between multiple source domains to improve the classification performance of the target domain.
[0090] The three convolutional layers of the sonnet main structure and the convolutional layers of the residual connection part have the following process steps:
[0091] Step 1: The feature transformation parameters are passed into the convolutional layers conv1 and conv4 with 1*1 convolution kernels, which do not change the size of the feature block, thereby realizing channel dimension downsampling, linear combination changes of information between channels, and realizing information interaction between channels.
[0092] Step 2: Send the channel information output in the previous step to the convolution layer conv3 with a 1*1 convolution kernel to realize the information interaction between channels and learn complex functions again, and increase the nonlinear characteristics of the feature map.
[0093] Step 3: Send the nonlinear features output from the previous step to the convolution layer conv2 with a 3*3 convolution kernel to extract small features and fit complex functions. The output size of the feature map can be obtained by the following formula:
[0094]
[0095] Where O is the output size, Indicates rounding down, i indicates the input size, p indicates the padding size, k indicates the convolution kernel size, and s indicates the stride. It is not difficult to conclude that when kernel is 3, padding and stride are both 1, the output size is equal to the input size.
[0096] Step 4: Send the fine features extracted in the previous step to the residual structure composed of the conv4 convolution layer and the BatchNorm layer. Skip 3 convolution layers and 3 SCA modules and connect to the ReLU activation function before the output to solve the problem of gradient disappearance and network degradation. Prevent important features from being forgotten and improve the model's ability to handle complexity.
[0097] It is understandable that for the spatial attention module, such as Figure 4Compared with the spatial attention in the original CBAM, the present invention adopts the following process steps:
[0098] Step 1: The spatial attention module uses two consecutive convolutional layers to generate the attention weight matrix. Initially, a 1×1 convolution is used to reduce the number of channels to
[0099] Step 2: Perform channel compression through a 1×1 convolution layer to preliminarily screen out potential key features;
[0100] Step 3: Perform nonlinear transformation on key features through ReLU activation function to improve the nonlinear ability of the model;
[0101] Step 4: Restore to a single channel through a 7×7 convolutional layer with appropriate padding to obtain a fine-grained spatial attention distribution.
[0102] Step 5: Use the Sigmoid activation function to ensure that the attention weight range is between 0 and 1, reflecting each spatial position.
[0103] It is understandable that if Figure 5 The channel attention module process, the channel attention mechanism of the present invention complements the spatial attention described in 3.4.1. The channel attention module focuses on screening out the most critical information for the current task in the channel dimension. Its design closely follows the efficient strategy proposed in CBAM, aiming to enhance the network's ability to recognize the importance of different feature channels, thereby promoting the model to learn more discriminative feature representations. The specific process steps of channel attention are as follows:
[0104] Step 1: The channel attention module uses average pooling and maximum pooling operations to extract the global average feature and the maximum response feature from the input feature map. This step generates two feature maps of size 1×1×C, which are respectively denoted as F c_avg and F c_max , which represent the average statistical information and saliency distribution of feature maps in the channel dimension.
[0105] Step 2: In order to further refine these features and give each channel different importance weights, F c_avg and F c_maxThey are input into a carefully designed two-layer shared multi-layer perceptron (MLP). The first layer has C / r neurons (r is the reduction rate, which is used to reduce computational complexity while maintaining key information), and uses ReLU as the activation function, which is designed to introduce nonlinearity and filter features; the second layer restores the original channel C. This design allows the model to generate a weighting coefficient for each channel separately, reflecting the contribution of the channel to the final prediction.
[0106] Step 3: F after the above shared MLP processing c_avg and F c_max The feature vectors of are added element-by-element, combining the advantages of the two pooling methods to form a comprehensive channel importance assessment.
[0107] Step 4: Merge features and pass through the Sigmoid activation function to ensure that the output value is between 0 and 1 to form a channel attention map M c . Each element represents the weight of the corresponding channel, reflecting its relative importance to other channels.
[0108] It is understandable that in the key links of the domain adaptation task, especially in the feature extractor alignment step, the main goal is to narrow the gap between the source domain and the target domain data distribution and ensure that the model can generalize to the unlabeled target domain data. To achieve this goal, the present invention adopts a second-order statistical feature matching strategy, which is embodied as CORAL Loss, as an efficient means to measure and reduce the gap between the source domain and the target domain feature representation. In implementation, the strategy focuses on coordinating the feature vector F obtained by the source domain through a specific self-network sonnet source and the target domain feature vector F processed by a specific sub-network sonnet target By optimizing the deviation of the two statistics, the present invention promotes the close alignment of feature spaces, thereby enhancing the generalization ability of the model on the target domain data.
[0109] Assume that the source domain training sample D s ={x i},x∈R d , labelled L s ={y i},i∈{1,…,L}, target domain data D r ={u},u∈R d , where d can be understood as the number of network outputs, and the number of source domain samples and target domain samples are n respectively. s and n r ,make They represent the j-th dimension features of the i-th source domain (target domain) sample. s , C rrepresents the feature covariance matrix.
[0110]
[0111] in, represents the Frobenius norm. By minimizing the CORAL loss, the present invention can make the feature covariance of the source domain and the target domain closer, thereby reducing the distribution difference between the domains. This helps to improve the performance of the domain adaptation task, so that the model trained on the source domain can better adapt to the data on the target domain.
[0112] It can be understood that C is a set of N domain-specific predictors A multi-output network composed of. Each predictor Cj is a softmax classifier that receives specific domain-invariant features after the domain-specific feature extractor H(F(x)) of the jth source domain. For each classifier, the present invention adds a classification loss using smoothed cross entropy:
[0113]
[0114] Where p is the probability distribution predicted by the model, p i is the model's predicted probability for the i-th category. q is the true probability distribution, q i is the value corresponding to the i-th category in the true label, and K is the total number of categories. At the same time, the present invention introduces a regularization method of label smoothing to prevent overfitting. Specifically, label smoothing replaces 1 in the original label with a slightly smaller probability value 1-ε, and the remaining ε will be evenly distributed to other categories, that is, the probability of each non-true category becomes This prevents the model from overconfidently setting the probability to 1 or 0, and reduces the probability gap between different categories:
[0115]
[0116] By introducing label smoothing, the model no longer seeks to adjust the prediction probability to an extreme value during training, but instead considers the possibility of each category in a more balanced way. This can reduce the impact of incorrect labels on the model and better adapt to a certain degree of category relevance, thereby improving the performance and generalization ability of the model.
[0117] It can be understood that the present invention designs an application method of a multi-source domain adaptive histological pathology section transfer learning network in breast biopsy diagnosis, wherein the common feature extractor in the network architecture combined with the spatial transformer network (STN) and the ResNet101 model module is the key technology of the present invention, which can enhance the representation ability of pathological sections in the feature space and capture their spatial correlation, and is used to map images to the common feature space.
[0118] It can be understood that in order to further extract the invariant features of different domains of breast cancer histological pathology images, the present invention uses second-order statistical features (CORAL loss) to estimate the difference between the two domains during the network model training process, and reduces the distribution difference between domains by minimizing the CORAL loss to achieve distribution alignment in a specific domain. The label smoothing method is introduced to prevent overfitting and improve the generalization ability of the model.
[0119] Based on this, compared with the existing CNN network, the present invention not only takes into account the characteristics of prior knowledge in various fields of histological pathology images, but also designs a spatial transformer network (STN) and ResNet101 model module to enhance the representation ability of pathological slices in feature space and capture their spatial correlation. Specifically, in pre-training, it can extract features from source domain and target domain data respectively, learn feature representations that are invariant in multiple domains, and thus transfer prior knowledge between different data sets to improve the generalization ability of the model. In addition, the present invention also uses composite loss to achieve distribution alignment and model classification loss convergence in specific fields, making it easier to explain and understand the decision-making process of the model in the medical field.
[0120] In addition, if Figure 6 As shown, one embodiment of the present invention further discloses a breast pathology image detection device based on transfer learning, the device comprising:
[0121] An acquisition module 110 is used to acquire breast pathology images;
[0122] The detection module 120 is used to input the breast pathology image into the multi-source domain adaptive transfer learning network for detection to obtain the detection result, wherein the backbone network of the multi-source domain adaptive transfer learning network is the ResNet101 network, and the multi-source domain adaptive transfer learning network includes a common feature extractor, a domain-specific feature extractor and a specific classifier. The common feature extractor is used to extract common features shared by all domains, the domain-specific feature extractor is used to map each pair of source domain and target domain to a specific feature space, and the specific classifier is used to classify the target sample according to the features extracted by the specific feature extractor.
[0123] The breast pathology image detection device based on transfer learning in an embodiment of the present invention is used to execute the breast pathology image detection method based on transfer learning in the above embodiment. Its specific processing process is the same as the breast pathology image detection method based on transfer learning in the above embodiment, and will not be repeated here.
[0124] In addition, if Figure 7As shown, an embodiment of the present invention also discloses an electronic device, including: at least one processor 210; at least one memory 220, for storing at least one program; when the at least one program is executed by the at least one processor 210, a breast pathology image detection method based on transfer learning as in any of the previous embodiments is implemented.
[0125] In addition, an embodiment of the present invention further discloses a computer-readable storage medium storing computer-executable instructions for executing the breast pathology image detection method based on transfer learning as in any of the previous embodiments.
[0126] The system architecture and application scenarios described in the embodiments of the present invention are intended to more clearly illustrate the technical solutions of the embodiments of the present invention, and do not constitute a limitation on the technical solutions provided by the embodiments of the present invention. Those skilled in the art will appreciate that with the evolution of the system architecture and the emergence of new application scenarios, the technical solutions provided by the embodiments of the present invention are equally applicable to similar technical problems.
[0127] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.
[0128] In hardware implementation, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some physical components or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or temporary medium). As known to those of ordinary skill in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically embodies computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.
[0129] The terms "component", "module", "system", etc. used in this specification are used to represent computer-related entities, hardware, firmware, a combination of hardware and software, software, or software in execution. For example, a component can be, but is not limited to, a process running on a processor, a processor, an object, an executable file, an execution thread, a program, or a computer. By way of illustration, both applications running on a computing device and a computing device can be components. One or more components may reside in a process or an execution thread, and a component may be located on a computer or distributed between two or more computers. In addition, these components may be executed from various computer-readable media having various data structures stored thereon. Components may communicate, for example, through local or remote processes based on signals having one or more data packets (e.g., data from two components interacting with another component between a local system, a distributed system, or a network, such as the Internet interacting with other systems through signals).
Claims
1. A breast pathology image detection method based on transfer learning, comprising: Obtain breast pathology images; The breast pathology image is input into a multi-source domain adaptive transfer learning network for detection to obtain a detection result, wherein the backbone network of the multi-source domain adaptive transfer learning network is a ResNet101 network, and the multi-source domain adaptive transfer learning network includes a common feature extractor, a domain-specific feature extractor and a specific classifier, the common feature extractor is used to extract common features shared by all domains, the domain-specific feature extractor is used to map each pair of source domain and target domain to a specific feature space, and the specific classifier is used to classify the target sample according to the features extracted by the specific feature extractor.
2. The method according to claim 1, characterized in that The common feature extractor includes a spatial transformer network and a ResNet model. The spatial transformer network is responsible for performing spatial affine transformation and alignment operations of image features, and the ResNet model is used to extract and analyze image features adjusted by the spatial transformer network.
3. The method according to claim 2, characterized in that The implementation steps of the multi-source domain adaptive transfer learning network are as follows: The ResNet model is used to extract shared common features from source domain and target domain data; Inputting the universal features into the domain-specific feature extractor to map them into a specific feature space, wherein the domain-specific feature extractor comprises a three-layer combined convolution module and a residual connection, the output of the combined convolution module comprises a layer of second-order convolution kernel, a layer of batch normalization, a layer of ReLU activation function and a spatial channel attention module, and the spatial channel attention module comprises a spatial attention module and a channel attention module; A composite loss function is used to predict the target sample category to minimize the output difference of the specific classifier, wherein the composite loss function is obtained by combining smooth cross entropy loss, CORAL loss and feature difference loss between the specific classifiers.
4. The method according to claim 2, characterized in that: The implementation steps of the spatial transformer network are as follows: The feature tensor corresponding to the breast pathology image is passed into the positioning network to predict the transformation parameters, wherein the transformation parameters include translation transformation parameters, scaling transformation parameters and rotation transformation parameters; Constructing a deformation grid based on the transformation parameters of the positioning network to define the corresponding position of each pixel after the transformation; Inputting the transformation parameters into a sampler, interpolating and extracting pixel values from the original input image using the deformed grid, and generating a new image that has undergone spatial transformation according to the pixel values; The new image after spatial transformation is input into the next layer of the network for feature extraction.
5. The method according to claim 2, characterized in that: The domain-specific feature extractor maps each pair of source domain and target domain data to multiple different feature spaces to learn multiple domain-invariant representations, and then uses the multiple domain-invariant representations to train the specific classifiers of multiple domains.
6. The method according to claim 5, characterized in that The implementation steps of the domain-specific feature extractor are as follows: The transformation parameters are input to the convolution layers conv1 and conv4 with 1*1 convolution kernels to obtain channel information; The channel information is input into the convolution layer conv3 with a 1*1 convolution kernel, and the nonlinear features of the feature map are increased; The nonlinear features are input into the convolution layer conv2 with a 3*3 convolution kernel to extract small features and fit complex functions; The small features are input into the residual structure composed of the conv4 convolutional layer and the BatchNorm layer.
7. The method according to claim 3, characterized in that The implementation steps of the spatial attention module are as follows: The spatial attention module uses two consecutive convolutional layers to generate an attention weight matrix; A 1×1 convolutional layer is used to perform channel compression and preliminarily screen out potential key features; Perform nonlinear transformation on key features through ReLU activation function; Restore it to a single channel through a 7×7 convolutional layer with appropriate padding to obtain a delicate spatial attention distribution; The Sigmoid activation function is used to ensure that the attention weight range is between 0 and 1 to reflect each spatial position.
8. The method according to claim 3, characterized in that The implementation steps of the channel attention module are as follows: The channel attention module uses average pooling and maximum pooling operations to extract global average features and maximum response features from the input feature map respectively; Inputting the global average feature and the maximum response feature into a double-layer shared multi-layer perceptron respectively; The feature vectors of the global average feature and the maximum response feature processed by the two-layer shared multilayer perceptron are added element by element to obtain a merged feature; The merged features are input into the Sigmoid activation function so that the output value is between 0 and 1, and a channel attention map is generated.
9. The method according to claim 3, characterized in that: The CORAL loss is expressed as follows: Among them, let the source domain training sample D s ={x i },x∈R d , labelled L s ={y i },i∈{1,…,L}, target domain data D r ={u},u∈R d , d is the number of network outputs, the number of source domain samples and the number of target domain samples are n s and n r ,make Respectively represent the j-th dimension feature of the i-th source domain (target domain) sample, C s , C r represents the feature covariance matrix, represents the Frobenius norm.
10. The method according to claim 3, characterized in that For this particular classifier, add a classification loss using the smoothed cross entropy loss: Where p is the probability distribution predicted by the multi-source domain adaptive transfer learning network, p i is the predicted probability of the multi-source domain adaptive transfer learning network for the i-th category, q is the true probability distribution, and q i is the value corresponding to the i-th category in the true label, and K is the total number of categories. At the same time, label smoothing is introduced to replace the 1 in the original label with a slightly smaller probability value 1-ε, and the remaining ε will be evenly distributed to other categories, so that the probability of each non-true category becomes The following is obtained: Label smoothing is introduced to enable the multi-source domain adaptive transfer learning network to balance the probability of each classification category during the training process.
Citation Information
Cited By
Gastric cancer artificial intelligence screening method based on plain-scan CT (Computed Tomography) image
CN121837278A
