Weakly Supervised Recognition Method for Pathological Images Based on Novelty Detection
By constructing a novelty detection module and a feature aggregation network, and using negative image features for modeling, the problem of difficulty in identifying small regions of interest in pathological images was solved, achieving highly accurate pathological image recognition and improving recognition efficiency and accuracy.
Patent Information
- Application Number
- CN202311750397.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-12-19
- Publication Date
- 2026-11-10
- Estimated Expiration
- 2043-12-19
AI Technical Summary
Existing technologies struggle to effectively identify small regions of interest in pathological images, and general supervised learning methods cannot be used, making the recognition task difficult.
A feature aggregation network based on novelty detection was designed. By constructing a novelty detection module and a feature aggregation network, negative image features were used to model and filter small image patches, thereby enhancing the network's ability to identify regions of interest.
It improves the accuracy and efficiency of pathological image recognition, reduces the proportion of normal tissue, significantly enhances the ability to identify regions of interest, and saves manpower and time costs.
Smart Images

Figure CN117788794B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the fields of medical image processing and computer vision, and relates to the use of a deep learning neural network framework to recognize whole-slide image (WSI) images. Specifically, it relates to a weakly supervised recognition method for pathological images based on novelty detection. Background Technology
[0002] Histopathological image analysis is hailed as the "gold standard" for cancer diagnosis. By digitizing histopathological or cytopathological slides, the details of an entire tissue section can be recorded. This conversion process generates massive amounts of image data, reaching terabytes in size, providing rich information for systematic image analysis of microscopic morphology. These analyses not only help physicians gain a deeper understanding of the patient's condition but also provide crucial support for subsequent treatment. However, even experienced pathologists may disagree when analyzing the same image, highlighting the need for robust computer-aided methods to improve consistency. While considerable work has been dedicated to solving the problem of pathological image recognition, it remains challenging. On one hand, pathological images are inherently large, making manual annotation of large numbers of images difficult, and general supervised learning methods cannot be used. On the other hand, regions of interest (ROIs) in pathological images are often very small, and the images contain a large proportion of unrelated cells and tissues, making it easy for key information in the ROIs to be obscured or masked by the surrounding background. This increases the difficulty of the recognition task, and whether the network can find the true ROIs and learn their features is one of the key challenges. For the problem of pathological image recognition, although there is no pixel-level fine annotation, it is known that negative images do not contain regions of interest. If a weakly supervised recognition method can be developed by effectively mining and utilizing negative images, the aforementioned challenges can be improved. This invention achieves this by proposing a novelty detection-based method, which assists the network in learning relevant key information by modeling the features of negative images.
[0003] With the widespread application of deep learning technology in image processing, deep neural networks have brought revolutionary impacts to the fields of image processing and computer vision. Many studies have proposed deep neural networks for pathological image recognition. For example, one study built a neural network specifically for pathological image recognition (Lu MY, Williamson DFK, Chen TY, et al. Data-efficient and weakly supervised computational pathology on whole-slide images[J]. Nature biomedical engineering, 2021, 5(6): 555-570.), and Ilse et al. (Ilse M, Tomczak J, Welling M. Attention-based deep multiple instance learning[C]. International conference on machine learning. PMLR, 2018: 2127-2136.) designed a feature aggregation method for weak supervision, which has been widely used in pathological image recognition tasks. Currently, numerous studies have proposed various algorithms and techniques to improve the accuracy and efficiency of pathological image recognition. These methods have shown great potential in cancer diagnosis, disease staging, and treatment effect evaluation. However, despite the significant progress made in these studies, most existing methods still face challenges in dealing with small but clinically significant regions of interest. They often fail to address the difficulties of small regions of interest on large images, so more work is needed to further improve this task.
[0004] To address the challenge of accurately identifying regions of interest (ROIs) in pathological images due to their small proportion, this invention designs a novelty detection module and a feature aggregation network. Based on the characteristics of negative images (which consist entirely of normal tissue) and positive images (which include both normal and abnormal tissue), and considering that the dataset contains significantly more normal tissue than abnormal tissue, our model effectively filters images by simulating negative images, reducing the proportion of normal tissue and thus significantly improving the ability to identify ROIs. This method ultimately achieves more accurate weakly supervised identification results on both a private and a public dataset. Summary of the Invention
[0005] This invention aims to overcome the shortcomings of existing technologies and provides a weakly supervised method for pathological image recognition based on novelty detection. The method designs a feature aggregation module and a novelty detection module, thereby constructing a feature aggregation network that integrates novelty detection for high-accuracy pathological image recognition. First, a neural network pre-trained on ImageNet is used to extract features from small image patches and save these features for subsequent network training. Then, a feature aggregation network is constructed and trained. Since the model pre-trained on natural images does not contain pathological knowledge, the feature fine-tuning part of the feature aggregation network can be regarded as a feature extractor containing rich semantic information of pathological images, used to obtain more discriminative features of pathological image patches. These features are used to train a negative image model in the novelty detection module to achieve the filtering of image patches. Through the combined effect of these two modules, the accuracy of pathological image recognition is effectively improved.
[0006] The specific technical solution of this invention is as follows:
[0007] A weakly supervised identification method for pathological images based on novelty detection includes the following steps:
[0008] 1) Prepare initial data: Initial data includes normal (without abnormal tissue) and pathological images containing abnormal tissue.
[0009] 2) Data processing: The initial image obtained in step (1) is processed into pathological image patches. The features of each image patch are extracted using a pre-trained deep learning neural network for subsequent network training and testing.
[0010] 3) Construct a novelty detection and feature aggregation network: First, construct an aggregation network based on attention-weighted fusion, and then connect it to a novelty detection module.
[0011] 4) Training the novelty detection and feature aggregation network: Train the novelty detection and feature aggregation network obtained in step (3).
[0012] Step (3) involves constructing a novelty detection and feature aggregation network, including the following steps:
[0013] 3-1) Construct a feature aggregation network for image recognition, comprising a feature fine-tuning part, an image prediction branch, and an image patch prediction branch. The feature fine-tuning part optimizes the features of image patches to more effectively express pathological information. The image prediction branch calculates the attention score of each image patch and performs a weighted sum. The patch prediction branch predicts the label of the image patch. Specifically:
[0014] The input to the constructed feature aggregation network is the feature of a small image patch in an image. The feature fine-tuning part of the network consists of a fully connected layer. After the feature fine-tuning part, two branches are connected. The image prediction branch is used to predict the image label through the attention module and is composed of a fully connected layer. The image patch branch is only effective during the training phase and is used to predict the image patch label. It is composed of a fully connected layer. During the training process, pseudo-labels are set for some image patches to assist the network learning.
[0015] 3-2) Construct a novelty detection module for image patch selection. Use the support vector domain description algorithm to model negative image patches. This model will learn normal tissue information, specifically:
[0016] The input to the constructed novelty detection module is the output of the feature fine-tuning module in step 3-1). This feature has been integrated with pathological information. Features of small patches of images are randomly extracted from negative images, and the normal image patches are modeled using the support vector domain description algorithm. This model is saved for subsequent network training.
[0017] Step (4) involves training the novelty detection and feature aggregation network, including the following steps:
[0018] 4-1) The first stage of training does not include a feature aggregation network with a novelty detection module.
[0019] In the image prediction branch, a mini-batch training mode is used, and the average value of the class-balanced cross-entropy function is used as the loss function to measure the difference between the pathological image recognition results and the actual image labels.
[0020] In the image patch prediction branch, a subset of image patches are selected during training, and pseudo-labels are assigned to them. An unweighted cross-entropy function is used to measure the difference between the image patch recognition result and the image patch pseudo-label.
[0021] 4-2) The second stage of training uses the feature aggregation network trained in step (4-1). The output of the feature fine-tuning part in the feature aggregation network is used as the image feature. This feature contains rich semantic information of pathological image tissue and has initially shown separability. The negative image features are summarized, and image patches are randomly selected from them to train the novelty detection module and saved. In subsequent training, the similarity between each image patch and the negative model is evaluated. Based on the similarity, it is determined whether the image patch should participate in subsequent training to solve the problem of too many image patches affecting the training process.
[0022] 4-3) The third stage of training includes a feature aggregation network with a novelty detection module. After the feature fine-tuning module, the novelty detection module obtained in step (4-2) is used to perform online screening of image patches. The screened image patches are then fed into the subsequent two branches for further training. Through screening, the proportion of positive image patches is increased, enhancing the network's ability to perceive them.
[0023] Since the novelty detection model is trained on negative image patches, it learns a wealth of normal tissue information. However, abnormal tissues differ significantly from normal tissues. That is, normal image patches are closer to the novelty detection model, while abnormal image patches have a lower similarity to the novelty detection model. This difference can be used to rank the image patches. During training, image patches that are closer to the model can be filtered out to increase the proportion of abnormal image patches in the whole image. Furthermore, the selected image patches will be updated with each training round.
[0024] The beneficial effects of this invention are:
[0025] This invention, based on deep learning, constructs a feature aggregation network that integrates novelty detection. This network models the features of negative image patches and utilizes information hidden within normal tissue to rank image patches within an image. To address the problem of an excessive number of patches in an image, a reasonable filtering method is designed, enhancing the network's ability to identify regions of interest. This invention has the following characteristics:
[0026] 1. The system is easy to build, relying solely on pathological images and image-level labels, without the need for tedious and time-consuming pixel-level fine annotation, saving manpower and time costs and improving the practicality of the method;
[0027] 2. The method is simple and easy to implement;
[0028] 3. By designing a novelty detection module, the network learns a more suitable distribution of negative images for image patch screening, which enhances the network's discrimination ability. By designing a feature fusion network, the network's recognition ability is further improved.
[0029] 4. It significantly outperforms other methods on both private and public datasets. Attached Figure Description
[0030] Figure 1 This is a flowchart illustrating the specific implementation of the present invention.
[0031] Figure 2 This is a flowchart of the overall framework proposed in this invention. Detailed Implementation
[0032] The following provides a detailed explanation of the novelty detection-based weakly supervised identification method for pathological images proposed in this invention, with reference to examples and accompanying figures:
[0033] This invention constructs a feature aggregation network including a novelty detection module, trained using pathological images, and achieves high recognition accuracy in testing. The specific implementation process is as follows: Figure 1 As shown, the method includes the following steps:
[0034] 1) Collect and prepare initial data: Process pathological images to generate image patches.
[0035] 1-1) A publicly available pathological image cancer detection dataset and a private dataset for lung cancer biomarker identification were used. The publicly available dataset was Camelyon16 (Ehteshami Bejnordi B, Veta M, Johannes van Diest P, van Ginneken B, Karssemeijer N, Litjens G, van der Laak JAWM, and the CAMELYON16 Consortium. Diagnostic Assessment of Deep Learning Algorithms for Detection of Lymph Node Metastases in Women With Breast Cancer. JAMA. 2017; 318(22):2199-2210.). The Camelyon16 dataset contained 270 training images and 130 test images. The private dataset collected 401 pathological images for training and testing, of which 210 were positive images (i.e., containing lung cancer biomarkers) and 191 were negative images. The training, validation, and test sets were divided in a 3:1:1 ratio.
[0036] 1-2) For all pathological images in these two datasets, the foreground and background of the tissue region are separated by thresholding the images. Subsequent operations are only performed within the tissue region.
[0037] 1-3) For the Camelyon16 dataset, 512×512 image patches of non-overlapping size are extracted from the organized regions obtained in step (1-2) at a magnification of 20× for network training and testing. For the private dataset, 1792×1792 image patches of non-overlapping size are extracted from the organized regions obtained in step (1-2) at a magnification of 40× for network training and testing.
[0038] 2) Feature Extraction: For the Camelyon16 dataset, a pre-trained ResNet50 model (Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, Las Vegas, NV, USA, June 2016. IEEE.) was used as the feature extractor to encode image patches into feature representations for subsequent training and testing. For the private dataset, a pre-trained TinyViT model (WuK, Zhang J, Peng H, et al. Tinyvit: Fast pretraining distillation for small vision transformers[C]. European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022: 68-85.) was used as the feature extractor to encode image patches into feature representations for subsequent training and testing.
[0039] 3) Construction of novelty detection and feature aggregation network: Construct a novelty detection and feature aggregation network, which consists of two parts: first, an attention-based feature aggregation network, and then a novelty detection module connected to it. Figure 2 A network diagram is shown.
[0040] 3-1) The input of the feature aggregation network constructed in this invention is the feature of a small image patch in an image. The feature fine-tuning part of the network consists of a fully connected layer (the input dimension is the feature dimension of the image patch, and the output dimension is 512). Two branches are connected after the feature fine-tuning module. One branch is used to predict the image label through the attention module and is composed of a fully connected layer. The other branch is only effective during the training phase and is used to predict the image patch label. It is composed of a fully connected layer. For negative images, all image patches in the image are selected and their pseudo labels are set to 0. For positive images, the k image patches with higher prediction probabilities are selected and their pseudo labels are set to 1.
[0041] 3-2) The input of the novelty detection module constructed in this invention is the output of the feature fine-tuning module in step (3-1). The feature has been integrated with pathological information. A certain number of image patches are randomly selected from the negative image, and the negative image patches are modeled using the support vector domain description algorithm (Tax DMJ, Duin RP W. Support vector domain description[J]. Pattern recognition letters, 1999, 20(11-13): 1191-1199.). The model is saved for subsequent network training.
[0042] 4) Training the novelty detection and feature aggregation network: The novelty detection and feature aggregation network obtained in step (3) is trained step by step.
[0043] 4-1) In the first stage, a feature aggregation network without a novelty detection module is trained using mini-batch mode. In the image prediction branch, the average value of the class-balanced cross-entropy function is used as the loss function to measure the difference between the pathological image recognition result and the actual image label, as shown in the following formula:
[0044]
[0045] In the formula, L b The value of the class-balanced cross-entropy loss function is represented, N represents the number of images participating in training in this batch (N is 4 in this invention), and ∑ is the summation operator. m is the total number of positive images in the dataset, n is the total number of negative images in the dataset, and y mi For a positive image, y ni Let represent a negative image, α represent the proportion of negative images (labeled 0) in the entire dataset, and (1-α) represent the proportion of positive images (labeled 1) in the entire dataset. log(·) represents the logarithmic operation, and y′ represents the network's prediction result for the image.
[0046] In the image patch prediction branch, a subset of image patches are selected during training and labeled accordingly. For positive images, the k image patches with the highest predicted positive probability are selected and their pseudo-labels are set to 1. For negative images, all image patches are selected and their pseudo-labels are set to 0. An unweighted cross-entropy function is used to measure the difference between the image patch recognition result and the image patch pseudo-labels, as shown in the following formula:
[0047] L ins =-y ins log(y ins ′)-(1-y ins log(1-y) ins ′)
[0048] In the formula, L ins Let y represent the unweighted cross-entropy loss function. ins Represents pseudo-labels for small image patches, y ins ′ represents the network's prediction result for a small patch of image.
[0049] Total loss function L total as follows:
[0050] L total =βL b +(1-β)L ins
[0051] In the formula, β is a hyperparameter used to adjust the loss weight.
[0052] 4-2) After step (4-1), the second stage of training is carried out. The feature fine-tuning module in the feature aggregation network learns the semantic knowledge related to pathology. The image features output by this module contain rich semantic information of pathological image tissues. The negative image features are summarized, and 10,000 image patches are randomly selected from them for training of the novelty detection module (step (3-2)). The model is saved for subsequent training.
[0053] 4-3) In the third stage, the novelty detection model obtained in step (4-2) is used for online screening of image patches. Since the novelty detection model is trained on negative image patches, it has learned rich information about normal tissues. However, the difference between abnormal tissues and normal tissues is relatively large. That is, normal image patches are closer to the novelty detection model, while the similarity between abnormal image patches and the novelty detection model is smaller. This difference can be used to rank the image patches. During training, a% of the image patches that are closer to the model are removed to increase the proportion of abnormal image patches in the whole image. In this invention, a is 50%, that is, only half of the image patches will be used for training in the third stage, and the selected image patches will be updated with the number of training epochs. The loss function in step (4-1) is used to optimize the novelty detection and feature aggregation network until the loss value has no new minimum value in 25 consecutive training epochs, at which point training stops.
[0054] 5) Network Performance Testing: Using the test data from Camelyon16 and the private dataset obtained in step (1), the pathological image recognition network trained in step (4) was tested. The network was compared with other pathological image recognition methods by calculating three evaluation metrics: ACC (accuracy of recognition), AUC (area under the ROC curve), and F1 score. The test results of the method of this invention and its comparison with other methods are shown in Table 1.
[0055] Table 1 shows the results of the recognition method proposed in the paper (Ilse M, Tomczak J, Welling M. Attention-based deep multiple instance learning[C]. International conference on machine learning. PMLR, 2018: 2127-2136.), (d) and (e) are the results of the recognition method proposed in the paper (Lu MY, Williamson DFK, Chen TY, et al. Data-efficient and weakly supervised computational pathology on whole-slide images[J]. Nature biomedicalengineering, 2021, 5(6): 555-570.), and (f) is the result of the method of this invention.
[0056] Table 1 Comparison of results between the method of the present invention and other pathological image recognition methods.
[0057]
[0058] As shown in Table 1, the method of this invention outperforms existing pathological image recognition methods on both different datasets using three evaluation metrics. For the Camelyon16 dataset, the best results of the method of this invention are 4.74%, 2.33%, and 6.75% higher than other methods on the three metrics, respectively. For the private dataset, the best results of the method of this invention are 10.6%, 4.17%, and 5.12% higher than other methods on the three metrics, respectively.
Claims
1. A weakly supervised identification method for pathological images based on novelty detection, characterized in that, Includes the following steps: (1) Prepare initial data: Initial data includes pathological images of normal tissue and tissue containing abnormalities; (2) Data processing: The initial image obtained in step (1) is processed into pathological image patches, and the features of each image patch are extracted using a pre-trained deep learning neural network for subsequent network training and testing; (3) Construct a novelty detection and feature aggregation network: First, construct an aggregation network based on attention-weighted fusion, and then connect it to a novelty detection module; (4) Training the novelty detection and feature aggregation network: Training the novelty detection and feature aggregation network obtained in step (3); Step (3) involves constructing a novelty detection and feature aggregation network, including the following steps: 3-1) Construct a feature aggregation network for image recognition, comprising a feature fine-tuning part, an image prediction branch, and an image patch prediction branch. The feature fine-tuning part optimizes the features of image patches to more effectively represent pathological information. The image prediction branch calculates the attention score of each image patch and performs a weighted sum. The image patch prediction branch predicts the label of the image patch; specifically: The input to the constructed feature aggregation network is the feature of a small image patch in an image. The feature fine-tuning part of the network consists of a fully connected layer. After the feature fine-tuning part, two branches are connected. One image prediction branch is used to predict the image label through the attention module and is composed of a fully connected layer. The image patch branch is only effective during the training phase and is used to predict the image patch label. It is composed of a fully connected layer. During the training process, pseudo-labels are set for some image patches to assist the network learning. 3-2) Construct a novelty detection module for image patch selection. Use a support vector domain description algorithm to model negative image patches. This model will learn normal tissue information, specifically: The input to the constructed novelty detection module is the output of the feature fine-tuning module in step 3-1). This feature has been integrated with pathological information. The features of some small image patches are randomly extracted from the negative image, and the support vector domain description algorithm is used to model the normal image patches. The model is saved for subsequent network training. Step (4) involves training the novelty detection and feature aggregation network, including the following steps: 4-1) The first stage of training does not include a feature aggregation network with a novelty detection module; in the image prediction branch, a mini-batch training mode is used, and the average value of the class-balanced cross-entropy function is used as the loss function to measure the difference between the pathological image recognition result and the actual image label; in the image patch prediction branch, some image patches are selected during training, pseudo-labels are set for them, and an unweighted cross-entropy function is used to measure the difference between the image patch recognition result and the image patch pseudo-label. 4-2) The second stage of training uses the feature aggregation network trained in step (4-1), takes the output of the feature fine-tuning part of the feature aggregation network as the image feature, summarizes the negative image features, and randomly selects small image patches from them to train the novelty detection module. 4-3) The third stage of training includes a feature aggregation network with a novelty detection module; after the feature fine-tuning module, the novelty detection module obtained in step (4-2) is used to perform online screening of image patches, and the image patches retained after screening are sent to the next two branches for further training.
2. The weakly supervised identification method for pathological images based on novelty detection according to claim 1, characterized in that, In step 4-1), In the image prediction branch, a mini-batch training mode is used, and the average value of the class-balanced cross-entropy function is used as the loss function to measure the difference between the pathological image recognition result and the actual image label. The formula is as follows: In the formula, , Let represent the balanced cross-entropy loss function, N represent the number of images in this batch that participated in training, ∑ is the summation operator, m is the total number of positive images in the dataset, and n is the total number of negative images in the dataset. This is a positive image. This represents a negative image. This indicates the proportion of negative images in the entire dataset. This indicates the proportion of positive images in the entire dataset. Represents logarithmic operations. This represents the network's prediction result for the entire pathological image; In the image patch prediction branch, during training, a subset of image patches are selected and pseudo-labels are assigned to them. An unweighted cross-entropy function is used to measure the difference between the image patch recognition result and the pseudo-label, as shown in the following formula: In the formula, This represents the unweighted cross-entropy loss function. Indicates pseudo-labels for small image patches. This represents the network's prediction results for small patches of the image; Total loss function as follows: In the formula, It is a hyperparameter used to adjust the loss weights.
Citation Information
Patent Citations
Full-view digital pathological image classification system considering tumor microenvironment and construction method
CN115082743A
Meta RCNN-based few-sample target detection method
CN117036897A
