Breast cancer full-slice pathological image weak supervision classification method based on neighborhood aggregation graph network

Abstract

The invention discloses a breast cancer full-slice pathological image weak supervision classification method based on a neighborhood aggregation graph network, and the method comprises the steps: constructing a breast cancer full-slice pathological image data set, inputting segmented images into an FcaFormer network model, extracting instance features, and training the network model, screening out an instance with the highest malignant probability of each breast cancer full-slice pathological image, and storing the weight of the FcaFormer network model for the highest classification accuracy of the instances for feature coding of the instances; constructing an adjacent graph by the screened instance with the highest malignancy probability based on the position of the original image, and inputting the adjacent graph into a neighborhood aggregation graph network model to aggregate instance features; and training the neighborhood aggregation graph network model by using the training set, and performing benign and malignant classification on the breast cancer full-slice pathological image by using the trained neighborhood aggregation graph network model.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a weakly supervised classification method for breast cancer full-slice pathological images based on a neighborhood aggregation graph network. Background Art

[0002] According to the latest data released by the World Health Organization's International Agency for Research on Cancer (IARC) in 2022, breast cancer has become the second most common cancer in the world in terms of the number of new cases. Women are the main group of breast cancer patients and have a high mortality rate. Early diagnosis and treatment are crucial to the treatment of breast cancer patients. Traditional pathological diagnosis is based on pathologists looking at tissue pathological sections under a microscope to give results. With the development of digital pathology and whole-slice scanning technology, digital whole-slice pathology image (WSI) diagnosis has become the mainstream method. In order to improve the efficiency and accuracy of pathologists' cancer diagnosis, computer-assisted pathology image diagnosis research based on deep learning has received widespread attention in recent years and has become a research hotspot in the field.

[0003] Unlike natural images, WSI images have ultra-high resolution and cannot be directly trained by deep neural networks. Most of the work involves dividing the entire WSI into small pieces for analysis, but this requires manual annotation by experts, which is limited when the number of images is large. In addition, most WSIs only have labels for the entire image. Multiple instance learning (MIL), as a representative weakly supervised learning method, divides WSI into smaller pieces called bag instances. Each bag is associated with a label instead of a separate instance. The instances are then passed through a feature extractor to form a bag-level representation of the WSI. This can not only mine fine-grained information in pathological images, but also reduce the cost of manual annotation, becoming an important method for breast cancer WSI classification. Summary of the invention

[0004] In view of the problems existing in the prior art, the present invention discloses a weakly supervised classification method for breast cancer full-slice pathological images based on a neighborhood aggregation graph network, which specifically includes the following steps:

[0005] Construct a breast cancer full-slice pathology image dataset, divide the dataset into a training set and a test set, use preprocessing operations to remove the background area of ​​breast cancer full-slice pathology images, and segment each preprocessed image into multiple instances;

[0006] The segmented images are input into the FcaFormer network model to extract instance features and train the network model to screen out the instance with the highest probability of malignancy in each breast cancer full-slice pathology image, and the weight with the highest instance classification accuracy of the FcaFormer network model is saved for feature encoding of the instance;

[0007] The screened instances with the highest probability of malignancy are used to construct an adjacency graph based on the location of the original image, and the adjacency graph is input into the neighborhood aggregation graph network model to aggregate instance features;

[0008] The training set is used to train the neighborhood aggregation graph network model, and the trained neighborhood aggregation graph network model is used to classify benign and malignant breast cancer whole-slice pathology images.

[0009] Furthermore, Gaussian blurring and anti-binary thresholding were used to preprocess the breast cancer full-slice pathology images in the training set and the test set, remove the background area outside the cell tissue, and segment them into instances of a certain pixel size. Labels were set on each instance, and the instances were input into the FcaFormer network model for training and prediction of the malignancy probability of each instance. The label setting method is as follows:

[0010]

[0011] where y ij is the jth instance label in the i-th WSI, Y i is the label of the entire breast cancer full-slice pathology image. If the label I of a full-slice pathology image is 0, then all instance labels are 0, otherwise all instance labels are 1.

[0012] Further, when the FcaFormer network model is trained: the breast cancer full-slice pathology image is screened out and the instance is input into the model. The model includes two conventional convolution stages and two FcaFormer modules. The convolution stage consists of a convolution block and a downsampling layer. The downsampling uses deep convolution to reduce the resolution of the feature map by N times, increase the number of feature channels, and use a global average pooling operation before output. Then, the fully connected layer with an output dimension of one dimension is connected to obtain the classification result. The cross entropy loss function is used to optimize the model during training to save the optimal model weights. The loss function is as follows:

[0013] L cross =-(plogq+(1-p)log(1-q))

[0014] p is the probability of the first category of the sample, q is the predicted probability of the first category, and the d-dimensional features after the global average pooling operation in this stage are used as the input features of the subsequent neighborhood aggregation graph network.

[0015] Furthermore, for each breast cancer full-slice pathology image, the n instances with the highest malignancy probability were screened out through the first-stage FcaFormer network model, and the adjacency graph of each breast cancer full-slice pathology image was constructed. The Euclidean distance between each node and the node outside itself was calculated for each node screened out, and the average value of the Euclidean distance was extracted. If the distance between the node itself and other nodes is less than the average value, the two nodes are connected to form a neighbor relationship and converted into an adjacency matrix to construct an adjacency graph:

[0016]

[0017] In the formula (x i ,y i ) is the coordinate of the node, n is the number of nodes, d is the average distance, d(v i ,v j ) is the Euclidean distance between two points, e ij Indicates the relationship between two points, 1 means they are connected by an edge, and 0 means they are not connected by an edge.

[0018] Furthermore, the FcaFormer network model of the first stage is used to obtain the feature vector of the instance, which is used to form a feature matrix, and then combined with the adjacency matrix and input into the neighborhood aggregation graph network model to complete the aggregation operation, and enter the second stage graph aggregation process: select a target node, aggregate the features of neighboring nodes at different distances to form a feature sequence, and the aggregation method is to aggregate node features through matrix multiplication of the adjacency matrix and the feature matrix. The aggregation formula is as follows:

[0019]

[0020] In the formula, represents the self-loop adjacency matrix, is the degree matrix, represents the normalized adjacency matrix, X is the feature matrix, and k is the distance between neighbor nodes;

[0021] The node features are input into the attention layer of the network model to use the multi-head self-attention mechanism to capture the hidden information in the target node feature sequence with different aggregation distances.

[0022] Due to the adoption of the above technical scheme, the present invention provides a weakly supervised classification method for full-slice pathological images of breast cancer based on a neighborhood aggregation graph network. The method introduces a cross-multi-head attention module in the instance selection stage, and obtains more discriminative instance feature expression results by modeling the long-distance relationship of breast cancer pathological image instances; and converts highly discriminative pathological image instances into graph structure data, and then introduces a neighborhood aggregation attention graph to construct neighbor feature sequences of different distances to effectively aggregate node features, thereby achieving high-precision classification of full-size pathological images of breast cancer. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0024] Figure 1 Flow chart of the method of the present invention

[0025] Figure 2 The workflow diagram of the FcaFormer network model in the present invention is

[0026] Figure 3 This is the workflow diagram of the neighborhood aggregation graph network model in the present invention. DETAILED DESCRIPTION

[0027] In order to make the technical solutions and advantages of the present invention more clear, the technical solutions in the embodiments of the present invention are clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention:

[0028] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0029] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0030] like Figure 1The method shown is a weakly supervised classification method for breast cancer full-slice pathology images based on a neighborhood aggregation graph network. This method uses an attention network as an instance filter and feature extractor to complete the first-stage selection of breast cancer full-slice pathology image instances and feature extraction. Then, the positions of the selected instances in the original image are connected into an adjacency graph, an adjacency matrix is ​​constructed, and the features extracted by the first-stage model are input into the second-stage neighborhood aggregation graph model to effectively aggregate the instance features and finally output the benign and malignant classification results of the pathology image.

[0031] Figure 2 This is the first stage of model training flow chart. First, the full slice image needs to be segmented into small blocks, namely instances. Then, the FcaFormer model is built. The segmented image is input into the model for a classification prediction. The K instances classified as having the highest probability of malignancy are input into the model again to complete the first training of the model. According to this step, the model is trained for multiple rounds and the model weight with the highest classification accuracy is saved.

[0032] Figure 3 This is the second-stage feature aggregation model graph. This part uses the model of the first stage to screen instances and extract features. The screened instances are constructed into an adjacency matrix and input into the neighborhood aggregation graph network model, i.e., the graph Transformer network, together with the instance features. The aggregation operation of instance features at different distances is completed according to the adjacency matrix, and the final benign and malignant classification of the full-slice pathology image is output.

[0033] S1: Process the whole breast pathology image to remove useless parts, specifically in the following way:

[0034] S11: The image is Gaussian blurred to smooth the image and remove small noise and irrelevant details. The Gaussian blur formula is as follows:

[0035]

[0036] S12: Perform an anti-binary thresholding operation on the image to distinguish between cell tissue and background areas.

[0037] S13: Most of the pixels of the whole-slice pathology images are higher than 100,000*100,000, and can be divided into multiple equal-sized blocks according to actual needs. For example, the image can be divided into 512×512 non-overlapping small blocks. Since the number after division is still too large, small blocks with a higher proportion of tissue can be retained according to needs, and small blocks of images with more than 40% of cell tissue can be retained.

[0038] S2 builds the FcaFormer network model, completes the training of the FcaFormer network model, screens the instances and extracts features. The specific method is as follows:

[0039] S21: After data preprocessing, the image needs to be resized to 256×256, then input into the model convolution module, enter the FcaFormer layer, and connect with the tokens output by the previous stage through multiple cross multi-head attention (CMHA) inputs, which facilitates the interaction of token information at different levels. The construction formula of the query (Q′), key (K′), and value (V′) tokens of the cross multi-head attention module is as follows:

[0040] Q′=W Q x l-1

[0041]

[0042] x l-1 is the output of the previous block, is a set of tokens output by the previous token merge enhancement block, W Q ,W K ,W V are the weight matrices for query, key, and value, respectively, and α l is a learnable scaling factor.

[0043] S22: After the last FcaFormer layer, the dimension of the feature vector of the instance can be obtained to be 480. A fully connected layer is connected to predict the malignancy probability. The k instances with the highest malignancy probability are selected to train the model, and the value range of k is 1-5.

[0044] S23: The k selected instances are labeled with the same label as the full-slice pathology image, and are input into the model again to train the model. The cross entropy loss function is used to update the model weights, and the optimal weights are saved.

[0045] S3 constructs the graph structure of breast cancer full-slice pathology images and the neighborhood aggregation graph network model, aggregates instance features and implements the classification of breast cancer full-slice pathology images. The specific method is as follows:

[0046] S31: Filter out the instances as nodes of the graph, and construct a corresponding undirected graph for each full-slice pathology image of breast cancer. The specific construction process is to calculate the Euclidean distance between each node and the nodes outside itself, and calculate the average of these distances. If the distance between the node itself and other nodes is less than the average, the two nodes are connected to form a neighbor relationship. The formula of the construction process is as follows:

[0047]

[0048] In the formula (x i ,y i ) is the coordinate of the node, the number of nodes is n = 30, d is the average distance, d(v i,v j ) is the Euclidean distance between two points, e ij Represents the relationship between two points.

[0049] S32: The model aggregation method is to select a target node and aggregate the features of neighboring nodes at different distances to form a feature sequence. The node feature aggregation method is similar to the aggregation operation of graph convolution. The node features are aggregated by matrix multiplication of the adjacency matrix and the feature matrix. The aggregation formula is as follows:

[0050]

[0051] In the formula, represents the self-loop adjacency matrix, is the degree matrix, represents the normalized adjacency matrix, X is the feature matrix X∈R 30×480 , k=6 selects a farthest neighbor distance of 6.

[0052] S33: The aggregated feature sequence S=(x 0 ,x 1 ,....,x 6 ) is connected to a linear projection layer, mapped to the hidden layer of the Transformer, and then connected to a fully connected layer after the Transformer to calculate the classification result through the Softmax function. The mapping process formula is as follows:

[0053] Z=[x 0 E;x 1 E;.....;x 6 E]

[0054] where E∈R 480×120 , Z∈R 7×120 .

[0055] S4 trains the neighborhood aggregation graph network model, iteratively updates the network parameters, tests the model, and saves the model weights with the best classification effect for breast cancer full-slice pathology images. The specific method is as follows:

[0056] S41: During the training of the first-stage model, the Adam optimizer was used to alleviate the gradient oscillation problem, with the learning rate set to 1e-4 and the weight decay set to 1e-5. The optimal weights of the first stage were used to process the data set, and the top 30 instances with the highest probability ranking in each WSI were selected as the input for the second stage.

[0057] S42: The 30 instances of each WSI are used as nodes of the graph. According to the composition method of S31, an undirected graph is constructed and input into the neighborhood aggregation graph model for training. The Adam optimizer is also used, the learning rate is set to 1e-4, and the weight decay is set to 1e-5. Finally, in the benign and malignant classification task of the public Camelyon dataset, the model achieved the highest classification accuracy of 96.9%, which is highly competitive compared with the existing full-slice pathology image classification method.

[0058] Furthermore, when training the neighborhood aggregation graph network model, the network hyperparameters were set, the number of training epochs was set to 100, the number of image inputs per batch size was set to 32, the Adam optimizer was used to optimize the model parameters, the neighbor node distance was set to 6, and the softmax function was used to output the benign and malignant classification results of the WSI image. The model weight with the highest classification accuracy was saved.

[0059] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A weakly supervised classification method for breast cancer whole-slice pathology images based on neighborhood aggregation graph network, characterized in that include: Construct a breast cancer full-slice pathology image dataset, divide the dataset into a training set and a test set, use preprocessing operations to remove the background area of ​​breast cancer full-slice pathology images, and segment each preprocessed image into multiple instances; The segmented images are input into the FcaFormer network model to extract instance features and train the network model to screen out the instance with the highest probability of malignancy in each breast cancer full-slice pathology image, and the weight with the highest instance classification accuracy of the FcaFormer network model is saved for feature encoding of the instance; The screened instances with the highest probability of malignancy are used to construct an adjacency graph based on the location of the original image, and the adjacency graph is input into the neighborhood aggregation graph network model to aggregate instance features; The training set is used to train the neighborhood aggregation graph network model, and the trained neighborhood aggregation graph network model is used to classify benign and malignant breast cancer whole-slice pathology images.

2. According to claim 1, a weakly supervised classification method for breast cancer whole-slice pathological images based on a neighborhood aggregation graph network is characterized by: Gaussian blur and anti-binary thresholding were used to preprocess the breast cancer full-slice pathology images in the training set and test set, remove the background area outside the cell tissue, and segment them into instances of a certain pixel size. Labels were set on each instance, and the instances were input into the FcaFormer network model for training and prediction of the malignancy probability of each instance. The label setting method is as follows: where y ij is the jth instance label in the i-th WSI, Y i is the label of the entire breast cancer full-slice pathology image. If the label I of a full-slice pathology image is 0, then all instance labels are 0, otherwise all instance labels are 1.

3. The weakly supervised classification method for breast cancer whole-slice pathological images based on neighborhood aggregation graph network according to claim 1 is characterized by: When training the FcaFormer network model: the breast cancer full-slice pathology images are screened out and then input into the model. The model includes two conventional convolution stages and two FcaFormer modules. The convolution stage consists of a convolution block and a downsampling layer. Downsampling uses deep convolution to reduce the resolution of the feature map by N times, increase the number of feature channels, and use a global average pooling operation before output. Then, the fully connected layer with an output dimension of one dimension is connected to obtain the classification result. The cross entropy loss function is used to optimize the model during training to save the optimal model weights. The loss function is as follows: L cross =-(plogq+(1-p)log(1-q)) p is the probability of the first category of the sample, q is the predicted probability of the first category, and the d-dimensional features after the global average pooling operation in this stage are used as the input features of the subsequent neighborhood aggregation graph network.

4. The weakly supervised classification method for breast cancer whole-slice pathological images based on neighborhood aggregation graph network according to claim 2 is characterized by: For each breast cancer full-slice pathology image, the n instances with the highest malignancy probability are screened out through the first-stage FcaFormer network model, and the adjacency graph of each breast cancer full-slice pathology image is constructed. The Euclidean distance between each node and the node outside itself is calculated for each node screened out, and the average value of the Euclidean distance is extracted. If the distance between the node itself and other nodes is less than the average value, the two nodes are connected to form a neighbor relationship and converted into an adjacency matrix to construct an adjacency graph: In the formula (x i ,y i ) is the coordinate of the node, n is the number of nodes, d is the average distance, d(v i ,v j ) is the Euclidean distance between two points, e ij Indicates the relationship between two points, 1 means they are connected by an edge, and 0 means they are not connected by an edge.

5. The weakly supervised classification method for breast cancer whole-slice pathological images based on neighborhood aggregation graph network according to claim 3 is characterized by: The first-stage FcaFormer network model is used to obtain the feature vector of the instance, which is used to form a feature matrix, and then combined with the adjacency matrix and input into the neighborhood aggregation graph network model to complete the aggregation operation, and then enter the second-stage graph aggregation process: select a target node, aggregate the features of neighboring nodes at different distances to form a feature sequence, and the aggregation method is to aggregate node features through matrix multiplication of the adjacency matrix and the feature matrix. The aggregation formula is as follows: In the formula, represents the self-loop adjacency matrix, is the degree matrix, represents the normalized adjacency matrix, X is the feature matrix, and k is the distance between neighbor nodes; The node features are input into the attention layer of the network model to use the multi-head self-attention mechanism to capture the hidden information in the target node feature sequence with different aggregation distances.

Images