A prediction method, system and storage medium for sentinel lymph node metastasis in breast cancer

Abstract

The present invention discloses a method, system and storage medium for predicting sentinel lymph node metastasis in breast cancer. The method includes: obtaining labeled WSIs as a training data set, and performing preprocessing to obtain an image patch set; constructing a WSI classification model; using the image patch set to pre-train a feature extractor to obtain a feature vector set; inputting the feature vector set into a prototype clustering module, and extracting multiple prototypes through clustering; dividing the sentinel lymph node WSI of breast cancer into image patches and inputting them into the feature extractor to extract image patch features; inputting the image patch features and prototypes into a feature fusion module for matching to generate a soft assignment histogram, and constructing a feature vector of the sentinel lymph node WSI of breast cancer; sending the feature vector of the sentinel lymph node WSI of breast cancer into a fully connected layer to obtain a WSI classification score and perform metastasis judgment. This method can better solve the problem of micro-metastasis recognition while maintaining accurate recognition of macro-metastasis, so as to accurately diagnose sentinel lymph node metastasis in breast cancer.

Description

Technical Field

[0001] The present invention belongs to the technical field of digital pathological image processing, and particularly relates to a method, a system and a storage medium for predicting sentinel lymph node metastasis in breast cancer. Background Art

[0002] Breast cancer is the main cancer in women. When consulting, pathologists need to observe tissue specimen slides stained with hematoxylin and eosin (H&E) under a microscope to diagnose sentinel lymph node metastasis in breast cancer. This is a tedious, cumbersome and error-prone process. In recent years, with the rise of computational pathology, H&E-stained slides are scanned using a digital scanner and stored as whole-slide digital pathology images (WSIs), and then computer vision algorithms, especially deep learning algorithms, are used to automatically analyze these WSIs to assist in the diagnosis of breast cancer, effectively improving the diagnostic process. Computational pathology has the advantages of high efficiency, objectivity and repeatability. However, when analyzing WSIs, computational pathology faces problems: firstly, WSIs have extremely high resolutions, usually with billions of pixels, making it impossible to directly input a complete WSI into a common convolutional neural network; secondly, since tumor tissues usually only account for a small part of the WSI, fine annotation of the tumor area is required for fully supervised deep learning, but the cost of such fine annotation is extremely high, especially in the case of a shortage of qualified pathologists.

[0003] To address the above problems, a relatively promising method is the multi-instance learning method. When only the label of the whole WSI is given as weak supervision information, a WSI (bag) is cut into many small image patches (instances) for processing, and then the information gap between the image patches and the WSI is bridged through the common instance-space (IS) paradigm or embedding-space (ES) paradigm. However, there are still other challenges when applying the multi-instance learning method: one is that WSIs usually exhibit significant inter-tumor heterogeneity, which means that the pathological characteristics vary greatly among patients, posing difficulties for multi-instance learning; the other is that clinically, sentinel lymph node metastasis in breast cancer is divided into macro-metastasis and micro-metastasis. The former refers to a tumor metastasis area with a diameter greater than 2 mm, and the latter refers to a tumor metastasis area with a diameter between 0.2 and 2 mm; since the tumor metastasis area of micro-metastasis is very small, it is also more difficult to identify micro-metastasis than macro-metastasis.

[0004] In the existing WSI classification, one of the most direct multi-instance learning methods is the max-pooling multi-instance learning (MAX-MIL), which is a multi-instance learning method in the IS paradigm. This method first predicts each image patch, and each image patch obtains a prediction score. Then, the image patch with the maximum prediction score is selected to represent the entire WSI, and the prediction result of this image patch is the prediction result of the WSI. Another attention-based multi-instance learning (ABMIL) is the most popular multi-instance learning method in WSI classification, which is a multi-instance learning method in the ES paradigm. By introducing a self-attention mechanism, it learns the weights of each image patch, and then fuses the features of each instance by weighted average. There is also a dual-stream multi-instance learning (DSMIL) based on self-supervised contrast learning, which proposes a novel dual-stream attention mechanism and also uses contrast learning to obtain an effective feature extractor. However, the above existing technologies have two disadvantages: one is that they cannot effectively identify micro-metastases; because micro-metastatic lesions are very small, the number of positive and negative image patches in the WSI is highly unbalanced; at the same time, the existing methods all use pre-trained feature extraction networks to extract the features of image patches, resulting in weak discriminative ability of image patches, so that the discriminative information in positive image patches will be covered by negative image patches during the fusion process, leading to incorrect predictions. The other is that the above existing technologies extract the features of each image patch through a convolutional neural network (CNN), then perform weighted combination on these features to obtain the features of the entire WSI, and then use them for the final prediction; but the features extracted by the CNN have no clear physical meaning, resulting in the inability to clarify the physical meaning of each dimension in the WSI features and lacking interpretability. Summary of the Invention

[0005] The main purpose of the present invention is to overcome the disadvantages and deficiencies of the prior art, and provide a prediction method, system and storage medium for breast cancer sentinel lymph node metastasis. While maintaining accurate recognition of macro-metastases, this method can better solve the problem of micro-metastasis recognition, so as to accurately diagnose breast cancer sentinel lymph node metastasis.

[0006] To achieve the above purpose, the present invention adopts the following technical solutions:

[0007] On the one hand, the present invention provides a prediction method for breast cancer sentinel lymph node metastasis, which is characterized by including the following steps:

[0008] Obtain the labeled WSI as the training data set, and perform preprocessing to obtain the image patch set;

[0009] Construct a WSI classification model, where the WSI classification model includes a feature extractor, a prototype clustering module, a feature fusion module and a fully connected layer;

[0010] Pre-train the feature extractor using an image patch set to obtain a set of feature vectors, and fix the parameters of the feature extractor;

[0011] Input the set of feature vectors into the prototype clustering module, and extract multiple prototypes through clustering;

[0012] After dividing the breast cancer sentinel lymph node WSI into image patches, input them into the feature extractor with fixed parameters to extract image patch features;

[0013] Match the image patch features and prototypes in the feature fusion module to generate a soft assignment histogram and construct the feature vector of the breast cancer sentinel lymph node WSI;

[0014] Send the feature vector of the breast cancer sentinel lymph node WSI into the fully connected layer to obtain the WSI classification score and perform metastasis judgment.

[0015] As a preferred technical solution, the training data set is expressed as:

[0016]

[0017] where X i represents the i-th labeled WSI, and |S| represents the number of labeled WSIs in the training data set S;

[0018] The label is expressed as Y ∈ {0, 1}. When Y = 1, it means that the breast cancer sentinel lymph node has metastasized; when Y = 0, it means that the breast cancer sentinel lymph node has not metastasized;

[0019] The preprocessing refers to binarizing the labeled WSIs in the training data set and then dividing them into multiple image patches with the same length and width, which is expressed as: where represents the n-th image patch after dividing the i-th labeled WSI, and |X i | represents the number of image patches obtained by dividing the i-th labeled WSI.

[0020] As a preferred technical solution, the feature extractor is constructed based on a convolutional neural network and pre-trained using the max-pooling multi-instance learning method to convert the input image patches into feature vectors, which is expressed as:

[0021]

[0022] where represents the feature vector of the n-th image patch after dividing the i-th labeled WSI, and g θ represents the pre-trained feature extractor with parameters θ;

[0023] The step of inputting the image patch set into the feature extractor to obtain a set of feature vectors is expressed as:

[0024] As a preferred technical solution, inputting the feature vector set into the prototype clustering module, and extracting multiple prototypes through clustering, specifically:

[0025] The prototype clustering module uses the AP clustering algorithm to cluster the feature vectors of the image patches in any WSI to obtain the first-stage clustering centers:

[0026]

[0027] Among them, represents the set of first-stage clustering centers obtained by performing AP clustering on the feature vectors of the image patches in X i , and M i represents the number of first-stage clustering centers;

[0028] Use the AP clustering algorithm to cluster the first-stage clustering centers to obtain the second-stage clustering centers as prototypes:

[0029]

[0030] Among them, represents the set of second-stage clustering centers obtained by performing AP clustering on the first-stage clustering centers, and M represents the number of second-stage clustering centers;

[0031] The similarity measure of the AP clustering algorithm is defined as:

[0032]

[0033] Among them, S ab represents the similarity measure value between the features of image patch a and image patch b, λ is a hyperparameter, is the feature of image patch a, is the feature of image patch b, and ‖·‖ F represents the Frobenius norm.

[0034] As a preferred technical solution, the breast cancer sentinel lymph node WSI is denoted as X; the extracted image patch features are denoted as

[0035] Inputting the image patch features and prototypes into the feature fusion module for matching to generate a soft assignment histogram, specifically:

[0036] Introduce a metric learning mechanism, use the learnable fully connected layer FC2 to map the image patch features to a new feature space, and measure the similarity between the nth image patch feature and the pth prototype through cosine similarity. The formula is:

[0037]

[0038] Among them, represents similarity, W2 is the parameter of the learnable fully connected layer FC2, is the transpose matrix of W2;

[0039] Generate a soft assignment histogram {h n} n based on the calculated similarity, where the x-axis represents the prototype and the y-axis represents the similarity.

[0040] As a preferred technical solution, the construction of the feature vector of the breast cancer sentinel lymph node WSI is specifically:

[0041] Introduce the TOP-K selection mechanism, and use the fully connected layer FC3 to assign a score r n to each image patch feature to quantify its correlation with the positive breast cancer sentinel lymph node. The formula is:

[0042]

[0043] where W3 is the parameter of the learnable fully connected layer FC3, is the transpose matrix of W3;

[0044] Sort the scores {r n} n in descending order, and select the top K most relevant image patch features I K ={n1,…,n K};

[0045] Use selective pooling to aggregate the similarity scores of the K most relevant image patch features to obtain the feature vector h = [h (1) ,…,h (M) of the breast cancer sentinel lymph node WSI,

[0046]

[0047] where M represents the dimension of the WSI feature vector, and h (m) represents the value of the m-th dimension of the WSI feature vector.

[0048] As a preferred technical solution, the acquisition of the WSI classification score is specifically:

[0049] Send the feature vector of the breast cancer sentinel lymph node WSI into the fully connected layer FC1, and calculate the WSI classification score:

[0050]

[0051] where W1 is the parameter of the fully connected layer FC1, and o is the score output by softmax. Denotes the transpose matrix of W1.

[0052] As a preferred technical solution, the loss function of the WSI classification model is:

[0053]

[0054] Where is the loss function of the fully connected layer FC1, is the loss function of the fully connected layer FC3, Y is the true WSI label, [o, 1 - o] T is the WSI classification score, r * = max n {r n} is the correlation score with the largest positive sentinel lymph node in breast cancer.

[0055] On the other hand, the present invention provides a prediction system for breast cancer sentinel lymph node metastasis, characterized by comprising a data acquisition module, a model construction module, a vector extraction module, a prototype extraction module, a feature extraction module, a feature vector construction module, and a classification score acquisition module;

[0056] The data acquisition module is used to acquire labeled WSIs as a training data set and perform preprocessing to obtain an image patch set;

[0057] The model construction module is used to construct a WSI classification model; the WSI classification model includes a feature extractor, a prototype clustering module, a feature fusion module, and a fully connected layer;

[0058] The vector extraction module is used to pre-train the feature extractor using the image patch set, obtain a feature vector set, and fix the parameters of the feature extractor;

[0059] The prototype extraction module is used to input the feature vector set into the prototype clustering module and extract multiple prototypes through clustering;

[0060] The feature extraction module is used to divide the breast cancer sentinel lymph node WSI into image patches and input them into the feature extractor with fixed parameters to extract image patch features;

[0061] The feature vector construction module is used to match the image patch features and prototypes with the feature fusion module to generate a soft assignment histogram and construct a feature vector of the breast cancer sentinel lymph node WSI;

[0062] The classification score acquisition module is used to send the feature vector of the breast cancer sentinel lymph node WSI into the fully connected layer to obtain the WSI classification score and perform metastasis judgment.

[0063] On the other hand, the present invention provides a computer-readable storage medium storing a program, characterized in that when the program is executed by a processor, it implements the above-mentioned method for predicting breast cancer sentinel lymph node metastasis.

[0064] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0065] 1. The prototype clustering module of the present invention extracts prototypes by using a two-stage unsupervised clustering method. In the first stage, clustering is performed inside the WSI, and in the second stage, clustering is performed between WSIs; in this way, prototypes in the entire pathological dataset can be automatically obtained without the need to specify the number of prototypes in advance. And since these prototypes are extracted from the entire pathological dataset, these prototypes represent typical pathological features in the pathological dataset, can effectively model the inter-tumor heterogeneity, directly capture meaningful pathological patterns, and represent the multimodal distribution of pathological data in the feature space; at the same time, by using these prototypes to construct the overall features of the WSI, the method of the present invention is more interpretable because each dimension in the feature vector of the WSI represents the frequency of a prototype appearing in the WSI.

[0066] 2. In order to perform better in identifying micrometastases, the present invention introduces a metric learning mechanism and uses a learnable fully connected layer to measure the similarity between patch features and prototypes, generating a soft assignment histogram; since the fully connected layer is learnable, a more discriminative metric space can be learned through training. After mapping the feature vectors of patches and prototypes to this metric space, the discriminative ability of patches can be enhanced to achieve better matching; at the same time, a TOP-K selection mechanism is introduced, using the fully connected layer to quantify the correlation with positive sentinel lymph nodes in breast cancer, and using selective pooling to obtain the feature vector of the sentinel lymph node WSI in breast cancer, excluding the interference of irrelevant patches and alleviating the problem of excessive negative patches. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0068] Figure 1 It is a flowchart of a method for predicting breast cancer sentinel lymph node metastasis in an embodiment of the present invention;

[0069] Figure 2 It is a schematic structural diagram of a WSI classification model in an embodiment of the present invention;

[0070] Figure 3Schematic diagram of the prototype clustering module in the embodiment of the present invention;

[0071] Figure 4 Schematic diagram of the feature fusion module in the embodiment of the present invention;

[0072] Figure 5 Structural diagram of a prediction system for breast cancer sentinel lymph node metastasis in the embodiment of the present invention;

[0073] Figure 6 Schematic diagram of the computer-readable storage medium in the embodiment of the present invention. Detailed implementation manners

[0074] In order to enable those skilled in the art of the present technology to better understand the solutions of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative efforts fall within the scope of protection of the present application.

[0075] Referring to "embodiment" in the present application means that the specific features, structures or characteristics described in conjunction with the embodiment may be included in at least one embodiment of the present application. The phrase appears in various positions in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment mutually exclusive with other embodiments. Those skilled in the art explicitly and implicitly understand that the embodiments described in the present application can be combined with other embodiments.

[0076] The present invention proposes a new weakly supervised method to achieve the prediction of breast cancer sentinel lymph node metastasis, called prototype-based multi-instance learning (PMIL), which is a multi-instance learning of the vocabulary-based (VS) paradigm; the VS paradigm is to first obtain a batch of prototypes, and then use these prototypes to fuse the features extracted from the image patches, and then use the fused features to predict the WSI.

[0077] In the present invention, a set of WSI with labels is given as the training data set The task is to learn a binary classifier Y = F(X) from, so as to be able to predict the metastasis of breast cancer sentinel lymph nodes from an input WSI. A CNN-based model is used. Since WSI has ultra-high resolution, it is impossible to directly input WSI into CNN. The common practice is to use multi-instance learning to cut a WSI (bag) into many small image patches (Example), when only the labels of the WSI are required without these example labels, a WSI classifier F(X) is learned by processing these image patches.

[0078] As Figure 1 shown, this embodiment provides a method for predicting sentinel lymph node metastasis in breast cancer, including the following steps:

[0079] S1. Obtain labeled WSI as the training data set, and perform preprocessing to obtain an image patch set;

[0080] Specifically, the training data set is expressed as:

[0081]

[0082] where X i represents the i-th labeled WSI, and |S| represents the number of labeled WSIs in the training data set S;

[0083] The label Y ∈ {0, 1} is a weakly supervised binary classification label. When Y = 1, it indicates that sentinel lymph node metastasis in breast cancer has occurred; when Y = 0, it indicates that sentinel lymph node metastasis in breast cancer has not occurred;

[0084] Perform binarization processing on the labeled WSIs in the training data set, and then divide them into multiple image patches with the same length and width, which is expressed as: where represents the n-th image patch after the i-th labeled WSI is divided, and |X i | represents the number of image patches into which the i-th labeled WSI is divided.

[0085] In this embodiment, at 20 times magnification, each labeled WSI in the training data set is divided into 2000 - 8000 image patches of size 256×256.

[0086] S2. Construct a WSI classification model, including a feature extractor, a prototype clustering module, a feature fusion module, and a fully connected layer;

[0087] As Figure 2 shown, the goal of this embodiment is to construct a WSI classification model F(X) to achieve the prediction of sentinel lymph node metastasis in breast cancer, where the feature extractor is expressed as g θ , which is used to extract the feature vectors of the input image patches; the prototype clustering module is expressed as PD, which is used to learn multiple prototypes to facilitate the modeling of tumor heterogeneity in pathological data; the feature fusion module is expressed as PSE, which is used to construct the feature vector of the entire WSI; the fully connected layer is expressed as FC1, which is used to output the classification score and predict the metastasis of sentinel lymph node in breast cancer.

[0088] S3. Use the set of image patches to pre-train the feature extractor to obtain a set of feature vectors, and fix the parameters of the feature extractor;

[0089] Specifically, the feature extractor in the present invention is constructed based on a convolutional neural network (CNN) and pre-trained using the maximum pooling multi-instance learning method (MAX_MIL). The input image patches are converted into feature vectors, expressed as:

[0090]

[0091] Where, represents the feature vector of the nth image patch after the division of the ith labeled WSI, and g θ represents the pre-trained feature extractor, and θ is the parameter of the pre-trained feature extractor;

[0092] Input the set of image patches into the feature extractor to obtain a set of feature vectors, expressed as:

[0093] S4. Input the set of feature vectors into the prototype clustering module, and extract multiple prototypes through clustering;

[0094] Since there are too many image patches in the entire training set, it is difficult to directly apply traditional clustering algorithms (such as K-means). Therefore, as Figure 3 shown, the prototype clustering PD module uses an unsupervised clustering method and divides into two stages to aggregate multiple prototypes:

[0095] The first stage is to perform clustering within the WSI (ISC), that is: the PD module uses the AP clustering algorithm to cluster the feature vectors of the image patches in any WSI to obtain the first-stage clustering centers:

[0096]

[0097] Where, represents the set of first-stage clustering centers obtained by performing AP clustering on the feature vectors of the image patches in X i , and M i represents the number of first-stage clustering centers;

[0098] The second stage is to perform clustering between WSIs (XSC), that is: use the AP clustering algorithm to cluster the first-stage clustering centers to obtain the second-stage clustering centers as prototypes:

[0099]

[0100] Where, represents the set of second-stage clustering centers obtained by performing AP clustering on the first-stage clustering centers, and M represents the number of second-stage clustering centers;

[0101] Since the unsupervised Affinity Propagation Clustering algorithm is adopted, the number of categories can be automatically determined. At the same time, this algorithm also requires a clear similarity measure between the features of two image patches (such as a and b). Therefore, the similarity measure of the AP clustering algorithm is defined as:

[0102]

[0103] where S ab represents the similarity measure value between the features of image patch a and image patch b, λ is a hyperparameter, is the feature of image patch a, is the feature of image patch b, and ‖·‖ F represents the Frobenius norm.

[0104] By capturing typical pathological patterns, the prototype is expected to represent semantic classes with large intra-class variances (positive sentinel lymph node metastasis and negative sentinel lymph node metastasis in this task) in a more detailed manner, enabling the WSI classification model F(X) to effectively model the inter-tumor heterogeneity in pathological data.

[0105] S5. After dividing the breast cancer sentinel lymph node WSI into image patches, input them into a feature extractor with fixed parameters to extract the image patch features;

[0106] S6. Match the image patch features and the prototype by inputting them into a feature fusion module to generate a soft assignment histogram and construct a feature vector of the breast cancer sentinel lymph node WSI;

[0107] Let the breast cancer sentinel lymph node WSI be represented as X, and the image patch features obtained by the feature extractor be represented as

[0108] Based on the prototype obtained in the PD module, a WSI feature vector is constructed for the input breast cancer sentinel lymph node WSI through the PSE module, specifically:

[0109] To enable the WSI classification model to adapt to the task of computational pathology, especially to identify micrometastases in breast cancer sentinel lymph node metastasis prediction, as Figure 4 shown, two mechanisms are introduced in the PSE module:

[0110] First, a metric learning mechanism is introduced; in existing VS paradigm MIL methods, a predefined similarity measure is usually used to match image patches and prototypes, such as cosine distance or Mahalanobis distance. However, the present invention introduces a metric learning mechanism to learn the similarity measure. Specifically:

[0111] The image patch features are mapped to a new feature space using the learnable fully connected layer FC2, and the similarity between the nth image patch feature and the pth prototype is measured by cosine similarity. The formula is as follows:

[0112]

[0113] where, denotes the similarity, W2 is the parameter of the learnable fully connected layer FC2, is the transpose matrix of W2;

[0114] A soft assignment histogram {h n} n is generated according to the calculated similarity, where the x-axis represents the prototypes and the y-axis represents the similarity.

[0115] Second, the TOP-K selection mechanism is introduced to select only the K image patches most relevant to the class of interest (sentinel lymph node positive) for aggregation. Specifically:

[0116] A score r n is assigned to each image patch feature using the fully connected layer FC3 to quantify its relevance to sentinel lymph node positive in breast cancer. The formula is as follows:

[0117]

[0118] where, W3 is the parameter of the learnable fully connected layer FC3, is the transpose matrix of W3;

[0119] The scores {r n} n are sorted in descending order, and the top K most relevant image patch features I K ={n1,…,n K} are selected;

[0120] The similarity scores of the K most relevant image patch features are aggregated using selective pooling to obtain the feature vector h = [h (1) ,…,h (M) of the sentinel lymph node WSI in breast cancer,

[0121]

[0122] where, M represents the dimension of the WSI feature vector, and h (m) represents the value of the mth dimension of the WSI feature vector.

[0123] S7. The feature vector of the sentinel lymph node WSI in breast cancer is fed into a fully connected layer to obtain the WSI classification score and perform metastasis judgment. The formula for calculating the WSI classification score is as follows:

[0124]

[0125] Among them, W1 is the parameter of the fully connected layer FC1, and o is the probability value output by softmax, with a range of [0, 1]; represents the transpose matrix of W1.

[0126] The calculated WSI classification score ranges from 0 to 1. When the WSI classification score is greater than or equal to 0.5, it is determined that the sentinel lymph node of breast cancer has metastasized; otherwise, it is determined that the sentinel lymph node of breast cancer has not metastasized.

[0127] Due to the extremely large number of image patches, it is computationally unaffordable to train g θ together with the remaining parameters in the network (because it requires parallel storage of numerous intermediate feature maps of all image patches for backpropagation); therefore, following the common practice: first, use max-pooling multiple instance learning (MAX-MIL) of the IS paradigm to pre-train the feature extractor g θ alone, and then fix the parameters of the feature extractor during the model training process. To learn the parameters {W1, W2, W3} in the WSI classification model, the following loss function is used:

[0128]

[0129] Among them, is the loss function of the fully connected layer FC1, which is the common cross-entropy loss function in binary classification; is the loss function of the fully connected layer FC3, Y is the true WSI label, [o, 1 - o] T is the WSI classification score, r * = max n {r n} is the correlation score with the maximum positive sentinel lymph node of breast cancer.

[0130] It should be noted that for the foregoing method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations. However, those skilled in the art should know that the present invention is not limited by the described action sequence, because according to the present invention, certain steps can be performed in other sequences or simultaneously.

[0131] Based on the same idea as the method for predicting sentinel lymph node metastasis in breast cancer in the above embodiments, the present invention also provides a system for predicting sentinel lymph node metastasis in breast cancer, which can be used to execute the above method for predicting sentinel lymph node metastasis in breast cancer. For the convenience of description, in the structural schematic diagram of an embodiment of the system for predicting sentinel lymph node metastasis in breast cancer, only the parts related to the embodiments of the present invention are shown. Those skilled in the art can understand that the illustrated structure does not constitute a limitation on the device, and it may include more or fewer components than those shown, or combine some components, or have different component arrangements.

[0132] As Figure 5 shown, another embodiment of the present invention provides a system for predicting sentinel lymph node metastasis in breast cancer, including the following several modules:

[0133] The data acquisition module is used to acquire the labeled WSI as the training data set and perform preprocessing to obtain the image patch set;

[0134] The model construction module is used to construct a WSI classification model; the WSI classification model includes a feature extractor, a prototype clustering module, a feature fusion module, and a fully connected layer;

[0135] The vector extraction module is used to pre-train the feature extractor using the image patch set to obtain a feature vector set and fix the parameters of the feature extractor;

[0136] The prototype extraction module is used to input the feature vector set into the prototype clustering module and extract multiple prototypes through clustering;

[0137] The feature extraction module is used to divide the image patches of the breast cancer sentinel lymph node WSI and input them into the feature extractor with fixed parameters to extract the image patch features;

[0138] The feature vector construction module is used to match the image patch features and the prototypes by inputting them into the feature fusion module to generate a soft assignment histogram and construct the feature vector of the breast cancer sentinel lymph node WSI;

[0139] The classification score acquisition module is used to send the feature vector of the breast cancer sentinel lymph node WSI into the fully connected layer to obtain the WSI classification score and perform metastasis judgment.

[0140] It should be noted that the system for predicting sentinel lymph node metastasis in breast cancer of the present invention corresponds one-to-one with the method for predicting sentinel lymph node metastasis in breast cancer of the present invention. The technical features and their beneficial effects described in the embodiments of the above method for predicting sentinel lymph node metastasis in breast cancer are applicable to the embodiments of the system for predicting sentinel lymph node metastasis in breast cancer. For the specific content, reference can be made to the description in the method embodiments of the present invention, which will not be repeated here. This is hereby declared.

[0141] In addition, in the implementation of a prediction system for sentinel lymph node metastasis in breast cancer in the above embodiments, the logical division of each program module is only for illustrative purposes. In practical applications, according to needs, for example, considering the configuration requirements of the corresponding hardware or the convenience of software implementation, the above functions can be assigned to different program modules to complete, that is, the internal structure of the prediction system for sentinel lymph node metastasis in breast cancer is divided into different program modules to complete all or part of the functions described above.

[0142] As Figure 6 shown, in one embodiment, a computer-readable storage medium is provided, storing a program in a memory. When the program is executed by a processor, the prediction method for sentinel lymph node metastasis in breast cancer as described above is implemented. Specifically:

[0143] Obtain labeled WSIs as a training data set, and perform preprocessing to obtain an image patch set;

[0144] Construct a WSI classification model, where the WSI classification model includes a feature extractor, a prototype clustering module, a feature fusion module, and a fully connected layer;

[0145] Use the image patch set to pre-train the feature extractor to obtain a feature vector set, and fix the parameters of the feature extractor;

[0146] Input the feature vector set into the prototype clustering module, and extract multiple prototypes through clustering;

[0147] After dividing the image patches of the sentinel lymph node WSI in breast cancer, input them into the feature extractor with fixed parameters to extract image patch features;

[0148] Match the image patch features and prototypes in the feature fusion module to generate a soft assignment histogram, and construct a feature vector of the sentinel lymph node WSI in breast cancer;

[0149] Send the feature vector of the sentinel lymph node WSI in breast cancer into the fully connected layer to obtain a WSI classification score and perform metastasis judgment.

[0150] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, storage, database, or other medium used in the embodiments provided in the present application can include non-volatile and / or volatile memories.

[0151] The technical features of the above embodiments can be combined arbitrarily. For the sake of concise description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope recorded in this specification.

[0152] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications made without departing from the spirit and principle of the present invention shall be equivalent replacement methods and are all included in the protection scope of the present invention.

Claims

1. A method for predicting sentinel lymph node metastasis in breast cancer, characterized in that, It includes the following steps: Obtain labeled WSIs as the training dataset, and perform preprocessing to obtain a set of image patches; the training dataset is represented as: Among them, X i represents the i-th labeled WSI, and |S| represents the number of labeled WSIs in the training dataset S; The label is represented as Y ∈ {0, 1}, where when Y = 1, it indicates that the sentinel lymph node of breast cancer has metastasized; when Y = 0, it indicates that the sentinel lymph node of breast cancer has not metastasized; The preprocessing refers to binarizing the labeled WSIs in the training dataset and then dividing them into multiple image patches with the same length and width, which is expressed as: Among them, represents the nth image patch after the division of the ith labeled WSI, and |X i | represents the number of patches obtained by dividing the ith labeled WSI into image patches; Construct a WSI classification model, which includes a feature extractor, a prototype clustering module, a feature fusion module, and a fully connected layer; Use the set of image patches to pre-train the feature extractor, obtain a set of feature vectors, and fix the parameters of the feature extractor; The feature extractor is constructed based on a convolutional neural network and is pre-trained using the max-pooling multi-instance learning method to convert the input image patches into feature vectors, represented as: Among them, represents the feature vector of the nth image patch after the ith labeled WSI is divided, and g θ represents the pre-trained feature extractor with parameters θ; Input the set of image patches into the feature extractor to obtain a set of feature vectors, denoted as: Input the set of feature vectors into the prototype clustering module, and extract multiple prototypes through clustering. Specifically: The prototype clustering module uses the AP clustering algorithm to cluster the feature vectors of the image patches in any WSI to obtain the first-stage clustering centers: Among them, represents the set of first-stage clustering centers obtained by performing AP clustering on the feature vectors of the image blocks in X i , and M i represents the number of first-stage clustering centers; Use the AP clustering algorithm to cluster the first-stage clustering centers to obtain the second-stage clustering centers as prototypes: Among them, represents the set of the second-stage clustering centers obtained by performing AP clustering on the first-stage clustering centers, and M represents the number of the second-stage clustering centers; The similarity measure of the AP clustering algorithm is defined as: Among them, S ab represents the similarity metric value between the features of image patch a and image patch b, λ is a hyperparameter, is the feature of image patch a, is the feature of image patch b, ‖·‖ F represents the Frobenius norm; After dividing the whole slide image (WSI) of breast cancer sentinel lymph nodes into image patches, input the image patch features into a feature extractor with fixed parameters; the WSI of breast cancer sentinel lymph nodes is denoted as X; the extracted image patch features are denoted as Input the image patch features and prototypes into the feature fusion module for matching, generate a soft assignment histogram, and construct the feature vector of the sentinel lymph node WSI of breast cancer; The generation of the soft assignment histogram is specifically: Introduce a metric learning mechanism, use the learnable fully connected layer FC2 to map the image patch features to a new feature space, and measure the similarity between the nth image patch feature and the pth prototype through cosine similarity. The formula is: Among them, represents similarity, and W2 is the parameter of the learnable fully connected layer FC2. is the transposed matrix of W2; Generate a soft assignment histogram {h n} n based on the calculated similarity, where the x-axis represents the prototypes and the y-axis represents the similarity; The construction of the feature vector of the sentinel lymph node WSI of breast cancer is specifically: Introduce the TOP-K selection mechanism and use the fully connected layer FC3 to assign a score r to each image patch feature n to quantify its correlation with positive sentinel lymph nodes in breast cancer. The formula is as follows: Among them, W3 is the parameter of the learnable fully connected layer FC3, which is the transposed matrix of W3; Sort the scores {r n} in descending order, and select the top K most relevant patch features I n = {n1, …, n K}; K ​ Use selective pooling to aggregate the similarity scores of the K most relevant patch features to obtain the feature vector h = [h (1) ,…,h (M) of the breast cancer sentinel lymph node WSI. where M represents the dimension of the WSI feature vector, and h (m) represents the value of the m-th dimension of the WSI feature vector; Send the feature vector of the sentinel lymph node WSI of breast cancer into the fully connected layer to obtain the WSI classification score and perform metastasis judgment.

2. A method for predicting sentinel lymph node metastasis in breast cancer according to claim 1, characterized in that, The obtaining of the WSI classification score is specifically: Send the feature vector of the sentinel lymph node WSI of breast cancer into the fully connected layer FC1, and calculate the WSI classification score: Among them, W1 is the parameter of the fully connected layer FC1, [o, 1 - o] T is the WSI classification score, where o is the score output by softmax, represents the transpose matrix of W1.

3. A method for predicting sentinel lymph node metastasis in breast cancer according to claim 2, characterized in that, The loss function of the WSI classification model is: Among them, is the loss function of FC1, is the loss function of FC3, Y is the true WSI label, and r * = max n {r n} is the correlation score with the largest positive sentinel lymph node in breast cancer.

4. A system for predicting sentinel lymph node metastasis in breast cancer, characterized in that, It includes a data acquisition module, a model construction module, a vector extraction module, a prototype extraction module, a feature extraction module, a feature vector construction module, and a classification score acquisition module; The data acquisition module is used to obtain labeled WSIs as the training dataset, and perform preprocessing to obtain a set of image patches; the training dataset is represented as: Among them, X i represents the i-th labeled WSI, and |S| represents the number of labeled WSIs in the training dataset S; The label is represented as Y ∈ {0, 1}, where when Y = 1, it indicates that the sentinel lymph node of breast cancer has metastasized; when Y = 0, it indicates that the sentinel lymph node of breast cancer has not metastasized; The preprocessing refers to binarizing the labeled WSI in the training dataset and then dividing it into multiple image patches with the same length and width, which is expressed as: where represents the nth image patch after the division of the ith labeled WSI, and |X i | represents the number of patches obtained by dividing the ith labeled WSI into patches; The model construction module is used to construct a WSI classification model; the WSI classification model includes a feature extractor, a prototype clustering module, a feature fusion module, and a fully connected layer; The vector extraction module is used to pre-train the feature extractor using the set of image patches, obtain a set of feature vectors, and fix the parameters of the feature extractor; the feature extractor is constructed based on a convolutional neural network and is pre-trained using the max-pooling multi-instance learning method to convert the input image patches into feature vectors, represented as: Among them, represents the feature vector of the nth image patch after the i-th labeled WSI is divided, and g θ represents the pre-trained feature extractor with parameters θ; Input the set of image patches into the feature extractor to obtain a set of feature vectors, expressed as: The prototype extraction module is used to input the feature vector set into the prototype clustering module, and extract multiple prototypes through clustering. Specifically: The prototype clustering module uses the AP clustering algorithm to cluster the feature vectors of the image patches in any WSI to obtain the first-stage clustering centers: Among them, represents the set of first-stage clustering centers obtained by performing AP clustering on the feature vectors of image blocks in X i , and M i represents the number of first-stage clustering centers; Use the AP clustering algorithm to cluster the first-stage clustering centers to obtain the second-stage clustering centers as prototypes: Among them, represents the set of second-stage clustering centers obtained by performing AP clustering on the first-stage clustering centers, and M represents the number of second-stage clustering centers; The similarity measure of the AP clustering algorithm is defined as: Among them, S ab represents the similarity metric value between the features of image patch a and image patch b, λ is a hyperparameter, is the feature of image patch a, is the feature of image patch b, ‖·‖ F represents the Frobenius norm; The feature extraction module is used to divide the breast cancer sentinel lymph node WSI into image patches and input them into a feature extractor with fixed parameters to extract the image patch features; the breast cancer sentinel lymph node WSI is denoted as X; the extracted image patch features are denoted as The feature vector construction module is used to match the image patch features and the prototype input features in the feature fusion module to generate a soft assignment histogram and construct the feature vector of the breast cancer sentinel lymph node WSI; the generation of the soft assignment histogram is specifically: Introduce a metric learning mechanism, use the learnable fully connected layer FC2 to map the image patch features to a new feature space, and measure the similarity between the nth image patch feature and the pth prototype through cosine similarity. The formula is: Among them, represents similarity, and W2 is the parameter of the learnable fully connected layer FC2, which is the transpose matrix of W2; Generate a soft assignment histogram {h n} n based on the calculated similarity, where the x-axis represents the prototypes and the y-axis represents the similarity; The construction of the feature vector of the breast cancer sentinel lymph node WSI is specifically: Introduce the TOP-K selection mechanism and use the fully connected layer FC3 to assign a score r to each image patch feature n to quantify its correlation with the positive sentinel lymph nodes in breast cancer. The formula is as follows: Among them, W3 is the parameter of the learnable fully connected layer FC3, which is the transposed matrix of W3; Sort the scores {r n} in descending order, and select the top K most relevant patch features I n = {n1, …, n K}; K ​ Use selective pooling to aggregate the similarity scores of the K most relevant patch features to obtain the feature vector h = [h (1) , …, h (M) of the breast cancer sentinel lymph node WSI. Among them, M represents the dimension of the WSI feature vector, and h (m) represents the value of the m-th dimension of the WSI feature vector; The classification score acquisition module is used to send the feature vector of the breast cancer sentinel lymph node WSI into the fully connected layer to obtain the WSI classification score and make a metastasis judgment.

5. A computer-readable storage medium storing a program, characterized in that, When the program is executed by the processor, it implements the method for predicting breast cancer sentinel lymph node metastasis according to any one of claims 1-3.

Images