Prediction method and system for curative effect of neoadjuvant chemotherapy of breast cancer

Abstract

The invention discloses a breast cancer neoadjuvant chemotherapy curative effect prediction method and system, and relates to the medical image processing and analysis technology, and the method comprises the steps: carrying out the Macenko dyeing normalization of a sample HE pathological image of a patient; extracting a tissue region from the normalized sample HE pathological image by using an HEST method; inputting the extracted tissue region into a Prov-GigaPath network, extracting a feature vector of a specified dimension under set parameters, and performing dimension reduction on the extracted feature vector; inputting the feature vectors subjected to dimension reduction into an ABMIL model, and training the feature vectors; and predicting the HE pathological image of the patient based on the trained ABMIL model, and visualizing the prediction result by using the attention score of the trained ABMIL model. According to the method provided by the invention, multi-level complex features such as cellular morphology and tissue structure in the HE image can be automatically mined, and the accuracy and reliability of prediction are greatly improved.

Description

Technical Field

[0001] The present application relates to the field of medical image processing and analysis technology, and in particular to a method and system for predicting the efficacy of neoadjuvant chemotherapy for breast cancer. Background Art

[0002] Current technologies for predicting the efficacy of neoadjuvant chemotherapy for breast cancer primarily employ traditional machine learning methods or CNN-based deep learning models. Traditional machine learning-based prediction methods combine hand-designed features (such as nuclear morphology and texture parameters) with shallow classifiers (such as support vector machines and random forests) for prediction. Relying on fully supervised learning, they require a large amount of pixel-level labeled data, which is costly and highly subjective. CNN-based deep learning models use pre-trained CNNs (such as ResNet) to extract local image features and aggregate features through global pooling or simple attention mechanisms. Due to the high resolution of HE images (gigapixels), the images need to be segmented into small tiles for processing.

[0003] The defects or problems of the existing technology include:

[0004] 1. Insufficient feature expression capability: Manual features or shallow CNNs cannot capture the complex cell morphology and tissue structure in HE images, and are prone to missing key information.

[0005] 2. Strong dependency on labeling: Fully supervised methods require pathologists to label a large number of regional-level labels. This is costly and inefficient, limiting the training scale and application of the model.

[0006] 3. Poor model generalization: Traditional prediction models do not incorporate pre-trained features of large models, and their performance on cross-institutional data is unstable. Summary of the Invention

[0007] The present invention provides a method and system for predicting the efficacy of neoadjuvant chemotherapy for breast cancer, which greatly improves the accuracy and reliability of the prediction.

[0008] This embodiment of the application proposes a method for predicting the efficacy of neoadjuvant chemotherapy for breast cancer, comprising:

[0009] Macenko staining normalization was performed on the patient's sample HE pathological images;

[0010] For the normalized sample HE pathological images, the HEST method is used to extract tissue regions;

[0011] Input the extracted tissue region into the Prov-GigaPath network, extract the feature vector of the specified dimension under the set parameters, and perform dimensionality reduction on the extracted feature vector;

[0012] Input the reduced feature vector into the ABMIL model and train it;

[0013] Predictions are made for HE pathology images of patients based on the trained ABMIL model, and the prediction results are visualized using the attention scores of the trained ABMIL model.

[0014] An embodiment of the present application further provides a system for predicting the efficacy of neoadjuvant chemotherapy for breast cancer, comprising a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the steps of the method for predicting the efficacy of neoadjuvant chemotherapy for breast cancer as described above are implemented.

[0015] The Prov-GigaPath model of this application can automatically mine multi-level complex features such as cell morphology and tissue structure in HE images through large-scale self-supervised pre-training. In some examples, the attention branch and gating mechanism in the ABMIL model further focus on key features, and combine with the LabelSmoothingCrossEntropy loss function to suppress overfitting, making the model more accurate in predicting the chemotherapy efficacy of different patients, greatly improving the accuracy and reliability of the prediction.

[0016] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Various other advantages and benefits will become apparent to those skilled in the art upon reading the detailed description of the preferred embodiment below. The accompanying drawings are for illustration purposes only and are not to be considered as limiting the present invention. The same reference symbols are used throughout the drawings to represent the same components. In the drawings:

[0018] Figure 1 This is a schematic diagram of the basic process of the method for predicting the efficacy of neoadjuvant chemotherapy for breast cancer in this embodiment;

[0019] Figure 2 This is a schematic diagram of the WSI tissue segmentation effect of the HEST method of this embodiment;

[0020] Figure 3 This is a schematic diagram of the image block division effect of the method for predicting the efficacy of neoadjuvant chemotherapy for breast cancer in this embodiment;

[0021] Figure 4 This is a diagram illustrating the visualization of the results of the method for predicting the efficacy of neoadjuvant chemotherapy for breast cancer according to this embodiment. DETAILED DESCRIPTION

[0022] Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0023] This application embodiment proposes a method for predicting the efficacy of neoadjuvant chemotherapy for breast cancer, such as Figure 1 Shown, including:

[0024] In step S101, the patient's sample HE pathology images are normalized using Macenko staining. During the HE staining process, factors such as experimental conditions and staining batches can lead to variations in image staining intensity and color distribution, affecting subsequent analysis. Therefore, the Macenko algorithm is used to separate and normalize HE pathology images collected from breast cancer patients. Based on statistical principles, the Macenko method eliminates staining variations by converting and normalizing the image's color space.

[0025] In step S102, the normalized sample HE pathology image is used to extract tissue regions using the HEST method. The HEST segmentation model is a DeepLabV3 model that is fine-tuned specifically for segmenting tissue from the background of whole slide pathology images (WSIs).

[0026] In step S103, the extracted tissue region is input into the Prov-GigaPath network, and feature vectors of a specified dimension are extracted under set parameters. Furthermore, the extracted feature vectors are subjected to dimensionality reduction. In some examples, dimensionality reduction can be performed using principal component analysis (PCA), a commonly used dimensionality reduction technique that uses linear transformations to map original high-dimensional data to a low-dimensional space while preserving the data's primary information as much as possible.

[0027] In step S104, the reduced feature vector is input into the ABMIL model and trained. In some embodiments, LabelSmoothingCrossEntropy is used as the loss function during model training, and optim.AdamW is used as the optimizer. Traditional cross-entropy loss functions can easily cause the model to be overconfident about labels during training. LabelSmoothingCrossEntropy smoothes the labels, which, to a certain extent, reduces overfitting.

[0028] In step S105, the HE pathological image of the patient to be predicted is predicted based on the trained ABMIL model, and the prediction result is visualized using the attention score of the trained ABMIL model. The visualization can intuitively display the key areas of the model's attention in visual ways such as color depth and transparency.

[0029] The Prov-GigaPath model of this application can automatically mine multi-level complex features such as cell morphology and tissue structure in HE images through large-scale self-supervised pre-training. In some examples, the attention branch and gating mechanism in the ABMIL model further focus on key features, and combine with the LabelSmoothingCrossEntropy loss function to suppress overfitting, making the model more accurate in predicting the chemotherapy efficacy of different patients, greatly improving the accuracy and reliability of the prediction.

[0030] During HE staining, factors such as different experimental conditions and staining batches can lead to differences in image staining intensity and color distribution, affecting subsequent analysis. Therefore, the Macenko algorithm is used to achieve staining separation and normalization for HE pathology images collected from breast cancer patients. In some embodiments, performing Macenko staining normalization on patient sample HE pathology images includes:

[0031] The sample HE pathology image is converted from RGB space to optical density OD space. In OD space, the color distribution is more consistent with the Gaussian distribution law.

[0032] In OD space, the staining vector of the transformed sample HE pathology image is calculated;

[0033] According to the coloring vector, two principal component vectors representing H coloring and E coloring are determined by singular value decomposition (SVD);

[0034] The staining vectors of the sample HE pathology images were normalized using the mean and standard deviation of the staining concentration of the reference image to ensure that subsequent feature extraction was not interfered by staining differences.

[0035] In some embodiments, extracting tissue regions using the HEST method from the normalized sample HE pathology image includes:

[0036] Use the HEST segmentation model to segment the HE image after dye normalization;

[0037] After segmentation, the tissue area containing effective information is extracted and background interference is removed. The effective information includes breast cancer cells. The HEST segmentation effect is as follows: Figure 2 shown.

[0038] The extracted tissue area is divided into blocks, and the WSI image is divided into non-overlapping image blocks of a specified size. As an example, the extracted tissue area is divided into blocks of 256X256 pixels (corresponding to 0.18μm / pixel) at a magnification of 20 times. The division effect is as follows Figure 3 shown.

[0039] In some embodiments, extracting tissue regions using the HEST method for the normalized sample HE pathology image further includes: storing the coordinates of the extracted image blocks as an h5 file to facilitate subsequent feature extraction.

[0040] In some embodiments, the extracted tissue region is input into the Prov-GigaPath network, and a feature vector of a specified dimension is extracted under set parameters, including:

[0041] According to the coordinates of the extracted image blocks, each image block is input into the Prov-GigaPath model;

[0042] The Vision Transformer (ViT) is used as a feature encoder to obtain a feature vector of a specified dimension for each image block, for example, extracting a 1536-dimensional feature vector corresponding to each image block.

[0043] In a specific example, the Prov-GigaPath model is based on the DINOv2 self-supervised learning method and is pre-trained on 1.3 billion 256×256 pathology image blocks from 171,189 WSIs. It can automatically learn multi-level features such as cell morphology and tissue structure in pathology images. It has strong feature expression and generalization capabilities and can capture subtle features in pathology images related to neoadjuvant chemotherapy for breast cancer.

[0044] In some embodiments, reducing the dimension of the extracted feature vector includes performing feature compression on the extracted feature vector of a specified dimension using a principal component analysis method:

[0045] Calculate the covariance matrix of the image block features and determine the eigenvectors of the covariance matrix, which are the principal components of the image block features.

[0046] The importance of each principal component is determined based on the size of the eigenvalue, and the required number of principal components is selected based on the importance to achieve dimensionality reduction. For example, the first 512 important principal components are selected and the original features are projected into the low-dimensional space formed by these principal components, thereby reducing the 1536-dimensional feature vector to 512 dimensions through PCA, reducing the feature dimension and computational complexity, while avoiding the "curse of dimensionality" problem caused by high-dimensional data and improving the efficiency of subsequent model training and prediction.

[0047] In the multi-instance method, each patient's pathology image contains multiple image blocks, each of which can be considered an instance, and the entire image is a package. The ABMIL model is used to aggregate the 512-dimensional features of each image block after dimensionality reduction to obtain the prediction results of neoadjuvant chemotherapy efficacy of breast cancer for the entire WSI. In some embodiments, inputting the reduced feature vector into the ABMIL model includes:

[0048] In the ABMIL model, LayerNorm is used to normalize the input dimensionality-reduced feature vectors. By standardizing the elements of each feature vector, network training is more stable and convergence is accelerated. The GELU (Gaussian Error Linear Unit) activation function is then used. The GELU function adaptively adjusts activation behavior based on the distribution of input data, introducing nonlinear factors into the network and enhancing the model's expressiveness.

[0049] Feature aggregation is performed through a gated attention mechanism consisting of one attention branch and three attention heads. One attention branch calculates the attention weight of each image block, focusing on image blocks that play a key role in predicting the efficacy of neoadjuvant chemotherapy for breast cancer, and mining potential effective information. The gating mechanism controls the flow of information through learning, screens out more discriminative features, further enhances the model's ability to capture important features, and achieves effective aggregation of features from multiple image blocks.

[0050] The aggregated features are passed through a layer of GELU activation function to introduce nonlinear transformation and extract high-order information of the features.

[0051] The features are mapped to the prediction score space through the linear layer, and the predicted scores are normalized by softmax to convert the scores into probability distributions belonging to different categories. In a specific example, 0.5 can be set as the threshold. If the probability value is greater than 0.5, it is judged as positive (pathological complete remission), and if it is less than or equal to 0.5, it is judged as negative, thereby outputting the binary classification result of PCR (pathological complete remission).

[0052] In the specific embodiment of this application, the model training can use LabelSmoothingCrossEntropy as the loss function, where the smoothing parameter is set to 0.1. Use optim.AdamW as the optimizer and set the learning rate lr to 5e -4 , weight decay weight_decay is 1e -4 The above is only a training example and is not limited to any specific example.

[0053] The attention score in the ABMIL model is used to visualize the results of neoadjuvant chemotherapy efficacy prediction for breast cancer. The attention score reflects the degree of attention paid by the model to different image patches during the feature aggregation process. A higher score indicates a greater impact of the image patch on the final prediction result. In some embodiments, visualizing the prediction results using the attention score of the trained ABMIL model includes:

[0054] The attention scores are mapped to the corresponding areas of the original HE pathology image through the coolwarm color bar, and the mapped image blocks have a specified overlap rate, for example, the overlap between image blocks is 50%, so as to visually display the key areas that the model focuses on. Such visualization results can help clinicians intuitively understand the basis for the model's predictions and determine whether the model pays attention to characteristic areas such as cell morphology and tissue structure that are truly related to the efficacy of neoadjuvant chemotherapy for breast cancer. The visualization effect is as follows: Figure 4 shown.

[0055] Compared with the traditional method of manually designing features, this application uses the Prov-GigaPath model to automatically extract pathological image features, breaking through the limitations of manually designed features. Through large-scale self-supervised pre-training, the Prov-GigaPath model can automatically mine multi-level complex features such as cell morphology and tissue structure in HE images, output multi-dimensional feature vectors, and comprehensively capture subtle information related to the efficacy of neoadjuvant chemotherapy for breast cancer. At the same time, the attention branch and gating mechanism in the ABMIL model further focus on key features, combined with the LabelSmoothingCrossEntropy loss function to suppress overfitting, making the model more accurate in predicting the chemotherapy efficacy of different patients, greatly improving the accuracy and reliability of the prediction.

[0056] This application utilizes the ABMIL model combined with Macenko stain normalization and HEST tissue region extraction techniques within a weakly supervised learning framework, requiring only the patient's overall treatment outcome as supervisory information, eliminating the need for detailed annotation of each image patch. By uniformly applying stain normalization and tissue region extraction to images, the workload for data annotation is reduced, the data acquisition threshold is lowered, and model training becomes more efficient, facilitating rapid adoption in clinical scenarios.

[0057] Macenko staining normalization standardizes image staining, eliminating staining variations between images. Furthermore, the powerful feature learning capabilities of the Prov-GigaPath and ABMIL models adapt to image data from diverse sources and quality. The AdamW optimizer, combined with a weight decay strategy, further enhances the model's generalization performance, enabling it to stably and accurately predict the efficacy of neoadjuvant chemotherapy for breast cancer even in complex clinical settings.

[0058] An embodiment of the present application further provides a system for predicting the efficacy of neoadjuvant chemotherapy for breast cancer, comprising a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the steps of the method for predicting the efficacy of neoadjuvant chemotherapy for breast cancer as described above are implemented.

[0059] Furthermore, although exemplary embodiments have been described herein, the scope includes any and all embodiments based on the present disclosure with equivalent elements, modifications, omissions, combinations (e.g., solutions that intersect various embodiments), adaptations, or changes. The present invention is not limited to the examples described in this specification or during the practice of this application, which examples are to be construed as non-exclusive.

[0060] The above description is intended to be illustrative rather than restrictive. For example, the above examples (or one or more of them) can be used in combination with each other. For example, those of ordinary skill in the art may use other embodiments when reading the above description.

[0061] The above embodiments are merely exemplary embodiments of the present disclosure. Those skilled in the art may make various modifications or equivalent substitutions to the present invention within the essence and protection scope of the present disclosure, and such modifications or equivalent substitutions should also be deemed to fall within the protection scope of the present invention.

Claims

1. A method for predicting the efficacy of neoadjuvant chemotherapy for breast cancer, characterized in that: include: Macenko staining normalization was performed on the patient's sample HE pathological images; For the normalized sample HE pathological images, the HEST method is used to extract tissue regions; Input the extracted tissue region into the Prov-GigaPath network, extract the feature vector of the specified dimension under the set parameters, and perform dimensionality reduction on the extracted feature vector; Input the reduced feature vector into the ABMIL model and train it; Predictions are made for HE pathology images of patients based on the trained ABMIL model, and the prediction results are visualized using the attention scores of the trained ABMIL model.

2. The method for predicting the efficacy of neoadjuvant chemotherapy for breast cancer according to claim 1, wherein: Macenko staining normalization of patient sample HE pathology images includes: Convert the sample HE pathology image from RGB space to optical density OD space; In OD space, calculate the staining vector of the transformed sample HE pathology image; According to the coloring vector, two principal component vectors representing H coloring and E coloring are determined by singular value decomposition (SVD); The staining vectors of the sample HE pathology images were normalized using the mean and standard deviation of the staining concentration of the reference image.

3. The method for predicting the efficacy of neoadjuvant chemotherapy for breast cancer according to claim 1, wherein: For the normalized sample HE pathology image, the HEST method is used to extract tissue regions including: The HEST segmentation model is used to segment the HE image after dye normalization; After segmentation, extracting tissue regions containing valid information, wherein the valid information includes breast cancer cells; The extracted tissue area is divided into blocks, and the WSI image is divided into non-overlapping image blocks of a specified size.

4. The method for predicting the efficacy of neoadjuvant chemotherapy for breast cancer according to claim 3, wherein: Extracting the tissue region from the normalized sample HE pathological image using the HEST method further includes: storing the coordinates of the extracted image block as an h5 file.

5. The method for predicting the efficacy of neoadjuvant chemotherapy for breast cancer according to claim 4, wherein: The extracted tissue region is input into the Prov-GigaPath network, and the feature vectors of the specified dimensions are extracted under the set parameters, including: According to the coordinates of the extracted image blocks, each image block is input into the Prov-GigaPath model; The Vision Transformer (ViT) is used as a feature encoder to obtain the feature vector of a specified dimension for each image block.

6. The method for predicting the efficacy of neoadjuvant chemotherapy for breast cancer according to claim 1, wherein: Dimensionality reduction of the extracted feature vector includes feature compression of the extracted feature vector of specified dimension using principal component analysis method: Calculate the covariance matrix of the image block features and determine the eigenvectors of the covariance matrix, which are the principal components of the image block features; The importance of each principal component is determined according to the size of the eigenvalue, and based on the importance, the required number of principal components are selected to achieve dimensionality reduction.

7. The method for predicting the efficacy of neoadjuvant chemotherapy for breast cancer according to claim 1, wherein: Inputting the reduced feature vector into the ABMIL model includes: In the ABMIL model, layer normalization LayerNorm is used to normalize the input dimensionality-reduced feature vector; Feature aggregation is performed through a gated attention mechanism consisting of 1 attention branch and 3 attention heads, where 1 attention branch calculates the attention weight of each image block; The aggregated features are passed through a layer of GELU activation function to introduce nonlinear transformation and extract high-order information of the features; The features are mapped to the prediction score space through the linear layer Linear, and the prediction scores are normalized by softmax to convert the scores into probability distributions belonging to different categories.

8. The method for predicting the efficacy of neoadjuvant chemotherapy for breast cancer according to claim 7, wherein: It also includes: During the model training process, LabelSmoothingCrossEntropy is used as the loss function and optim.AdamW is used as the optimizer for training.

9. The method for predicting the efficacy of neoadjuvant chemotherapy for breast cancer according to claim 1, wherein: Visualizing the prediction results using the attention scores of the trained ABMIL model includes: The attention scores are mapped to the corresponding areas of the original HE pathology image through a coolwarm color bar, and the mapped image blocks have a specified overlap rate to visually display the key areas of the model's attention.

10. A system for predicting the efficacy of neoadjuvant chemotherapy for breast cancer, characterized in that: The method comprises a processor and a memory, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the steps of the method for predicting the efficacy of neoadjuvant chemotherapy for breast cancer according to any one of claims 1 to 9 are implemented.

Images