High-resolution breast cancer HER2 scoring method based on deep learning

Abstract

The invention discloses a high-resolution breast cancer HER2 scoring method based on deep learning. The high-resolution breast cancer HER2 scoring method comprises the following steps: constructing a full-slide image data set containing hematoxylin-eosin and HER2 staining; accurately marking by a pathological expert, and distinguishing normal, in-situ and invasive tumor areas; cutting, classifying and marking blocks with different sizes and magnification times; carrying out gray level conversion, binarization and morphological operation on the image blocks to extract a tissue region; designing a deep learning model combining a visual converter and Fourier transform, and optimizing feature extraction and frequency domain analysis; an optimal backbone network structure is selected through comparison; the model is trained, parameters are optimized, and the recognition and evaluation accuracy is improved; and finally integrating to an automatic scoring software system to realize automatic identification, HER2 protein expression evaluation and automatic scoring. The system is high in automation degree, the HER2 scoring process is simplified, the auxiliary diagnosis precision is improved, the workload of doctors is reduced, and pathologists are helped to concentrate on complex diagnosis tasks, so that the patient management effect is improved.

Description

Technical Field

[0001] The present invention relates to the field of tissue pathology image processing, and in particular to a high-resolution breast cancer HER2 scoring method based on deep learning. Background Art

[0002] Targeted therapies are available for patients with human epidermal growth factor receptor 2 (HER2)-positive breast cancer. In addition, some new HER2-positive breast cancer drugs, such as trastuzumab-drastogen (DS-8201) and trastuzumab-drugamazine (SYD985), not only have satisfactory therapeutic effects in classic HER2-positive breast cancer, but may also have clinical activity in HER2-low-expressing breast cancer (HER2-low expression refers to HER2 IHC 2+ and ISH negative or IHC 1+). However, IHC 0 and IHC 1+ are difficult to distinguish accurately due to their unclear characteristics. Whether to use such expensive drugs is related to the patient's recovery and economic burden. Therefore, it is particularly important to accurately distinguish the HER2 status between IHC 0 and IHC 1+ to determine whether these drugs can be used. At the same time, the results of HER2 2+ are considered equivocal and require further fluorescence in situ hybridization (FISH) testing, and it is meaningful to judge only by the HER2 indicator.

[0003] The current scoring relies heavily on the experience of pathologists, which is subjective and non-reproducible. The diagnostic results performed by several doctors or even the same doctor at different times are inconsistent. In contrast, artificial intelligence (AI) can provide a more objective and precise assessment of HER2 expression in cancer cells to guide clinical treatment. Secondly, considering the current assessment, the four different grades of HER2 staining (IHC 0, IHC 1+, IHC 2+, and IHC 3+) can be subdivided. In this case, a new HER2 staining grading system can be established, which may provide a more accurate assessment for clinical breast cancer targeted therapy and patient prognosis.

[0004] HER2 scoring is performed on invasive tumor areas of whole slide images (WSIs). Specifically, HER2 scoring does not include carcinoma in situ. Carcinoma in situ cancer cells are confined to the breast ducts and are therefore generally considered non-invasive and have a better prognosis. On the other hand, invasive cancer cancer cells have already crossed the basement membrane and have a poorer prognosis. Due to the fundamental difference in HER2 expression levels between carcinoma in situ and invasive cancer, only invasive cancer is considered for HER2 scoring to ensure that treatment decisions are based on accurate molecular features of invasive cancer. Since the release of the HER2 grading competition at the University of Warwick in 2016, computer recognition algorithms have been widely studied in terms of HER2 status. However, existing methods ignore the identification of invasive tumor areas and instead perform them directly on certain areas of the WSI. For example, in the HER2 status assessment work, they calculated HER2 scores directly in epithelial tissues, which contain both cancerous and normal epithelial tissues. Pathologists focus on the invasive part of the cancer tissue because HER2 status in breast cancer is determined by the invasive tumor. Therefore, this method is not suitable for clinical use. Although Qaiser can predict HER2 scores through a series of automatically selected tiles and has made algorithmic breakthroughs, the network may ignore key tiles. This may have a fatal impact on pathological diagnosis. Algorithms based on the HER2 dataset of the Warwick University competition and other studies either do not take into account invasive tumor areas or fail to identify invasive tumor areas (some only consider cancer areas, not invasive cancer areas). Therefore, these algorithms are not practical and convincing in clinical settings and cannot be used in clinical settings.

[0005] It should be noted that the information disclosed in the above background technology section is only used for understanding the background of the present application, and therefore may include information that does not constitute prior art known to ordinary technicians in the field. Summary of the invention

[0006] The main purpose of the present invention is to solve the problems existing in the above-mentioned background technology and provide a high-resolution breast cancer HER2 scoring method based on deep learning.

[0007] To achieve the above object, the present invention adopts the following technical solutions:

[0008] A high-resolution breast cancer HER2 scoring method based on deep learning, including:

[0009] Construct a WSI dataset of whole slide images containing hematoxylin-eosin staining and HER2 staining;

[0010] Pathologists annotated the whole slide images in the dataset to distinguish normal tissue, in situ tumor, and invasive tumor areas;

[0011] Crop the whole slide image, extract tiles of different sizes and magnifications, and classify and annotate the tiles;

[0012] Preprocess the cropped image blocks, extract the tissue area, and perform grayscale conversion, binarization, and morphological operations;

[0013] Construct a deep learning model that combines visual transformers and Fourier transforms to achieve feature extraction and image frequency domain analysis, where the visual transformer captures local features and contextual relationships, and the Fourier transform identifies frequency domain features, thereby identifying invasive tumors and evaluating the expression level of HER2 protein in breast cancer tissues;

[0014] Training the deep learning model using the tile dataset;

[0015] The trained deep learning model is integrated into the automated scoring software system. The deep learning model is used to identify and evaluate the expression level of HER2 protein in breast cancer tissue, score it based on the expression level of HER2 protein, and automatically output the scoring results.

[0016] Furthermore, pathologists annotated the whole slide images in the dataset, including: using H&E staining WSI paired with HER2 to guide the annotation of tumor regions and their HER2-positive regions, so as to more effectively characterize the structure of the tumor region and improve the accuracy and speed of annotation.

[0017] Furthermore, the cropping of the whole slide image specifically includes: cropping a 2048×1024 pixel tile at a magnification of 20× for tumor identification; and cropping a 100×100 pixel tile at a magnification of 40× for HER2 classification.

[0018] Furthermore, for tumor identification, the classification and labeling of image blocks specifically includes:

[0019] For pure region tiles, they are directly labeled with corresponding annotations;

[0020] For tiles with mixed areas, classification and labeling are performed according to the following criteria: if the normal tissue area in the tile accounts for more than 50% and does not contain the carcinoma in situ area, the tile is labeled as normal tissue; if the in situ tumor area accounts for more than 1 / 3 of the tile, and the in situ tumor accounts for 2 / 3 of the carcinoma in situ area in the tile, the tile is labeled as in situ tumor; if the invasive tumor area in the tile accounts for more than 50% and does not contain the carcinoma in situ area, the tile is labeled as invasive tumor.

[0021] Further, with respect to HER2 classification, the classification and labeling of image blocks specifically includes:

[0022] The tiles were divided into five categories according to the HER2 staining intensity: no staining, weak staining, moderate staining, strong staining and background; the ResNet50 model pre-trained on the ImageNet-1k dataset was used as the initial model to accelerate the subsequent automatic annotation process; in the initial stage, pathologists annotated and reviewed the tiles; the ResNet50 model was trained using the obtained annotated data to identify different categories of HER2-stained tiles; the trained ResNet50 model was applied to unlabeled tiles to automatically predict their categories; for tiles whose model predicted a probability of a certain category greater than or equal to the probability threshold, they were automatically classified into that category and used as a newly generated dataset; pathologists checked the automatically classified tile labels.

[0023] Furthermore, the preprocessing of the cropped image blocks to extract the tissue area specifically includes: converting the whole slide image WSI into a grayscale image to reduce the complexity of the color channel and focus on the morphological features; calculating the median of all pixels in the grayscale image, and using the median as a threshold to convert the grayscale image into a binary image to distinguish between tissue and background; performing a morphological closing operation on the binary image, using a 1×1 pixel kernel to fill small holes and connect adjacent objects; performing a morphological opening operation on the image after the closing operation, using a 15×15 pixel kernel to remove small noise and separate contacting objects; and normalizing the pixel values ​​in the binary image after the opening operation to the range of [0,1].

[0024] Furthermore, in the deep learning model, the tokenization capability of the visual transformer is utilized Processing image graph Blocks capture the local features of each block and the context of the entire image, making The model is able to recognize complex patterns of HER2 expression in different tissue regions; Fourier transforms are incorporated into the deep learning architecture to enhance the model's ability to interpret image frequency domain features, especially for variable staining intensities and texture patterns commonly found in pathology slides; staining and texture patterns are decomposed into basic frequency components through Fourier transforms, providing a simplified feature representation for the visual transformer, thereby improving the model's learning and generalization capabilities for pathology images.

[0025] Furthermore, the backbone network of the deep learning model is selected from ResNet18, ResNet50, ResNet101 and ResNet152, preferably ResNet50.

[0026] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the high-resolution breast cancer HER2 scoring method based on deep learning.

[0027] A computer program product, when executed by a processor, implements the high-resolution breast cancer HER2 scoring method based on deep learning.

[0028] The present invention has the following beneficial effects:

[0029] The present invention provides a high-resolution breast cancer HER2 scoring method based on deep learning, establishes a scoring system for HER2 high resolution, introduces a multi-scale model to identify invasive tumor areas of WSIs before HER2 scoring, simulates the clinical diagnostic procedure, and realizes an automatic process from WSIs to HER2 high-resolution scoring.

[0030] Compared with the traditional technology, the present invention has the following advantages:

[0031] 1. The high-resolution breast cancer HER2 scoring system based on deep learning proposed in this invention has made innovations in algorithms and is highly competitive in invasive tumor identification and HER2 block classification.

[0032] 2. The integration of this invention into clinical workflow can significantly simplify HER2 scoring, reduce physician workload and improve auxiliary diagnosis accuracy. This automation can free up pathologists to focus on more nuanced aspects of diagnostic pathology, potentially helping physicians achieve better patient management outcomes.

[0033] Other beneficial effects of the embodiments of the present invention will be further described below. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 This is a method flow chart of a high-resolution breast cancer HER2 scoring system based on deep learning in an embodiment of the present invention.

[0035] Figure 2 It is a network framework diagram of the algorithm of an embodiment of the present invention.

[0036] Figure 3 Schematic diagram of a scoring software system according to an embodiment of the present invention. DETAILED DESCRIPTION

[0037] The following is a detailed description of the embodiments of the present invention. It should be emphasized that the following description is only exemplary and is not intended to limit the scope and application of the present invention.

[0038] The embodiment of the present invention provides a high-resolution breast cancer HER2 scoring method based on deep learning, comprising the following steps:

[0039] Construct a WSI dataset of whole slide images containing hematoxylin-eosin staining and HER2 staining;

[0040] Pathologists annotated the whole slide images in the dataset to distinguish normal tissue, in situ tumor, and invasive tumor areas;

[0041] Crop the whole slide image, extract tiles of different sizes and magnifications, and classify and annotate the tiles;

[0042] Preprocess the cropped image blocks, extract the tissue area, and perform grayscale conversion, binarization, and morphological operations;

[0043] A deep learning model is constructed, which combines a visual transformer (ViT) and Fourier transform for feature extraction and image frequency domain analysis, wherein the visual transformer is used to capture local features and contextual relationships, and the Fourier transform enhances the model's recognition ability of frequency domain features, thereby identifying invasive tumors and evaluating the expression level of HER2 protein in breast cancer tissues; wherein the optimal backbone network structure can be selected for the deep learning model by comparing the performance of different backbone networks on the invasive tumor recognition task;

[0044] Using the tile dataset to train the deep learning model, optimizing the model parameters to improve the accuracy of invasive tumor identification and HER2 expression assessment;

[0045] The trained deep learning model is integrated into the automated scoring software system. The trained deep learning model is used to identify and evaluate the expression level of HER2 protein in breast cancer tissues, and the score is automatically output according to the expression level of HER2 protein. The system can integrate the user interface, image processing module and scoring result display.

[0046] The breast cancer HER2 scoring method of the embodiment of the present invention introduces a multi-scale model, which is trained using images or tiles at different magnifications to simulate the process of pathologists observing tissue sections at different magnifications. The model can process inputs of different sizes to adapt to the actual size and distribution of the tumor area in the image, and combines features at different scales to improve the recognition accuracy of invasive tumors and HER2 expression patterns.

[0047] In some specific embodiments, the automatic HER2 high-resolution scoring algorithm based on multi-scale model and multi-domain model provided by the present invention realizes an automated breast cancer HER2 scoring software system. A 5-level grading of IHC 0 / IHC 1+ and a 5-level grading of IHC 2+ are designed, and the HER2 status between IHC 0 and IHC 1+ is subdivided into five categories, and the IHC 2+ status is subdivided into five categories (based on the newly revised guidelines of "Human Epidermal Growth Factor Receptor 2 Detection in Breast Cancer" in the United States in 2018), providing a more accurate assessment for targeted treatment of breast cancer and patient prognosis.

[0048] The high-resolution breast cancer HER2 scoring method based on deep learning of the present invention provides an efficient and accurate automated scoring system through innovative technical solutions, which significantly improves the accuracy and efficiency of clinical diagnosis. The method uses a deep learning model combined with a visual transformer and Fourier transform to not only capture the local features and contextual relationships of image blocks, but also enhances the recognition ability of image frequency domain features through Fourier transform, especially for variable staining intensity and texture patterns in pathological slides. This deep learning architecture combined with a multi-scale model can accurately identify invasive tumor areas and simulate clinical diagnostic procedures, thereby accurately identifying regions before HER2 scoring. Compared with the traditional subjective scoring method that relies on the experience of pathologists, the automated scoring system of the present invention reduces human errors, provides more consistent and reliable scoring results, helps guide clinical treatment decisions, and reduces the workload of doctors, allowing pathologists to focus more on more complex tasks in diagnostic pathology, and further improves the effect of patient management. In addition, the innovation of the scoring system of the present invention in algorithm design enhances the competitiveness of invasive tumor identification and HER2 block classification, ensuring its practicality and persuasiveness in clinical settings.

[0049] The following further describes a specific embodiment, a method for a high-resolution breast cancer HER2 scoring system based on deep learning includes:

[0050] Step S0 constructs a data set of digital slices of breast cancer, and the data of each patient includes a WSI of hematoxylin-eosin (H&E) staining and a WSI of HER2 staining.

[0051] Step S1: The WSI is annotated by two pathologists and checked by one pathologist.

[0052] In step S2, the WSI is cropped using self-developed software and divided into three types of blocks, namely normal tissue, in situ tumor, and invasive tumor.

[0053] Step S3 preprocesses the WSI image to extract the target tissue area and reduce unnecessary detection.

[0054] Step S4 designs a deep learning framework combining visual transformer and Fourier transform to train pathological images.

[0055] The details are as follows:

[0056] Step S0-0 A total of 256 WSIs were collected from 128 breast cancer patients in the Department of Pathology, The First Affiliated Hospital of Sun Yat-sen University, Guangzhou, China. The data for each patient included one hematoxylin-eosin (H&E)-stained WSI and one HER2-stained WSI.

[0057] Step S0-1 uses a SQS-120P digital slide scanner provided by Shenzhen Shengqiang Technology Co., Ltd. to scan these tissue sections at a high resolution of 40× (0.115 μm / pixel).

[0058] Step S1-0 Annotation of HER2-stained WSIs was performed by two pathologists and annotated by one pathologist. Expert inspection. In each case, paired H&E staining WSI with HER2 was used to guide Annotation of tumor regions and their HER2-positive areas. This is because H&E can more effectively characterize the structure of tumor regions and improve the accuracy and speed of annotation.

[0059] Step S2-0 implemented self-developed software to trim WSIs and construct different datasets for aggressive tumor identification and HER2 classification.

[0060] Step S2-1 crops tiles of 2048×1024 pixels at 20× magnification for tumor identification. These tiles are classified into three categories—normal tissue, in situ tumor, and invasive tumor. The tiles are annotated by a pathologist and reviewed by an expert.

[0061] The labeling is as follows. (1) Tiles with pure areas are labeled with corresponding annotations. (2) Tiles with mixed areas. If the normal tissue area occupies more than 50% and there is no carcinoma in situ area, the tile label is set to normal tissue. If more than 1 / 3 of the tile is composed of in situ tumors, and the in situ tumors occupy 2 / 3 of the cancer site in the tile, the label is set to in situ tumors (because in situ carcinomas usually have a regular spherical morphology). If the invasive tumor area occupies more than 50% and there is no carcinoma in situ area, the tile label is set to invasive tumors.

[0062] Step S2-2 cropped 100×100 pixel tiles at 40× magnification for HER2 classification.

[0063] These tiles were grouped using five categories: no staining, weak staining, moderate staining, strong staining, and background. A ResNet50 model pre-trained on the ImageNet-1k dataset was used to speed up the annotation process. Specifically, 2,000 tiles were first annotated by a pathologist and checked by an expert, and then used to train the ResNet50 model. The newly trained model was applied to unlabeled tiles, and those tiles with a probability of greater than 80% for a certain category were used as a newly generated dataset for that category. These automatically selected tile labels were checked by two pathologists. In this way, the speed of data production was greatly improved.

[0064] Step S3-0 Each WSI contains tissue and blank areas. Extracting tissue areas is beneficial and can reduce unnecessary detections of the neural network. The method is as follows. First, the WSI is converted into a grayscale image, and then the median of the grayscale pixels is taken as the threshold to obtain a binary image. Next, a closing operation of a 1×1 kernel and an opening operation of a 15×15 kernel are performed on the binary image in sequence. Finally, the value of each pixel is normalized to [0,1].

[0065] Step S4-0 To address the unique challenges presented by analyzing pathological images for HER2 scoring of breast cancer, a deep learning architecture combining a visual transformer (ViT) and a Fourier transform was designed. The visual transformer was chosen based on its ability to treat image patches as tokens similar to words in a sentence, which enables the model to capture local features within each patch as well as the context of the entire image. This is critical for identifying complex patterns of HER2 expression that are Organization There may be significant changes in the area.

[0066] Step S4-1 combines the Fourier transform with the transformer architecture to enhance the model's ability to interpret the frequency domain of the image. Pathology slides often exhibit variable staining intensity and texture patterns, which are critical for accurate HER2 scoring. The Fourier transform helps decompose these patterns into simpler frequency components, allowing the transformer to better learn from these features and generalize to different samples.

[0067] experiment

[0068] The present invention conducted multiple experiments on the selection of backbone networks, and selected ResNet18, ResNet50, ResNet101 and ResNet152 for invasive tumor recognition, and the models were named Inva-Res18, Inva-Res50, Inva-Res101 and Inva-Res152 respectively. The results are shown in the following table. Inva-Res50 outperformed the other three backbones, and its accuracy was more than 1.2% higher than the other three models.

[0069]

[0070] At the same time, the present invention uses 10×, 20× and 40× different magnifications but the same size of the image blocks to train Inva-Res50 (best performance). The results show that the image block with a magnification of 10× performs best, with an accuracy of more than 2.7% higher than other image blocks and an AUC of 4.8% higher. The results are shown in the following table.

[0071]

[0072] The present invention selects 1024×512 (10×) and 1024×512 (20×) size tiles to train the multi-scale model based on ResNet50. Compared with the above best model, the multi-scale model Inva-Res50-10-20× performs better, with an improvement of 2.1% and 0.65% in precision and accuracy respectively, as shown below.

[0073]

[0074] In order to compare the performance of different models on HER2 patch recognition, this paper compares CNN-based and ViT-based methods, and the results are shown in the following table. ResNeXt101-32x8d and DenseNet121 perform best in accuracy and AUC, while Fourier ViT improves by 3.62% and 1.6%, respectively. In addition, Fourier ViT outperforms DeiT, a Transformer-based model, by 1.65% and 1.62% in accuracy and F1 score, respectively.

[0075] In practice, the pre-trained Fourier ViT was implemented for HER2 patch classification. Specifically, the Visual Transformer (ViT) model trained on ImageNet1K was applied to the corresponding ViT positions of Fourier ViT. In this way, the pre-trained Fourier ViT outperformed the non-pre-trained model by more than 1.3% in accuracy, precision, recall, and F1 score.

[0076]

[0077] Compared with the traditional technology, the present invention has the following advantages:

[0078] The high-resolution breast cancer HER2 scoring method based on deep learning proposed in the present invention has made innovations in the algorithm and is highly competitive in invasive tumor identification and HER2 block classification.

[0079] The method of the present invention provides a high-resolution, clinically significant and fully automated breast cancer HER2 scoring system, which provides a more accurate assessment for breast cancer targeted therapy and patient prognosis.

[0080] The integration of this invention into clinical workflow can significantly simplify HER2 scoring, reduce physician workload and improve auxiliary diagnostic accuracy. This automation can free up pathologists to focus on more nuanced aspects of diagnostic pathology, potentially helping physicians achieve better patient management outcomes.

[0081] An embodiment of the present invention further provides a storage medium for storing a computer program, which at least performs the above method when executed.

[0082] An embodiment of the present invention further provides a control device, comprising a processor and a storage medium for storing a computer program; wherein the processor is configured to execute at least the method described above when executing the computer program.

[0083] An embodiment of the present invention further provides a processor, wherein the processor executes a computer program and at least executes the method described above.

[0084] The storage medium can be implemented by any type of non-volatile storage device, or a combination thereof. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a ferromagnetic random access memory (Flash Memory), a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); the magnetic surface memory can be a disk memory or a tape memory. The storage medium described in the embodiments of the present invention is intended to include, but is not limited to, these and any other suitable types of memory.

[0085] In the several embodiments provided by the present invention, it should be understood that the disclosed systems and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0086] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0087] In addition, all functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0088] It will be understood by those skilled in the art that all or part of the steps of the above method embodiments are implemented by Steps can be passed The program instructions are completed by the hardware related to the program. The aforementioned program can be stored in a computer In the readable storage medium, when the program is executed, it executes the steps of the above-mentioned method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disks or optical disks and other media that can store program codes.

[0089] Alternatively, if the above-mentioned integrated unit of the present invention is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention can be essentially or partly reflected in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, magnetic disks or optical disks.

[0090] The methods disclosed in the several method embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method embodiments.

[0091] The features disclosed in several product embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new product embodiments.

[0092] The features disclosed in several method or device embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0093] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art of the present invention, several equivalent substitutions or obvious variations can be made without departing from the concept of the present invention, and the performance or use is the same, which should be regarded as belonging to the protection scope of the present invention.

Claims

1. A high-resolution breast cancer HER2 scoring method based on deep learning, characterized in that: The method comprises the following steps: Construct a WSI dataset of whole slide images containing hematoxylin-eosin staining and HER2 staining; Pathologists annotated the whole slide images in the dataset to distinguish between normal tissue, in situ tumor, and invasive tumor areas; Crop the whole slide image, extract tiles of different sizes and magnifications, and classify and annotate the tiles; Preprocess the cropped image blocks, extract the tissue area, and perform grayscale conversion, binarization, and morphological operations; Construct a deep learning model that combines visual transformers and Fourier transforms to achieve feature extraction and image frequency domain analysis, where the visual transformer captures local features and contextual relationships, and the Fourier transform identifies frequency domain features, thereby identifying invasive tumors and evaluating the expression level of HER2 protein in breast cancer tissues; Training the deep learning model using the tile dataset; The trained deep learning model is integrated into the automated scoring software system. The deep learning model is used to identify and evaluate the expression level of HER2 protein in breast cancer tissue, score it based on the expression level of HER2 protein, and automatically output the scoring results.

2. The high-resolution breast cancer HER2 scoring method based on deep learning according to claim 1, characterized in that: The pathology expert annotation of the whole slide images in the data set specifically includes: using the H&E staining WSI paired with HER2 to guide the annotation of the tumor area and its HER2-positive area, so as to more effectively characterize the structure of the tumor area and improve the accuracy and speed of the annotation.

3. The high-resolution breast cancer HER2 scoring method based on deep learning according to claim 1 or 2, characterized in that: The cropping of the whole slide image specifically includes: cropping a 2048×1024 pixel block at a magnification of 20× for tumor identification; and cropping a 100×100 pixel block at a magnification of 40× for HER2 classification.

4. The high-resolution breast cancer HER2 scoring method based on deep learning according to any one of claims 1 to 3, characterized in that: For tumor identification, the classification and labeling of image blocks specifically include: For pure region tiles, they are directly labeled with corresponding annotations; For tiles with mixed areas, classification and labeling are performed according to the following criteria: if the normal tissue area in the tile accounts for more than 50% and does not contain the carcinoma in situ area, the tile is labeled as normal tissue; if the in situ tumor area accounts for more than 1 / 3 of the tile, and the in situ tumor accounts for 2 / 3 of the carcinoma in situ area in the tile, the tile is labeled as in situ tumor; if the invasive tumor area in the tile accounts for more than 50% and does not contain the carcinoma in situ area, the tile is labeled as invasive tumor.

5. The high-resolution breast cancer HER2 scoring method based on deep learning according to any one of claims 1 to 4, characterized in that: For HER2 classification, the classification and labeling of image blocks specifically include: The tiles were divided into five categories according to the HER2 staining intensity: no staining, weak staining, moderate staining, strong staining and background; the ResNet50 model pre-trained on the ImageNet-1k dataset was used as the initial model to accelerate the subsequent automatic annotation process; in the initial stage, pathologists annotated and reviewed the tiles; the ResNet50 model was trained using the obtained annotated data to identify different categories of HER2-stained tiles; the trained ResNet50 model was applied to unlabeled tiles to automatically predict their categories; for tiles whose model predicted a probability of a certain category greater than or equal to the probability threshold, they were automatically classified into that category and used as a newly generated dataset; pathologists checked the automatically classified tile labels.

6. The high-resolution breast cancer HER2 scoring method based on deep learning according to any one of claims 1 to 5, characterized in that: The preprocessing of the cropped image blocks to extract the tissue area specifically includes: converting the whole slide image WSI into a grayscale image to reduce the complexity of the color channel and focus on the morphological features; calculating the median of all pixels in the grayscale image, and using the median as a threshold to convert the grayscale image into a binary image to distinguish between tissue and background; performing a morphological closing operation on the binary image, using a 1×1 pixel kernel to fill small holes and connect adjacent objects; performing a morphological opening operation on the image after the closing operation, using a 15×15 pixel kernel to remove small noise and separate contacting objects; and normalizing the pixel values ​​in the binary image after the opening operation to the range of [0,1].

7. The high-resolution breast cancer HER2 scoring method based on deep learning according to any one of claims 1 to 6, characterized in that: In the deep learning model, the visual transformer's tokenization capability is used to process image tiles, capturing local features of each tile and the overall context of the image, so that the model can recognize complex patterns of HER2 expression in different tissue regions; Fourier transform is combined with the deep learning architecture to enhance the model's ability to interpret image frequency domain features, especially for variable staining intensity and texture patterns commonly seen in pathology slides; Fourier transform is used to decompose staining and texture patterns into basic frequency components, providing a simplified feature representation for the visual transformer, thereby improving the model's learning and generalization capabilities for pathology images.

8. The high-resolution breast cancer HER2 scoring method based on deep learning according to any one of claims 1 to 7, characterized in that: The backbone network of the deep learning model is selected from ResNet18, ResNet50, ResNet101 and ResNet152, preferably ResNet50.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the high-resolution breast cancer HER2 scoring method based on deep learning is implemented as described in any one of claims 1 to 8.

10. A computer program product, characterized in that When the computer program product is executed by a processor, the high-resolution breast cancer HER2 scoring method based on deep learning is implemented as described in any one of claims 1 to 8.

Images