Construction method and device of breast cancer molecular subtype prediction model based on filter and pathological image, prediction method

Abstract

The application provides a breast cancer molecular subtype prediction model construction method and device based on a filter and pathological images, and a prediction method. The breast cancer molecular subtype prediction model construction method first constructs a filter using blocks on spatial transcriptome data, and then uses the filter to screen valuable blocks for the construction of a breast cancer molecular subtype prediction model. The construction of the filter includes cutting histopathological images in the spatial transcriptome data into many blocks; then extracting corresponding gene expression information of ESR1, ESR2, PGR and ERBB2 to form labels; and then constructing a training model to obtain the filter. The construction of the molecular subtype prediction model includes: after standardizing the WSI blocks, using the filter to screen blocks containing the expression of at least one of ESR1, ESR2, PGR and ERBB2, and then putting the blocks into a convolutional neural network model for training to obtain the final breast cancer molecular subtype prediction model.

Description

Technical Field

[0001] This invention relates to the field of machine learning and medical interdisciplinary technology, and in particular to a method, apparatus and prediction method for constructing a molecular subtype prediction model for breast cancer based on a screener and pathological images. Background Technology

[0002] Cancer has long been a major threat to human physical and mental health. According to a report by the International Agency for Research on Cancer (AIRC), an estimated 19.3 million new cancer cases and nearly 10 million cancer deaths occurred in 2020. Breast cancer in women has surpassed lung cancer to become the most commonly diagnosed cancer, with an estimated 2.3 million new cases (11.7%), seriously endangering women's health. How to effectively treat breast cancer is a crucial research question.

[0003] Traditionally, breast cancer patients are classified pathologically, and treatment plans are then developed based on the classification results. However, breast cancer is a highly heterogeneous tumor, and traditional pathological morphological classification has been found to have some limitations in clinical practice. For example, breast cancers with the same pathological morphology can exhibit high heterogeneity at the molecular level due to molecular genetic alterations, leading to significant differences in their response to treatment. Molecular subtyping based on molecular biology technologies such as gene chips and next-generation sequencing can more accurately reflect the biological behavior of tumors, which is more conducive to prognosis assessment and the selection of more targeted and personalized treatment methods. Breast cancer is a heterogeneous disease composed of different molecular subtypes, and heterogeneity is one of the key factors leading to drug resistance and treatment failure. Therefore, accurately identifying the molecular subtype of cancer is an important prerequisite for precision medicine.

[0004] In clinical diagnosis, doctors determine the benign or malignant nature of breast cancer based on HE-stained pathological sections, and then use immunohistochemistry (IHC) to perform molecular subtyping of breast cancer patients. Immunohistochemistry is a technique that detects antigens in tissue samples. It utilizes the principle of specific binding between antigens and antibodies, achieving antigen detection through chemical reactions and chromogenic agents. Immunohistochemistry classifies breast cancer into molecular subtypes—Luminal A, Luminal B, Her2, and Basal-like—based on the status of ER, PR, and HER2. This method has several drawbacks: firstly, molecular subtyping using immunohistochemistry requires multiple staining of multiple sections, resulting in a lengthy process and hindering rapid diagnosis; secondly, due to tumor heterogeneity, the expression of ER, PR, and HER2 genes often differs in different sections, potentially leading to inaccurate molecular subtype diagnosis. Therefore, it is necessary to develop a faster and more effective molecular subtyping method for clinical diagnosis. From the process of predicting molecular subtypes using immunohistochemistry, we know that the prediction of breast cancer molecular subtypes is based on the gene expression of estrogen receptor (ER), progesterone receptor (PR), and human epidermal growth factor receptor 2 (HER2). H&E-stained histopathological images contain morphological information that can infer gene expression, and artificial intelligence (AI) technology has been successfully applied to predict gene expression information. Therefore, we consider using AI technology to directly predict molecular subtypes from H&E-stained histopathological images. If we use AI to directly predict molecular subtypes from H&E-stained histopathological images, the diagnosis of tumor benignity / malignancy and molecular subtype can be completed in the same histopathological section without the need for additional immunohistochemistry. This would significantly reduce the patient's diagnostic time and improve and reduce the workload of physicians.

[0005] Histopathological images are generally stored in Whole Slide Image (WSI) format, which consists of images with millions of pixels. Due to computer limitations, it is difficult to process the entire whole slide image directly. Therefore, these images need to be segmented into many patches, which are then processed by the computer. How to utilize these patches is key to accurately predicting molecular subtypes. In recent years, some studies have attempted to predict molecular subtypes from H&E-stained histopathological images, proposing different strategies for using these patches, which can be broadly categorized into three strategies: first, using all patches; second, extracting patches from the tumor region for analysis; and third, selecting patches according to certain morphological methods. However, these methods have certain drawbacks: using all patches introduces a large amount of noise that affects prediction accuracy; extracting tumor region information relies excessively on manual labeling; and using morphological methods lacks clinical interpretability. Summary of the Invention

[0006] This invention provides a method for constructing a molecular subtype prediction model for breast cancer based on a filter and pathological images, thereby solving or at least partially solving the technical problem of poor model performance in the prior art.

[0007] To address the aforementioned technical problems, the first aspect of this invention provides a method for constructing a breast cancer molecular subtype prediction model based on a screener and pathological images, comprising:

[0008] S1: Divide the histopathological images in the spatial transcriptome data into multiple blocks;

[0009] S2: Extract partial gene expression information from the segmented spatial transcriptome data to obtain the gene information of ESR1, ESR2, PGR and ERBB2 for each spot corresponding to each block. If at least one gene is expressed in a block, the label is set to 1, otherwise it is 0. The spot is a spot, which is a data attribute of the spatial transcriptome data.

[0010] S3: Build a convolutional neural network classification model A and randomly initialize the parameters of model A;

[0011] S4: Standardize the color of the blocks obtained from step S1, and input the blocks and corresponding gene labels into model A for training, so as to obtain the trained model A as the block filter;

[0012] S5: Perform block segmentation and color normalization on the WSI of the pathological image dataset;

[0013] S6: Use the trained model A to predict gene expression in each block in S5 to obtain blocks containing ESR1, ESR2, PGR and ERBB2 gene expression.

[0014] S7: Annotate the WSIs of the pathological image dataset in S5 to obtain the molecular subtype label corresponding to each WSI, which will be used as the molecular subtype label for each block.

[0015] S8: Build a convolutional neural network classification model B and randomly initialize the parameters of model B;

[0016] S9: Input the molecular subtype labels of the blocks obtained in S6 and S7 into model B for training. The trained model B is used as the molecular subtype prediction model.

[0017] In one implementation, step S1 includes:

[0018] The center of the spot in the histopathological image of the spatial transcriptome data is used as the center point of the cutting block. Each spot is cut around a preset length to obtain the block corresponding to each spot.

[0019] In one implementation, model A in step S3 uses ResNet50 as the feature extraction layer and modifies the output of the last fully connected layer to 2.

[0020] In one implementation, step S5 segments the WSI of the pathological image dataset, including:

[0021] Select the number of layers in WSI, and save images of the set size sequentially based on that layer, then divide the image into blocks;

[0022] Remove blank and irregular blocks from the segmented blocks. The method for removing blank blocks is as follows: calculate the average pixel value of each block. If the average pixel value of a block is less than a set threshold, the block is retained; otherwise, it is discarded. The method for removing irregular blocks is as follows: calculate whether the length and width of each block are equal to the set block size. If they are equal, the block is retained; otherwise, it is discarded.

[0023] In one implementation, step S6 includes:

[0024] Each block obtained from S5 is sequentially input into the trained model A to obtain the expression status of ESR1, ESR2, PGR, and ERBB2 for each block. When the prediction result is 1, it means that at least one gene is expressed in a block. When the prediction result is 0, it means that none of the four genes are expressed in the block. The block with a prediction result of 1 will be used to represent the features of the corresponding WSI and to construct a breast cancer molecular subtype prediction model.

[0025] In one implementation, model B in step S8 uses vgg16 as the feature extraction layer and modifies the output of the last fully connected layer to 4.

[0026] Based on the same inventive concept, a second aspect of the present invention provides an apparatus for constructing a breast cancer molecular subtype prediction model based on a screener and pathological images, comprising:

[0027] The pathological image segmentation module is used to segment histopathological images in spatial transcriptome data into multiple blocks;

[0028] The gene expression information extraction module is used to extract partial gene expression information from the segmented spatial transcriptome data, and obtain the gene information of ESR1, ESR2, PGR and ERBB2 of each spot corresponding to each block. If a block has at least one gene expression, the label is set to 1, otherwise it is 0. The spot is a spot, which is a data attribute of the spatial transcriptome data.

[0029] The Model A building module is used to build a convolutional neural network classification model A and randomly initialize the parameters of model A.

[0030] The filter training module is used to standardize the colors of the blocks obtained by the pathological image segmentation module, and input the blocks and corresponding gene labels into model A for training, so as to obtain the trained model A as the block filter.

[0031] The preprocessing module is used to perform block segmentation and color normalization on the WSI of the pathological image dataset;

[0032] The block selection module is used to predict gene expression in each block of the preprocessing module using the trained model A to obtain blocks containing the expression of ESR1, ESR2, PGR and ERBB2 genes.

[0033] The block annotation module is used to annotate the WSIs of the pathological image dataset in the preprocessing module to obtain the molecular subtype label corresponding to each WSI, which serves as the molecular subtype label for each block.

[0034] The Model B building module is used to build a convolutional neural network classification model B and randomly initialize the parameters of model B.

[0035] The molecular subtype prediction model training module is used to input the molecular subtype labels of the blocks obtained by the block screening module and the blocks obtained by the block labeling module into model B for training. The trained model B serves as the molecular subtype prediction model.

[0036] Based on the same inventive concept, a third aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed, implements the method described in the first aspect.

[0037] Based on the same inventive concept, a fourth aspect of the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method described in the first aspect.

[0038] Based on the same inventive concept, the fifth aspect of the present invention provides a method for predicting molecular subtypes of breast cancer, comprising: using a molecular subtype prediction model trained in the first aspect to predict the molecular subtype of a block, and using a majority voting method to obtain the final prediction result of WSI.

[0039] Compared with the prior art, the advantages and beneficial technical effects of the present invention are as follows:

[0040] This invention provides a method for constructing a breast cancer molecular subtype prediction model based on a filter and pathological images. The block filter, built using spatial transcriptomic data, can identify valuable blocks for molecular subtype prediction tasks without relying on pathologists to manually label tumor regions. This improves classification accuracy while saving computational resources. More importantly, the selected blocks are clinically interpretable. Clinically, immunohistochemistry relies on the expression of ER, PR, and HER2; similarly, this invention focuses only on the expression regions of the ESR1, ESR2, PGR, and ERBB2 genes corresponding to these three receptors. The filter constructed in this invention infers the expression of these genes. Built on spatial transcriptomic data, the filter can be directly applied to general H&E-stained histopathological images without retraining, maximizing the use of spatial transcriptomic technology. The molecular subtype prediction model is trained based on the blocks selected by the trained filter and their corresponding molecular subtype labels, improving the model's performance. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 This is a spatial transcriptome data map of breast cancer used in embodiments of the present invention, which includes H&E-stained histopathological images and gene expression data;

[0043] Figure 2 This is a patch image of H&E-stained histopathological images from spatial transcriptome data, as shown in this embodiment of the invention.

[0044] Figure 3 This is a graph showing the gene expression information of ESR1, ESR2, PGR, and ERBB2 extracted from spatial transcriptome data in an embodiment of the present invention.

[0045] Figure 4 This is a flowchart illustrating the process of constructing a block filter according to an embodiment of the present invention;

[0046] Figure 5 This is a WSI schematic diagram of a conventional H&E stained tissue pathology image used in an embodiment of the present invention;

[0047] Figure 6 This is a schematic diagram illustrating the cutting of WSI into patches in an embodiment of the present invention;

[0048] Figure 7 This is a flowchart illustrating the training and testing process of a breast cancer molecular subtype prediction method based on a spatial transcriptome screener and histopathological whole slide images in this embodiment of the invention.

[0049] Figure 8 This is a module structure diagram of the breast cancer molecular subtype prediction system based on spatial transcriptome screener and histopathological whole slide images in an embodiment of the present invention. Detailed Implementation

[0050] Through extensive research and practice, the inventors of this application have discovered that while immunohistochemistry is commonly used for molecular subtype diagnosis of breast cancer in clinical practice, this technique is cumbersome and time-consuming, hindering timely diagnosis. To address this issue, this invention utilizes artificial intelligence to construct a breast cancer molecular subtype prediction model based on a filter and pathological images. This model can directly predict the molecular subtype of breast cancer from H&E-stained histopathological images. WSI (Wide Slide Image), the most common storage format for predicting breast cancer molecular subtypes from H&E-stained histopathological images, contains millions of pixels. Since computers cannot directly process the entire image, WSI images are typically divided into many blocks. Although some related work has directly predicted molecular subtypes from H&E-stained histopathological images and used different strategies to utilize these blocks, it suffers from problems such as severe noise, over-reliance on manual labeling, and a lack of clinical interpretability. To solve these problems, this invention proposes a method for constructing a breast cancer molecular subtype prediction model based on a spatial transcriptome filter and histopathological whole-slide images. This method utilizes spatial transcriptome data to construct a block filter, and then uses this filter to select blocks from WSI images that are valuable for molecular subtype prediction. It is worth noting that the screener of this invention selects blocks containing genes such as ESR1, ESR2, PGR, and ERBB2. Clinically, immunohistochemistry determines receptors based on the expression of ER, PR, and HER2, which are primarily expressed by genes such as ESR1, ESR2, PGR, and ERBB2. Spatial transcriptomics technology provides aligned histopathological images and gene expression information, providing technical support for the data requirements of this invention. This invention utilizes a screener to select blocks containing the expression of genes such as ESR1, ESR2, PGR, and ERBB2, consistent with the clinical focus on ER, PR, and HER2 receptors in immunohistochemistry, thus possessing clinical interpretability. Furthermore, the blocks selected by this invention reduce noise and avoid reliance on manual labeling; therefore, this invention addresses the problems existing in current related methods. This invention utilizes spatial transcriptomics technology to link pathological images with genetic information. Based on research findings, this invention is the first to use spatial transcriptomics technology to combine histopathological images and genetic information to assist in predicting molecular subtypes of breast cancer using H&E-stained histopathological images, which is pioneering and forward-looking.

[0051] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0052] Example 1

[0053] This invention provides a method for constructing a breast cancer molecular subtype prediction model based on a filter and pathological images, including:

[0054] S1: Divide the histopathological images in the spatial transcriptome data into multiple blocks;

[0055] S2: Extract partial gene expression information from the segmented spatial transcriptome data to obtain the gene information of ESR1, ESR2, PGR and ERBB2 for each spot corresponding to each block. If at least one gene is expressed in a block, the label is set to 1, otherwise it is 0. The spot is a spot, which is a data attribute of the spatial transcriptome data.

[0056] S3: Build a convolutional neural network classification model A and randomly initialize the parameters of model A;

[0057] S4: Standardize the color of the blocks obtained from step S1, and input the blocks and corresponding gene labels into model A for training, so as to obtain the trained model A as the block filter;

[0058] S5: Perform block segmentation and color normalization on the WSI of the pathological image dataset;

[0059] S6: Use the trained model A to predict gene expression in each block in S5 to obtain blocks containing ESR1, ESR2, PGR and ERBB2 gene expression.

[0060] S7: Annotate the WSIs of the pathological image dataset in S5 to obtain the molecular subtype label corresponding to each WSI, which will be used as the molecular subtype label for each block.

[0061] S8: Build a convolutional neural network classification model B and randomly initialize the parameters of model B;

[0062] S9: Input the molecular subtype labels of the blocks obtained in S6 and S7 into model B for training. The trained model B is used as the molecular subtype prediction model.

[0063] Specifically, the English meanings involved are as follows: Whole Slide Image (WSI), patch, spot. Spatial transcriptome data is a type of gene expression data. "Spot" refers to a unique data attribute of spatial transcriptome data.

[0064] Please see Figure 1 The data provided refers to the spatial transcriptome data of breast cancer used in this embodiment of the invention. It includes H&E histopathological images and corresponding gene expression data. Each spot in the H&E histopathological image corresponds to a set of gene expression data, which represents the gene expression situation in that spot. Figure 2 The image shown is a slice of histopathological image of the spatial transcriptome data in the example.

[0065] Information on ESR1, ESR2, PGR, and ERBB2 can be extracted from the gene expression data corresponding to spot. If the expression level of at least one of these genes is greater than 0, the block is considered to be expressed (label set to 1); otherwise, it is considered not expressed (label set to 0).

[0066] Please see Figure 3 This is a schematic diagram of the gene expression information of ESR1, ESR2, PGR and ERBB2 extracted in this embodiment.

[0067] In one implementation, step S1 includes:

[0068] The center of the spot in the histopathological image of the spatial transcriptome data is used as the center point of the cutting block. Each spot is cut around a preset length to obtain the block corresponding to each spot.

[0069] In the specific implementation process, the histopathological images in the spatial transcriptome data are divided into many blocks, including: using the center of the spot of the histopathological image as the center point of the block. These center points used for each histopathological image are denoted as the set C = {c1, c2, ..., c...}. n}, where n represents the number of spots in the histopathological image. Each spot is cut around its center point in C with a length of 112 to form a 224*224 pixel block.

[0070] In one implementation, model A in step S3 uses ResNet50 as the feature extraction layer and modifies the output of the last fully connected layer to 2.

[0071] In the specific implementation process, a convolutional neural network classification model A is built, including using ResNet50 as the feature extraction model. Then, the output of the last fully connected layer is changed to 2, thus constructing a binary classification model. Specifically, ResNet50 is a residual network. Residual networks refer to the addition of residual learning to traditional convolutional neural networks, solving the problems of gradient vanishing and accuracy degradation (training set) in deep networks. This allows the network to become increasingly deeper while maintaining accuracy and controlling speed. The main idea of ​​residual networks is residual learning. The objective function consists of two parts: an identity function and a residual function, as shown in the following formula:

[0072] h(x) = x + (h(x) - x)

[0073] Where h(x) represents the objective function, x represents the identity function, and h(x)-x represents the residual function. The ReLU activation function is:

[0074] f(x) = max(0,x)

[0075] Here, x represents the input of the neuron, which can turn all negative values ​​into 0 while keeping positive values ​​unchanged. This unilateral inhibition function enables neurons in the neural network to have sparse activation.

[0076] The training block filter includes: putting the cut blocks and the corresponding 0 or 1 labels (representing no expression in ESR1, ESR2, PGR and ERBB2 genes and expression in at least one gene, respectively) into the constructed model for training. During training, the optimal value of the model is found according to the loss function and gradient descent method. The loss function adopts the cross-entropy loss function, and the gradient descent method adopts the Adam adaptive momentum estimation algorithm.

[0077] In practice, the training process uses a loss function and gradient descent to find the optimal model value. The loss function quantifies the difference between the model's prediction and the true label, while gradient descent finds a set of parameters that minimizes structural risk. The loss function used is the cross-entropy loss function, which is:

[0078]

[0079] Where y represents the actual label, and its value is either 0 or 1. J represents the probability that the sample prediction is positive; the greater the difference between the predicted output and y, the larger the J value.

[0080] Gradient descent employs the adaptive momentum estimation algorithm, Adam, which combines momentum estimation and RMSprop. It not only uses momentum as the parameter for updating the direction but also adaptively adjusts the learning rate. Specifically, it calculates the squared gradient g. t2 The exponentially weighted average is then used to calculate the gradient g. t The exponentially weighted average, the relevant formula is:

[0081] M t =β1M t-1 +(1-β1)g t

[0082] G t =β2G t-1 +(1-β2)g t ⊙g t

[0083] Where M t It can be viewed as the mean of the gradient (first moment), G t This can be viewed as the variance (second moment) of the gradient before subtracting the mean. β1 and β2 are the decay rates of the two moving averages, respectively, and g t The gradient for the update is calculated as follows:

[0084]

[0085]

[0086] in It is for M t Perform deviation correction. It is G t Perform deviation correction. and Let ε represent the decay rates of the two moving averages in the t-th iteration, where ε is a very small number. The parameter update difference Δθt of Adam is:

[0087]

[0088] In one specific implementation, the training process is as follows: specify the number of training rounds L, use the Adam optimizer to perform gradient descent calculation, and train for a total of L rounds.

[0089] Please see Figure 4 This is a flowchart of the building block filter in an embodiment of the present invention.

[0090] In one implementation, step S5 segments the WSI of the pathological image dataset, including:

[0091] Select the number of layers in WSI, and save images of the set size sequentially based on that layer, then divide the image into blocks;

[0092] Remove blank and irregular blocks from the segmented blocks. The method for removing blank blocks is as follows: calculate the average pixel value of each block. If the average pixel value of a block is less than a set threshold (e.g., 235), the block is retained; otherwise, it is discarded. The method for removing irregular blocks is as follows: calculate whether the length and width of each block are equal to the set block size. If they are equal, the block is retained; otherwise, it is discarded.

[0093] Please see Figure 5 , is the WSI used in the embodiments of the present invention. Figure 6 This is a schematic diagram of the blocks cut from WSI.

[0094] In one implementation, step S6 includes:

[0095] Each block obtained from S5 is sequentially input into the trained model A to obtain the expression status of ESR1, ESR2, PGR, and ERBB2 for each block. When the prediction result is 1, it means that at least one gene is expressed in a block. When the prediction result is 0, it means that none of the four genes are expressed in the block. The block with a prediction result of 1 will be used to represent the features of the corresponding WSI and to construct a breast cancer molecular subtype prediction model.

[0096] In one implementation, the molecular subtype tag of each block is derived from the tag of its corresponding WSI. For example, if a block is derived from a WSI tagged Luminal A, then the tag of that block is also Luminal A.

[0097] In one implementation, model B in step S8 uses vgg16 as the feature extraction layer and modifies the output of the last fully connected layer to 4.

[0098] Specifically, a convolutional neural network classification model B is built, including using VGG16 as the feature extraction model, and then changing the output of the last fully connected layer to the number of molecular subtypes, thereby constructing a multi-classification model.

[0099] In the specific implementation process, a convolutional neural network classification model B is trained, and the parameter selection during the training process is consistent with that of model A. After model B is trained, the final WSI prediction result is obtained using the majority voting method, which includes: a WSI block may be predicted as Luminal A, Luminal B, Her2, and Basal-like, and the class with the most blocks among these classes is considered the final WSI prediction class.

[0100] Please see Figure 7 This is a flowchart illustrating the training and testing process of a breast cancer molecular subtype prediction model based on a spatial transcriptome screener and histopathological whole slide images, as described in this invention example.

[0101] When filtering training blocks, the number of training rounds is set to 50, and the batch_size is set to 16.

[0102] To address the drawbacks of cumbersome and time-consuming procedures in immunohistochemistry for molecular subtype diagnosis of breast cancer in clinical practice, this invention utilizes artificial intelligence to directly predict the molecular subtype of breast cancer from H&E-stained histopathological images. First, a block filter is constructed using aligned histopathological images and gene expression data provided by spatial transcriptomics. This filter identifies blocks expressing at least one of the following genes: ESR1, ESR2, PGR, and ERBB2, for subsequent molecular subtype prediction. Table 1 shows the performance of the filter constructed using ResNet50 on the test set.

[0103] Table 1 shows the performance of the filter on the test set.

[0104] A block filter was constructed, and then blocks expressing at least one gene, ESR1, ESR2, PGR, and ERBB2, were selected using this block filter. These blocks were then used to train a molecular subtype prediction model. Experiments were conducted on spatial transcriptome data and a general H&E breast cancer histopathology image dataset. To evaluate the method of this invention, comparative experiments were conducted with all blocks and randomly selected blocks. Table 2 shows the experimental comparison results of molecular subtype prediction on the two datasets.

[0105] Table 2 shows the experimental comparison results of molecular subtype prediction on the two datasets.

[0106]

[0107] Patch-level accuracy represents the accuracy on the block-level test set, and image-level accuracy represents the accuracy on the WSI-level test set. The result of randomly selecting blocks is the average of 5 random selections.

[0108] Compared with existing technologies, the advantages of this invention are as follows:

[0109] 1. This invention uses artificial intelligence technology to predict the molecular subtype of breast cancer directly from H&E stained histopathological images, eliminating the need for clinical immunohistochemistry, greatly reducing the number of steps and improving diagnostic efficiency, while also reducing the workload of doctors.

[0110] 2. The block screener constructed using spatial transcriptome data in this invention can screen out valuable blocks for molecular subtype prediction tasks. It can screen out effective blocks without relying on pathologists to manually mark tumor regions, thereby improving classification accuracy and saving computational resources. More importantly, the screened blocks are clinically interpretable.

[0111] 3. The filter constructed in this invention is used to infer whether the genes ESR1, ESR2, PGR and ERBB2 are expressed. This filter is constructed on spatial transcriptome data. The constructed filter can be directly used on general H&E stained histopathological images without retraining the filter, thus making the most of spatial transcriptome technology.

[0112] Example 2

[0113] Based on the same inventive concept, this embodiment provides an apparatus for constructing a breast cancer molecular subtype prediction model based on a screener and pathological images, comprising:

[0114] The pathological image segmentation module is used to segment histopathological images in spatial transcriptome data into multiple blocks;

[0115] The gene expression information extraction module is used to extract partial gene expression information from the segmented spatial transcriptome data, and obtain the gene information of ESR1, ESR2, PGR and ERBB2 of each spot corresponding to each block. If a block has at least one gene expression, the label is set to 1, otherwise it is 0. The spot is a spot, which is a data attribute of the spatial transcriptome data.

[0116] The Model A building module is used to build a convolutional neural network classification model A and randomly initialize the parameters of model A.

[0117] The filter training module is used to standardize the colors of the blocks obtained by the pathological image segmentation module, and input the blocks and corresponding gene labels into model A for training, so as to obtain the trained model A as the block filter.

[0118] The preprocessing module is used to perform block segmentation and color normalization on the WSI of the pathological image dataset;

[0119] The block selection module is used to predict gene expression in each block of the preprocessing module using the trained model A to obtain blocks containing the expression of ESR1, ESR2, PGR and ERBB2 genes.

[0120] The block annotation module is used to annotate the WSIs of the pathological image dataset in the preprocessing module to obtain the molecular subtype label corresponding to each WSI, which serves as the molecular subtype label for each block.

[0121] The Model B building module is used to build a convolutional neural network classification model B and randomly initialize the parameters of model B.

[0122] The molecular subtype prediction model training module is used to input the molecular subtype labels of the blocks obtained by the block screening module and the blocks obtained by the block labeling module into model B for training. The trained model B serves as the molecular subtype prediction model.

[0123] Since the device described in Embodiment 2 of this invention is the same device used in implementing the method for constructing a breast cancer molecular subtype prediction model based on a screener and pathological images in Embodiment 1 of this invention, those skilled in the art can understand the specific structure and variations of this device based on the method described in Embodiment 1 of this invention, and therefore will not be repeated here. All devices used in the method of Embodiment 1 of this invention fall within the scope of protection of this invention.

[0124] Example 3

[0125] Based on the same inventive concept, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed, implements the method described in Embodiment 1.

[0126] Since the computer-readable storage medium described in Embodiment 3 of this invention is the same computer-readable storage medium used in the method for constructing a breast cancer molecular subtype prediction model based on a screener and pathological images in Embodiment 1 of this invention, those skilled in the art can understand the specific structure and variations of this computer-readable storage medium based on the method described in Embodiment 1 of this invention, and therefore will not be repeated here. All computer-readable storage media used in the method of Embodiment 1 of this invention fall within the scope of protection of this invention.

[0127] Example 4

[0128] Based on the same inventive concept, this application also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method in Embodiment 1.

[0129] Since the computer device described in Embodiment 4 of this invention is the same computer device used to implement the method for constructing a breast cancer molecular subtype prediction model based on a screener and pathological images in Embodiment 1 of this invention, those skilled in the art can understand the specific structure and variations of this computer device based on the method described in Embodiment 1 of this invention, and therefore will not be repeated here. All computer devices used in the method of Embodiment 1 of this invention fall within the scope of protection of this invention.

[0130] Example 5

[0131] Based on the same inventive concept, this application also provides a method for predicting molecular subtypes of breast cancer, which includes: using a molecular subtype prediction model trained as in Example 1 to predict the molecular subtype of a block, and using a majority voting method to obtain the final prediction result of WSI.

[0132] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0133] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0134] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the invention.

[0135] Obviously, those skilled in the art can make various modifications and variations to the embodiments of the present invention without departing from the spirit and scope of the embodiments of the present invention. Thus, if these modifications and variations to the embodiments of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention also intends to include these modifications and variations.

Claims

1. A method for constructing a breast cancer molecular subtype prediction model based on a filter and pathological images, characterized in that, include: S1: Divide the histopathological images in the spatial transcriptome data into multiple blocks, including: taking the center of the spot in the histopathological image in the spatial transcriptome data as the center point of the block, cutting each spot around a preset length to obtain the block corresponding to each spot; S2: Extract partial gene expression information from the segmented spatial transcriptome data to obtain the gene information of ESR1, ESR2, PGR and ERBB2 for each spot corresponding to each block. If at least one gene is expressed in a block, the label is set to 1, otherwise it is 0. The spot is a spot, which is a data attribute of the spatial transcriptome data. S3: Build a convolutional neural network classification model A and randomly initialize the parameters of model A; S4: Standardize the color of the blocks obtained from step S1, and input the blocks and corresponding gene labels into model A for training, so as to obtain the trained model A as the block filter; S5: Perform block segmentation and color normalization on the WSI of the pathological image dataset; S6: Use the trained model A to predict gene expression in each block of S5, obtaining blocks containing ESR1, ESR2, PGR, and ERBB2 gene expression. This includes: inputting each block obtained from S5 into the trained model A sequentially to obtain the expression status of ESR1, ESR2, PGR, and ERBB2 in each block. When the prediction result is 1, it indicates that at least one gene in a block is expressed; when the prediction result is 0, it indicates that none of the four genes are expressed in the block. The blocks with a prediction result of 1 will be used to represent the characteristics of the corresponding WSI and to construct a breast cancer molecular subtype prediction model. S7: Annotate the WSIs of the pathological image dataset in S5 to obtain the molecular subtype label corresponding to each WSI, which will be used as the molecular subtype label for each block. S8: Build a convolutional neural network classification model B and randomly initialize the parameters of model B; S9: Input the molecular subtype labels of the blocks obtained in S6 and S7 into model B for training. The trained model B is used as the molecular subtype prediction model.

2. The method for constructing a breast cancer molecular subtype prediction model based on a filter and pathological images as described in claim 1, characterized in that, In step S3, model A uses ResNet50 as the feature extraction layer and modifies the output of the last fully connected layer to 2.

3. The method for constructing a breast cancer molecular subtype prediction model based on a screener and pathological images as described in claim 1, characterized in that, Step S5 involves slicing the WSI of the pathological image dataset into blocks, including: Select the number of WSI layers, save images of the set size sequentially based on the selected WSI layers, and divide the images into blocks; Remove blank and irregular blocks from the segmented blocks. The method for removing blank blocks is as follows: calculate the average pixel value of each block. If the average pixel value of a block is less than a set threshold, the block is retained; otherwise, it is discarded. The method for removing irregular blocks is as follows: calculate whether the length and width of each block are equal to the set block size. If they are equal, the block is retained; otherwise, it is discarded.

4. The method for constructing a breast cancer molecular subtype prediction model based on a screener and pathological images as described in claim 1, characterized in that, In step S8, model B uses vgg16 as the feature extraction layer and modifies the output of the last fully connected layer to 4.

5. A device for constructing a molecular subtype prediction model for breast cancer based on a filter and pathological images, characterized in that, Based on the method described in claim 1, it includes: The pathological image segmentation module is used to segment histopathological images in spatial transcriptome data into multiple blocks; The gene expression information extraction module is used to extract partial gene expression information from the segmented spatial transcriptome data, and obtain the gene information of ESR1, ESR2, PGR and ERBB2 of each spot corresponding to each block. If a block has at least one gene expression, the label is set to 1, otherwise it is 0. The spot is a spot, which is a data attribute of the spatial transcriptome data. The Model A building module is used to build a convolutional neural network classification model A and randomly initialize the parameters of model A. The filter training module is used to standardize the colors of the blocks obtained by the pathological image segmentation module, and input the blocks and corresponding gene labels into model A for training, so as to obtain the trained model A as the block filter. The preprocessing module is used to perform block segmentation and color normalization on the WSI of the pathological image dataset; The block selection module is used to predict gene expression in each block of the preprocessing module using the trained model A to obtain blocks containing the expression of ESR1, ESR2, PGR and ERBB2 genes. The block annotation module is used to annotate the WSIs of the pathological image dataset in the preprocessing module to obtain the molecular subtype label corresponding to each WSI, which serves as the molecular subtype label for each block. The Model B building module is used to build a convolutional neural network classification model B and randomly initialize the parameters of model B. The molecular subtype prediction model training module is used to input the molecular subtype labels of the blocks obtained by the block screening module and the blocks obtained by the block labeling module into model B for training. The trained model B serves as the molecular subtype prediction model.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed, it implements the method as described in any one of claims 1 to 4.

7. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1 to 4.

8. A method for predicting molecular subtypes of breast cancer, characterized in that, This includes: using the molecular subtype prediction model trained in the method for constructing a breast cancer molecular subtype prediction model based on a screener and pathological images as described in any one of claims 1 to 4 to predict the molecular subtype of the block, and using the majority voting method to obtain the final WSI prediction result.

Images