Method and system for identifying hormone receptor status - Patents.com

Abstract

To provide a system and method of training a model capable of identifying the hormone receptor status of a subject by means of whole slide images (WSIs) of hematoxylin and eosin (H&E) stain of his / her biopsies, and to provide a method of identifying the hormone receptor status of a subject using the above mentioned method and model.SOLUTION: A method disclosed herein comprises a step S101 of acquiring multiple WSIs having known hormone receptor information, a step S102 of dividing each of the WSIs into a plurality of patches, a step S104 of selecting and combining patches expressing abnormal H&E stain to produce a combined image; and a step S105 of training a plurality of combined images independently with the aid of the known hormone receptor information of the WSIs so as to construct a model.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to the field of cancer diagnosis and treatment. More specifically, the disclosed invention relates to a method and system for determining and identifying a hormone receptor status of a subject based on a whole slide image (WSI) of hematoxylin and eosin (H&E) staining of his / her biopsy and treating the subject based on the identified hormone receptor status. [Background technology]

[0002] Breast cancer (BC) is the most common cancer affecting women worldwide and the most frequent cause of cancer death in women. In 2020, over 2 million women were diagnosed with BC worldwide, with over 680,000 BC deaths worldwide. Recent advances in early diagnostic approaches, including mammography, magnetic resonance imaging, ultrasound, computed tomography, positron emission tomography, and biopsy, have improved BC-related mortality and morbidity rates. However, limitations of these techniques, such as high cost and time-consuming procedures, prevent their universal application. There is an urgent need to develop efficient and highly sensitive methods to diagnose early BC.

[0003] Various biomarkers have been developed for the detection of BC. The majority of invasive breast cancers are hormone receptor positive--tumor cells grow in the presence of estrogen (ER) and / or progesterone (PR). Patients with hormone receptor positive tumors often benefit clinically from receiving hormone therapy that targets the ER / PR signaling pathway. In the traditional diagnostic workflow, biopsy samples collected from patients are thinly sectioned onto microscope slides for staining before visual diagnosis by a pathologist. Generally, hematoxylin and eosin (H&E) staining is used for primary diagnosis, followed by immunohistochemistry (IHC) staining for diagnostic confirmation and subtyping to determine the hormone receptor status (HRS) of the biopsy. Although hormone receptor status is an important tool for prognostic purposes and a predictor of endocrine therapy response, the process of HRS determination by visual inspection of slides has limitations. In addition to the disadvantages of being expensive and time-consuming, IHC test output is expressed in terms of color and is subject to variation due to sample quality, antibody source and clone, and technician proficiency. Also, the pathologist's decision-making process is inherently subjective and subject to human error. These factors lead to discrepancies in ERS determinations, and an estimated 20% of current IHC-based determinations of ER and PR tests may be inaccurate, putting these patients at risk for suboptimal treatment.

[0004] In view of the above, there is a need in the related art for improved methods and systems for determining a subject's hormone receptor status by hematoxylin and eosin (H&E) whole slide imaging (WSI) of his / her biopsy. Summary of the Invention

[0005] The following presents a simplified summary of the disclosure in order to provide the reader with a basic understanding. This summary is not an exhaustive overview of the disclosure, nor does it identify key / critical elements of the invention or delineate the scope of the invention. Its sole purpose is to present some concepts disclosed herein in a simplified form as a prelude to the more detailed description that is presented later.

[0006] As embodied and broadly described herein, it is an object of the present disclosure to provide diagnostic systems and methods implemented by the system that identify hormone receptor status in a subject such that the efficiency and accuracy of breast cancer diagnosis can be significantly improved.

[0007] In one aspect, the present disclosure is directed to a method for constructing a model for determining hormone receptor status from whole slide images (WSIs) of hematoxylin and eosin (H&E) staining of biopsies, the method comprising the steps of: (a) acquiring a plurality of WSIs of H&E staining of biopsies, each WSI including hormone receptor information; (b) dividing each of the WSIs of step (a) into a plurality of patches; (c) classifying each of the patches of step (b) into normal and abnormal H&E staining by performing tile extraction; (d) selecting and combining the classified patches indicating abnormal H&E staining of step (c) to generate a composite image of each of the WSIs of H&E staining; and (e) establishing a model by training the plurality of composite images generated independently of step (d) with the hormone receptor information of step (a). In step (a) of this method, the hormone receptor information comprises the positive or negative expression of a hormone receptor selected from the group consisting of estrogen receptor (ER), progesterone receptor (PR) and / or combinations thereof.

[0008] According to an embodiment of the present disclosure, in step (e) of the method, the multiple composite images are trained by performing a vector regularization complex matrix factorization (CMF) method, which includes the following steps: (e-1) obtaining a complex matrix from the complex values ​​of each composite image; (e-2) converting the complex matrix into a complex column vector for each composite image; and (e-3) classifying each composite image into a positive expression or a negative expression of a hormone receptor based on the similarity between the complex column vectors obtained in step (e-2).

[0009] In one embodiment, the step (e-3) is performed by implementing a k-nearest neighbor (k-NN) algorithm.

[0010] According to an embodiment of the present disclosure, steps (c), (d) and (e) of the method are performed by a deep learning algorithm.

[0011] According to certain embodiments of the present disclosure, the subject has or is suspected of having breast cancer.

[0012] In another aspect, the present disclosure relates to a method for determining hormone receptor status based on a hematoxylin and eosin (H&E) stained whole slide image (WSI) of a biopsy of a subject, the method comprising the steps of: (a) dividing the H&E stained WSI into a plurality of patches; (b) performing tile extraction to select and synthesize patches showing abnormal H&E staining to generate a test image; and (c) determining the hormone receptor status by processing the test image generated in step (b) within the model established by the method. In this method, the hormone receptor status comprises positive or negative expression of a hormone receptor selected from the group consisting of estrogen receptor (ER), progesterone receptor (PR) and / or combinations thereof.

[0013] According to an embodiment of the present disclosure, in step (c) of the method, the test image is processed by performing a vector regularized complex matrix factorization (CMF) method comprising the steps of: (c-1) obtaining a complex matrix from the complex values ​​of the test image; (c-2) converting the complex matrix into a complex column vector for the test image; and (c-3) classifying the test image into a positive or negative expression of hormone receptors based on the absolute distance between the complex column vector of the test image obtained in step (c-2) and the complex column vector of the synthetic image in the model established by the above method.

[0014] In one embodiment, the step (c-3) is performed by implementing a k-nearest neighbor (k-NN) algorithm.

[0015] In a preferred embodiment, the hormone receptor status further comprises the intensity of expression of the hormone receptor.

[0016] Alternatively or optionally, the vector regularization CMF method further comprises (c-4) determining the expression intensity of hormone receptors in the test image based on the ratio between complex column vectors corresponding to positive and negative expression, respectively, in the composite image in the model established by the above method.

[0017] According to an embodiment of the present disclosure, steps (b) and (c) of the method are performed by a deep learning algorithm.

[0018] According to certain embodiments of the present disclosure, the subject has or is suspected of having breast cancer.

[0019] In yet another aspect, the present disclosure is directed to a system including an image capture unit, a server, and a processor configured to perform the present method.

[0020] More specifically, the image acquisition unit is configured to acquire one or more candidate whole slide images (WSIs) of hematoxylin and eosin (H&E) staining of a biopsy from a subject. The server is configured to store the model established by the above method and receive one or more candidate WSIs of H&E staining transmitted from the image acquisition unit. The processor is further programmed with instructions to execute a method for determining a hormone receptor status of one or more candidate WSIs of H&E staining transmitted from the server, the method comprising: (a) dividing each of the candidate WSIs of H&E staining into a plurality of patches; (b) performing tile extraction to select and synthesize patches each showing abnormal H&E staining to generate a test image; and (c) determining a hormone receptor status by processing the test image generated in step (b) with the model stored in the server, the hormone receptor status including positive or negative expression of a hormone receptor selected from the group consisting of estrogen receptor (ER), progesterone receptor (PR) and / or combinations thereof.

[0021] In an embodiment of the present disclosure, in step (c) of the method, the test image is processed by performing a vector regularized complex matrix factorization (CMF) method comprising the steps of: (c-1) obtaining a complex matrix from the complex values ​​of the test image; (c-2) converting the complex matrix to a complex column vector for the test image; and (c-3) classifying the test image as positive or negative expression of hormone receptors based on the absolute distance between the complex column vector of the test image obtained in step (c-2) and the complex column vector of the synthetic image in the model stored in the server.

[0022] In one embodiment, step (c-3) of the method is performed by implementing a k-nearest neighbor (k-NN) algorithm.

[0023] In certain embodiments of the present disclosure, the hormone receptor status further comprises the intensity of expression of the hormone receptor.

[0024] Alternatively or optionally, the vector regularization CMF method further comprises (c-4) determining the expression intensity of the hormone receptor in the test image based on a ratio between complex column vectors corresponding to positive and negative expression, respectively, in the synthetic image in the model stored in the server.

[0025] In one embodiment, steps (b) and (c) of the method may be performed by a deep learning algorithm.

[0026] In yet another aspect, the present disclosure is directed to a method of assessing and treating breast cancer in a subject in need thereof, comprising the steps of: (a) obtaining a hematoxylin and eosin (H&E) stained whole slide image (WSI) from a biopsy of the subject, (b) assessing a hormone receptor status of the subject using the above method, and (c) administering an anti-cancer treatment to the subject based on the hormone receptor status of step (b), wherein the hormone receptor status comprises positive or negative expression and intensity of expression of a hormone receptor selected from the group consisting of estrogen receptor (ER), progesterone receptor (PR) and / or combinations thereof, and the anti-cancer treatment is selected from the group consisting of surgery, radiofrequency ablation, systemic chemotherapy, transarterial chemoembolization (TACE), immunotherapy, targeted drug therapy, hormone therapy, and combinations thereof.

[0027] According to certain embodiments of the present disclosure, the subject is a human.

[0028] With the above configuration, the method and system for determining and identifying the hormone receptor status of a subject can be performed in a rapid manner, thereby improving the efficiency and accuracy of breast cancer diagnosis.

[0029] Many of the attendant features and advantages of the present disclosure will become better understood by reference to the following detailed description considered in conjunction with the accompanying drawings.

[0030] The present description will be better understood from the following detailed description read in light of the accompanying drawings. [Brief description of the drawings]

[0031] [Figure 1] FIG. 1 is a flow chart of a method 10 according to one embodiment of the present disclosure. [Diagram 2] FIG. 2 is a diagram illustrating a system 20 according to another embodiment of the present disclosure. [Diagram 3] FIG. 3 is a flow chart of a method 30 implemented in the system 20 according to another embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0032] In accordance with common practice, the various components / elements depicted are not drawn to scale, but instead are drawn to best show the particular components / elements relevant to the present invention, and like numerals and designations in the various drawings are used to indicate like elements / portions.

[0033] The detailed description provided below in conjunction with the accompanying drawings is intended as a description of the present embodiment and is not intended to represent the only manner in which the present embodiment may be constructed or utilized. The description sets forth the functions of the present embodiment and the sequence of steps for constructing and operating the embodiment, although the same or equivalent functions and sequences may be accomplished by different embodiments.

[0034] 1.Definition For convenience of explanation, certain terms employed in the specification, examples, and appended claims are collected here. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0035] The singular forms "a," "an," and "the" are used herein to include the plural forms unless the context clearly dictates otherwise.

[0036] As used herein, the term "hormone receptor information" refers to the expression status of one or more hormone receptors, including, but not limited to, estrogen receptor (ER), progesterone receptor (PR), and combinations thereof. According to the present disclosure, the expression status can be positive or negative expression and / or expression intensity of the hormone receptor.

[0037] The terms "biopsy" or "biopsy specimen", as used interchangeably herein, refer to a tissue and / or cell sample taken from anywhere on or within a subject's body, including normal and / or abnormal skin and organs. In practice, biopsies are often obtained for pathology evaluation, where a full specimen of the biopsy is prepared and examined under a microscope. Thus, according to the present disclosure, a biopsy includes any specimen derived from tumor or cancer tissue, including breast cancer, preferably ER / PR positive breast cancer and ER / PR negative breast cancer.

[0038] The term "synthetic image" as used herein refers to an image that has been split into multiple patches for feature extraction, processed to remove non-feature, and then realigned and merged. According to the present disclosure, the synthetic image used to train the machine learning model is the "reference image", whereas the synthetic image acquired from the subject to identify his / her hormone receptor status is the "test image".

[0039] As used herein, the term "vector regularized complex matrix factorization (CMF)" refers to a matrix factorization method on the complex domain for image representation. Real-valued data is transformed to the complex domain, which decomposes a complex-valued matrix into two matrices, basis and coefficients, which are typically derived from the solution to an unconstrained optimization problem in the complex domain. According to the present disclosure, vector regularized CMF is used to simplify complex matrices and ultimately extract feature vectors from discriminative real image data.

[0040] The terms "complex number" and "complex value" are used interchangeably herein and are denoted by i, which is referred to as the imaginary unit and has the formula i 2=-1, and each complex number or value can be expressed in the form a+bi, where a and b are real numbers.

[0041] As used herein, the terms "treat," "treating," and "treatment" are interchangeable and include partially or completely preventing, ameliorating, alleviating, and / or managing symptoms, secondary disorders, or conditions related to breast cancer.

[0042] 2. Description of the Invention It has been reported that tumor morphology captured in hematoxylin and eosin (H&E) staining contains predictive signals for the tumor's molecular marker status, and pattern recognition algorithms can be applied to determine the molecular marker status directly from H&E stained images. In the field of pattern recognition, it is important that images are represented in a manner that highlights relevant information, and that high-dimensional data spaces are transformed into low-dimensional feature subspaces. Different modes of image representation produce different recognition results. Thus, a suitable representation can explicitly represent the latent structure of the data, reducing redundancy and computational costs. The present invention aims to provide an improved method and system that addresses the above-mentioned problems. Furthermore, the present invention also aims to develop an improved method of complex matrix factorization (CMF) for pattern recognition in H&E stained images to quantify the expression intensity of hormone receptor status.

[0043] 2.1 How to build a model for determining hormone receptor status A first aspect of the present disclosure is directed to a method for constructing a model for determining hormone receptor status from hematoxylin and eosin (H&E) stained whole slide images (WSI) of biopsies. See FIG.

[0044] 1 is a flowchart of a method 10 implemented on a computer or processor according to an embodiment of the present disclosure. The method 10 comprises the following steps, respectively indicated by reference characters S101 to S105 in FIG. S101: A step of acquiring a plurality of WSIs of H&E staining of a biopsy, each WSI including hormone receptor information; S102: Dividing each of the WSIs in step S101 into a plurality of patches; S103: classifying normal and abnormal H&E staining in each of the patches in step S102 by performing tile extraction; S104: Selecting and combining the classified patches showing abnormal H&E staining in step S103 to generate a combined image of each of the WSIs of H&E staining; and S105: A step of establishing a model by learning a plurality of composite images generated independently of step S104 using the hormone receptor information of step S101.

[0045] A biopsy in the method 10 is generally a tissue slice or cell sample obtained from a human body. According to an exemplary embodiment, the biopsy is a breast tissue slice obtained from a healthy or diseased subject. To build and train a model, multiple WSIs derived from the subject and independently containing known hormone receptor information are used in the training method 10. In practice, multiple whole slide images (WSIs) of H&E staining of biopsies may be collected from an existing database of a medical center (S101). According to the present disclosure, the hormone receptor information includes positive or negative expression of hormone receptors selected from the group consisting of estrogen receptor (ER), progesterone receptor (PR) and / or combinations thereof. Additionally or optionally, diagnostic information (e.g., cancer stage) corresponding to each subject may be collected. The WSIs are then automatically transferred to a device and / or system (e.g., computer or processor) having instructions embedded therein to perform the subsequent steps (S102-S105). In steps S102 and S103, each of the WSIs is divided into a number of patches (i.e., small pieces of the WSI), and each patch undergoes tile extraction to classify the normal and abnormal H&E staining exhibited for each of the patches. Tile extraction may be performed by an algorithm using pre-set pathological criteria known in the art to distinguish and extract patches with abnormal H&E staining from patches exhibiting normal H&E staining. The classified patches exhibiting abnormal H&E staining are then subjected to a synthesis process (S104) that leads to the generation of a composite image of each of the H&E stained WSIs. Note that the patches in one composite image are all from one individual, such that each composite image has known hormone receptor information and clinical diagnosis information indicating when it underwent the learning process described in step S105.

[0046] Then, in step S105, the multiple composite images are used to train a machine learning algorithm implemented in a computer (eg, a processor) using the hormone receptor information described above, thereby establishing a desired model.

[0047] According to an embodiment of the present disclosure, the synthetic image is trained by a vector regularized complex matrix factorization (CMF) method including steps S105a to S105c in FIG. S105a: obtaining a complex matrix from the complex values ​​of each composite image; S105b: converting the complex matrix to a complex column vector for each composite image; and S105c: Classifying each composite image as positive or negative expression of hormone receptors based on the similarity between the complex column vectors obtained in step S105b.

[0048] Assuming that each composite image is composed of multiple real numbers per pixel and has a pixel real data matrix X, the final goal of step S105a is to normalize and convert the real data matrix X into complex values, thereby obtaining a complex matrix Z for one composite image. Note that the complex values ​​in the composite image can be obtained by Fourier transformation, and Euler's formula is utilized to convert the Cartesian coordinate points into polar coordinates. In this approach, the vector of pixel intensity values ​​is first normalized and then transformed to a unit sphere using Euler's formula by mapping f from N-dimensional real space to N-dimensional complex space using equation (1).

number

[0049] According to the present disclosure, there are N patches (or images) to be trained, and each patch contains M real pixels, which means there are M complex values. Using equation (1), we find the column vector (i.e.,

number

[0050] We proceed to step S105b, the aim of which is to obtain a complex column vector for each composite image from the complex matrix Z. Note that there are N patches and M complex values, and the complex matrix Z for the composite image is Z∈C N×M To minimize the objective function, we use two submatrices W∈C N×K and V∈C K×M is factored from the complex matrix Z, where K is a constant. Thus, the two submatrices W and V are calculated using equation (2).

number

number

[0051] Finally, the relationships between the complex matrix Z and the sub-matrices W and V are obtained and are shown by equations (3) and (4). Z=WV (3) z=Wv (4)

[0052] Note that equation (3) can be transformed into equation (4) by substituting z into equation (1). Note that v represents the complex column vector of the image. Thus, a complex column vector v for each composite image is finally transformed from the complex matrix Z or submatrix V (S105b). In a preferred embodiment, the complex column vector v is a feature vector of each composite image, which allows further analysis of the composite image by its feature vectors. In summary, the complex column vector v (i.e., feature vector) of the composite image is obtained by training N patches and decomposing the complex matrix Z.

[0053] Once the complex column vector v is obtained, the vector regularization CMF method proceeds to a classification step (S105c). In step S105c, since each composite image has its complex column vector v and known hormone receptor information including the expression of hormone receptor, by comparing the similarity between multiple complex column vectors v, the expression pattern of hormone receptor in the composite image can be classified into either a positive group or a negative group. Alternatively, the current diagnosis information corresponding to the biopsy source serves the purpose of double confirmation. Thus, with the known hormone receptor information and / or diagnosis information in step S101, the image is trained to recognize and classify the positive or negative expression of hormone receptor, thereby collectively establishing a model for determining hormone receptor status mainly based on biopsy by H&E staining. It is worth noting that the algorithm suitable for use in step S105c of the method can be any well-known classification algorithm. In one embodiment, step S105c is performed by implementing a k-nearest neighbor (k-NN) algorithm.

[0054] A learning algorithm suitable for use in the present method or system, and in particular in steps S103 to S105, may be a deep learning algorithm, examples of which may include, but are not limited to, a convolutional neural network (CNN), a long short-term memory network (LSTM), a recurrent neural network (RNN), a generative adversarial network (GAN), a radial basis function network (RBFN), a multi-layer perceptron (MtLP), a self-organizing map (SOM), a deep belief network (DBN), a restricted Boltzmann machine (RBM), and an autoencoder.

[0055] By performing steps S101 to S105 described above, a well-trained model is established for determining hormone receptor status directly from H&E staining of biopsies.

[0056] 2.2 Systems and Methods for Identifying Hormone Receptor Status in a Subject A second aspect of the present disclosure is directed to a method and system for determining hormone receptor status based on candidate WSIs of H&E staining of a biopsy collected from a subject. See Figures 2 and 3.

[0057] 2 illustrates a system 20 comprising an image acquisition unit 210, a server 220, and a processor 230, each of which is coupled to the processor 230. According to the present disclosure, the image acquisition unit 210 is configured to acquire one or more candidate WSIs of H&E staining of a biopsy from a subject. In an embodiment, the image acquisition unit 210 is a microscope camera or a whole slide scanner. The server 220 is configured to store the model 2201 established by the method 10 described above (i.e., steps S101-S105). The processor 230 is configured to perform the image recognition of the present method for hormone receptor status identification.

[0058] In an embodiment, the server 220 and the processor 230 may be located separately as two individual devices or may be located within the same hardware. In an embodiment, the server 220 is communicatively connected to the image acquisition unit 210 and the processor 230 and configured to store one or more candidate WSIs of H&E staining received from the image acquisition unit 210 and analyzed by the processor 230. The processor 230 is programmed with instructions to execute a method for determining the hormone receptor status of the candidate WSIs of H&E staining using the model 2201 constructed in the server 220.

[0059] According to an embodiment of the present disclosure, the image acquisition unit 210, the server 220, and the processor 230 are communicatively connected to each other. The communication between the image acquisition unit 210, the server 220, and the processor 230 may be realized using various technologies. For example, the server 220 may be a cloud server that communicates with the image acquisition unit 210 and the processor 230 via a network (such as a local area network (LAN), a wide area network (WAM), the Internet, or a wireless network).

[0060] 3, which shows a flow chart of a method 30 implemented on the processor 230 for determining hormone receptor status of candidate WSIs of H&E staining of a biopsy collected from a subject having or suspected of having breast cancer. The method 30 includes the following steps (see reference numbers S301-S303 shown in FIG. 3). S301: Dividing a candidate H&E stained WSI into multiple patches; S302: Selecting and synthesizing patches showing abnormal H&E staining by performing tile extraction to generate a test image; and S303: Determining hormone receptor status by processing the test images generated in step S302 with the model 2201 established by the method 10.

[0061] According to the present disclosure, the hormone receptor status comprises positive or negative expression of the hormone receptor. In an alternative embodiment, the hormone receptor status further comprises the expression intensity of the hormone receptor. The hormone receptor used in the present method is selected from the group consisting of estrogen receptor (ER), progesterone receptor (PR) and / or combinations thereof.

[0062] Once the candidate WSI for H&E staining is obtained, the processor 230 performs tile extraction, in which the candidate WSI for H&E staining is divided into multiple patches, and then the patches showing abnormal H&E staining are selected and synthesized into one test image (steps S301-S302). Similar to steps S102 and S103 of method 10, the strategy utilized in steps S301 and S302 can be implemented by algorithms and pre-set pathological criteria well known in the art, preferably deep learning algorithms including, but not limited to, convolutional neural networks (CNNs), long short-term memory networks (LSTMs), recurrent neural networks (RNNs), generative adversarial networks (GANs), radial basis function networks (RBFNs), multi-layer perceptrons (MtLPs), self-organizing maps (SOMs), deep belief networks (DBNs), restricted Boltzmann machines (RBMs) and autoencoders. For the sake of simplicity, steps S301 and S302 will not be mentioned again here.

[0063] Proceeding to step S303, the test image is processed and compared with the reference information stored in the model 2201 to determine its hormone receptor status. According to one embodiment of the present disclosure, the test image is processed by performing a vector regularization complex matrix factorization (CMF) method performed by the processor 230. As shown in FIG. 3, the vector regularization CMF method generally includes the steps of (S303a) obtaining a complex matrix from the complex values ​​of the test image, (S303b) converting the complex matrix into a complex column vector for the test image, and (S303c) classifying the test image into a positive expression or a negative expression of the hormone receptor based on the absolute distance between the complex column vector of the test image obtained in step S303b and the complex column vector of the composite image in the model 2201 stored in the server 220.

[0064] After obtaining the complex column vector for the test image, a classification step (S303c) proceeds. Except for step S303c, the strategy utilized in steps S303a and S303b is similar to that described in steps S105a and S105b of method 10, which also aims to obtain feature vectors (i.e., complex column vectors) of real images (i.e., test images) for image recognition. The conversion of real data to complex values ​​has been described in detail in steps S105a-S105b and is omitted here for the sake of brevity.

[0065] The main difference between step S105c and step S303c is the classification strategy applied to the test image. Different from step S105, in step S303c, the complex column vector of the test image is utilized to compare with that of the composite image in the model 2201 by calculating the absolute distance between them. In general, the closer the distance between the complex column vectors of two images, the more similar the images are. Preferably, the calculation is performed by executing a k-nearest neighbor (k-NN) algorithm. If the complex column vector of the test image is closer to the complex column vector of the positive expression composite image than to the complex column vector of the negative expression composite image, the complex column vector of the test image is identified as a positive expression vector. Conversely, if it is closer to the negative expression composite image, it is identified as a negative expression vector. In practice, in the model 2201, the complex column vector of the test image is compared with that of all the composite images. Each comparison results in a single identification result, generating multiple identification results for an overall comparison of one test image. If the number of complex column vectors corresponding to positive hormone receptor expression is greater than that of negative expression, the test image is determined to have positive hormone receptor expression. Conversely, if the number of complex column vectors corresponding to negative expression is greater, the test image is determined to exhibit negative hormone receptor expression. Thus, step S303c accurately determines the positive or negative expression of hormone receptors in the test image derived from the subject.

[0066] In a preferred embodiment, the vector regularization CMF method for processing the test image further comprises a step of expression intensity determination (step S303d). Since the comparison in step S303c results in a number of discriminant numbers for positive and negative hormone receptor expression, the expression intensity is further calculated as the ratio of these numbers, thereby representing the proportion of positive or negative expression. For example, if the model has a total of 15 composite images used as reference images, the comparison of these composite images with the test image results in 10 positive expressions out of 15 and 5 negative expressions. As a result, the hormone receptor status is considered positive, and its expression intensity is represented as 10 / 15 (10 expressions out of 15). By performing this calculation, step S303d further determines the intensity of hormone receptor expression in the test image from the subject.

[0067] 2.3 Methods for Diagnosing and Treating Cancer The present disclosure also aims to provide diagnosis and treatment to subjects suspected of having or developing breast cancer. To this end, the above-described methods, models and systems can be utilized to assist clinicians with accurate determination of hormone receptor status. Thus, the present disclosure encompasses other aspects directed to methods of diagnosing and treating breast cancer in a subject.

[0068] According to an embodiment of the present disclosure, the method comprises: (a) obtaining a hematoxylin and eosin (H&E) stained whole slide image (WSI) from a biopsy of a subject; (b) determining the hormone receptor status of the subject using the methods and systems described above; (c) administering an anti-cancer treatment to the subject based on the hormone receptor status of step (b).

[0069] The method begins by obtaining an H&E stained whole slide image (WSI) from a biopsy of a subject, which may be a mammal, for example, a human, mouse, rat, hamster, guinea pig, rabbit, dog, cat, cow, goat, sheep, monkey, or horse. Preferably, the subject is a human. Any suitable tools and / or procedures may be implemented to obtain the biopsy and its WSI. In one embodiment, the biopsy is a breast biopsy stained with hematoxylin and eosin, and the WSI is imaged and collected by an image acquisition device, such as the image acquisition unit 210 of the system 20 (e.g., a microscope camera or a whole slide scanner).

[0070] The hormone receptor status of the subject can then be determined by the above-described method 30. According to the present disclosure, the hormone receptor status comprises the positive or negative expression and intensity of expression of a hormone receptor selected from the group consisting of estrogen receptor (ER), progesterone receptor (PR) and / or combinations thereof.

[0071] Once the hormone receptor status of a subject is determined and optionally confirmed, it can be used as an indicator for determining whether anti-cancer treatment should be administered to the subject. In an embodiment, if the WSI is determined to show positive ER / PR expression, the subject may develop or is at risk of developing ER / PR positive breast cancer, and anti-cancer treatment is administered to the subject to prevent or ameliorate symptoms associated with ER / PR positive breast cancer. In another embodiment, if the WSI is determined to show negative ER / PR expression, the subject may develop or is at risk of developing ER / PR negative breast cancer, and anti-cancer treatment is administered to the subject to prevent or ameliorate symptoms associated with ER / PR negative breast cancer.

[0072] Examples of anti-cancer treatments suitable for use in the present method (i.e., for administration to subjects whose hormone receptor status shows positive or negative expression) include, but are not limited to, surgery, radiofrequency ablation, systemic chemotherapy, transarterial chemoembolization (TACE), immunotherapy, targeted drug therapy, hormonal therapy, and combinations thereof. Any clinician may select a treatment suitable for use in the present method based on factors such as the particular condition being treated, the severity of the condition, individual patient parameters (including age, physical condition, size, sex, and weight), duration of treatment, the nature of concomitant treatment (if any), the specific route of administration, and similar factors within the knowledge and expertise of the medical practitioner.

[0073] With the above features, the method provides accurate determination and identification of hormone receptor status based primarily on WSI of H&E staining without immunohistochemistry (IHC) staining, thereby improving the accuracy and efficiency of breast cancer diagnosis and enabling identified patients to be appropriately treated. EXAMPLES

[0074] Materials and Methods Data collection A total of 166 estrogen receptor (ER) expression whole slide images (WSIs) of H&E staining of breast biopsies and a total of 163 progesterone receptor (PR) WSIs of H&E staining were obtained from the Department of Thoracic Surgery of Mackay Memorial Hospital (Taipei) and used to build the model for image recognition and validation.

[0075] Image Processing and Tile Extraction Each WSI obtained from the database was regularized by 8 times pixel size and then divided into 256 × 256 patches for further deep learning processing by utilizing the CNN model.

[0076] Vector Regularized Complex Matrix Factorization (CMF) for Pattern Recognition The present vector regularization CMF method aims to directly convert the constrained optimization problem into an unconstrained optimization problem. Based on the principle of Euler's formula, after normalizing the vector of pixel intensity values,

number

[0077] Next, the matrix Z ∈ C N×M Subject to the condition, the objective function

number

number

number

[0078] Example 1 Construction of an image recognition model according to the present disclosure This experiment aimed to provide a trained machine learning model for WSI recognition. For this purpose, two models for estrogen receptor (ER) recognition and progesterone receptor (PR) recognition were established, respectively, following the procedure described in the "Materials and Methods" chapter. Specifically, for ER recognition (Model I), a total of 133 WSIs were used, including 107 WSIs with positive ER expression and 26 WSIs with negative ER expression, and for PR recognition (Model II), a total of 130 WSIs were used, including 91 with positive PR expression and 39 with negative PR expression.

[0079] Example 2 Evaluation of the image recognition model Next, we verified the image recognition efficiency of the trained model and method for hormone receptor status determination in Example 1. For this purpose, 33 candidate WSIs including ER expression and PR expression were processed and input into the present model (i.e., Models I and II described above) utilizing the present vector regularization CMF method. In formulas (1) and (2), the values ​​of the two parameters α and λ were adjusted to the interval [0,2] and 0.01, respectively.

[0080] The model was found to have a recognition rate of 86% for ER and 81% for PR, respectively.

[0081] Using the present methods and systems, pathological biopsies obtained from patients can be automatically interpreted and identified without additional IHC testing, thereby improving the efficiency and accuracy of breast cancer diagnosis.

[0082] It should be understood that the above description of the embodiments is given by way of example only, and that various modifications may be made by those skilled in the art. The above specification, examples and data provide a complete description of the structure and use of the exemplary embodiments of the invention. Although various embodiments of the invention have been described above with a certain degree of particularity or with reference to one or more specific embodiments, those skilled in the art may make numerous modifications to the disclosed embodiments without departing from the spirit or scope of the invention.

Claims

1. 1. A method for constructing in silico a model for determining hormone receptor status from whole slide images (WSI) of hematoxylin and eosin (H&E) stained biopsies of a subject, comprising: (a) acquiring a plurality of WSIs of H&E staining of the biopsy, each WSI including hormone receptor information; (b) dividing each of the WSIs of step (a) into a plurality of patches; (c) classifying normal and abnormal H&E staining in each of said patches of step (b) by performing tile extraction; (d) selecting and combining the classified patches indicative of the abnormal H&E staining of step (c) to generate a combined image of each of the WSIs of the H&E staining; (e) establishing the model by training a plurality of synthetic images generated independently of step (d) using the hormone receptor information of step (a); Equipped with The hormone receptor information of step (a) comprises positive or negative expression of a hormone receptor selected from the group consisting of estrogen receptor (ER), progesterone receptor (PR) and / or combinations thereof; A method, comprising the steps of (a) to (e) being performed one or more times by the computer based on instructions sent from a processor programmed within the computer.

2. In step (e), the plurality of composite images are (e-1) obtaining a complex matrix from the complex values ​​of each composite image; (e-2) converting the complex matrix into a complex column vector for each composite image; (e-3) classifying each composite image into a positive expression or a negative expression of the hormone receptor based on the similarity between the complex column vectors obtained in step (e-2); The method of claim 1 , wherein the vector-regularized complex matrix factorization (CMF) method comprises:

3. 3. The method of claim 2, wherein step (e-3) is performed by implementing a k-nearest neighbor (k-NN) algorithm.

4. The method of claim 1 , wherein steps (c), (d) and (e) are performed by a deep learning algorithm.

5. The method of claim 1 , wherein the subject has or is suspected of having breast cancer.

6. 1. A method for determining hormone receptor status in silico based on whole slide images (WSI) of hematoxylin and eosin (H&E) stained biopsies of a subject, comprising: (a) dividing the H&E stained WSI into a number of patches; (b) performing tile extraction to select and combine patches showing abnormal H&E staining to generate a test image; (c) determining the hormone receptor status by processing the test images generated in step (b) within the model established by the method of claim 1; Equipped with The hormone receptor status comprises positive or negative expression of a hormone receptor selected from the group consisting of estrogen receptor (ER), progesterone receptor (PR), and / or combinations thereof; A method in which the computer performs steps (a) to (c) one or more times based on instructions sent from a processor programmed within the computer.

7. In step (c), the test image is (c-1) obtaining a complex matrix from the complex values ​​of the test image; (c-2) converting the complex matrix into a complex column vector for the test image; (c-3) classifying the test image into positive or negative expression of the hormone receptor based on the absolute distance between the complex column vector of the test image obtained in step (c-2) and the complex column vector of the composite image in the model established by the method of claim 1; The method of claim 6, wherein the vector-regularized complex matrix factorization (CMF) method comprises:

8. 8. The method of claim 7, wherein step (c-3) is performed by implementing a k-nearest neighbor (k-NN) algorithm.

9. 9. The method of claim 8, wherein the hormone receptor status further comprises an expression intensity of the hormone receptor.

10. The method of claim 9, wherein the vector regularization CMF method further comprises: (c-4) determining the expression intensity of the hormone receptor in the test image based on a ratio between the numbers of complex column vectors corresponding to positive and negative expression, respectively, in the synthetic image of the model established by the method of claim 1.

11. The method of claim 6 , wherein steps (b) and (c) are performed by a deep learning algorithm.

12. The method of claim 6 , wherein the subject has or is suspected of having breast cancer.

13. 1. A system for identifying a hormone receptor status of a subject, comprising: an image acquisition unit configured to acquire one or more candidate whole-slide images (WSIs) of hematoxylin and eosin (H&E) stained biopsies from the subject; A server configured to store a model established by the method of claim 1 and to receive one or more candidate WSIs of the H&E stain transmitted from the image acquisition unit; a processor programmed with instructions to execute a method for determining the hormone receptor status of one or more candidate WSIs of the H&E stains transmitted from the server; The method comprises: (a) dividing each of the H&E stained candidate WSIs into a plurality of patches; (b) performing tile extraction to select and combine patches each exhibiting abnormal H&E staining to generate a test image; (c) determining the hormone receptor status by processing the test images generated in step (b) with the model stored on the server, wherein the hormone receptor status comprises positive or negative expression of a hormone receptor selected from the group consisting of estrogen receptor (ER), progesterone receptor (PR), and / or combinations thereof.

14. In step (c) of the method, the test image is (c-1) obtaining a complex matrix from the complex values ​​of the test image; (c-2) converting the complex matrix into a complex column vector for the test image; (c-3) classifying the test image into positive or negative expression of the hormone receptor based on the absolute distance between the complex column vector of the test image obtained in step (c-2) and the complex column vector of the synthetic image in the model stored in the server; 14. The system of claim 13, wherein the vectors are processed by performing a vector regularized complex matrix factorization (CMF) method comprising:

15. 15. The system of claim 14, wherein step (c-3) is performed by executing a k-nearest neighbor (k-NN) algorithm.

16. The system of claim 14 , wherein the hormone receptor status further comprises an expression intensity of the hormone receptor.

17. 17. The system of claim 16, wherein the vector regularization CMF method further comprises: (c-4) determining the expression intensity of the hormone receptor in the test image based on a ratio between the numbers of complex column vectors corresponding to positive expression and negative expression, respectively, in the synthetic image in the model stored in the server.

18. 14. The system of claim 13, wherein steps (b) and (c) are performed by a deep learning algorithm.

Images