Method and system for identifying hormone receptor state

Abstract

The present disclosure relates to an improved system and a method performed by the system for training a model that can identify a hormone receptor status of an individual through a hematoxylin-eosin stained full slide image of a living tissue section of the individual. The method comprises the following steps: (a) obtaining a plurality of full-slide images with known hormone receptor information; (b) dividing each full-slide image into a plurality of image blocks; (c) picking out the Hamp with abnormal performance; e, combining the dyed image blocks into a combined image; and (d) respectively training the plurality of merged images by means of known hormone receptor information of the full-slide image so as to construct the model. The present disclosure also relates to methods of identifying hormone receptor status of an individual using the foregoing methods and models built in the system of the present disclosure.

Description

Technical Field

[0001] The present disclosure relates to the field of cancer diagnosis and treatment. The present disclosure more particularly relates to methods and systems for determining and identifying the hormone receptor status of an individual based on hematoxylin-eosin (H&E) stained whole slide images of a biopsy tissue section of the individual, and treating the individual based on the identified hormone receptor status. Background Art

[0002] Breast cancer (BC) is the most common cancer affecting women worldwide and the most common cause of cancer death in women. According to statistics, in 2020, more than two million women worldwide will be diagnosed with breast cancer, and more than 680,000 will die from breast cancer. Recent advances in early diagnostic methods including mammography, magnetic resonance imaging, ultrasound imaging, computed tomography imaging, positron emission tomography imaging, and biopsy have improved breast cancer-related mortality and morbidity. However, the aforementioned technologies have limitations such as high prices and time-consuming, and therefore cannot be widely popularized. Therefore, the relevant field needs to develop efficient and highly sensitive methods for diagnosing early breast cancer.

[0003] Among these, a variety of biomarkers have been used to detect breast cancer. Most invasive breast cancers are hormone receptor positive, meaning that the tumor cells grow in the presence of estrogen (ER) and / or progesterone (PR). Clinically, patients with hormone receptor positive tumors often benefit from hormone therapy that targets the ER / PR signaling pathway. In conventional diagnostic procedures, biopsy samples collected from patients are thinly sliced ​​into stainable slides and then visually diagnosed by a pathologist. Typically, hematoxylin and eosin (H&E) staining is used for initial diagnosis, and immunohistochemistry (IHC) staining is then used for diagnosis confirmation and typing to interpret the hormone receptor status (HRS) of the biopsy. Although hormone receptor status is a key tool for prognostic purposes and a predictor of response to endocrine therapy, the process of identifying hormone receptor status by visual inspection of slides still has its limitations. In addition to disadvantages such as being expensive and time-consuming, immunohistochemical staining results are expressed in colors that can vary depending on sample quality, antibody source and strain, and technician skill level. Furthermore, the pathologist’s decision-making process is inherently subjective and can lead to human error. These factors contribute to inconsistent interpretation of hormone receptor status; currently an estimated 20% of estrogen and progesterone tests based on immunohistochemical staining are inaccurate, placing these patients at risk for suboptimal treatment.

[0004] In view of the above, the relevant field is in urgent need of an improved method and system for determining the hormone receptor status of an individual by means of H&E stained whole slide images of individual biopsy tissue sections. Summary of the invention

[0005] In order to provide the reader with a basic understanding, a brief summary of the present disclosure is provided below. This summary is not an extensive overview of the present disclosure, nor is it intended to identify the key / essential components of the present invention or to outline the scope of the present invention. Its sole purpose is to present some concepts of the present disclosure in a simplified conceptual form as a preface to a more detailed description presented later.

[0006] As embodied and broadly described herein, an object of the present disclosure is to provide an improved diagnostic system and method for identifying the hormone receptor status of an individual performed by the system, whereby the efficiency and accuracy of breast cancer diagnosis may be greatly improved.

[0007] The present disclosure relates to a method for establishing a model for determining hormone receptor status using hematoxylin-eosin (H&E) stained whole slide images of a biopsy tissue section. The method comprises: (a) obtaining a plurality of H&E stained whole slide images of a biopsy tissue section, wherein each of the whole slide images comprises hormone receptor information; (b) dividing each of the whole slide images of step (a) into a plurality of image blocks; (c) distinguishing normal and abnormal H&E staining in each of the image blocks of step (b) by performing image block extraction; (d) selecting and merging image blocks distinguished as showing abnormal H&E staining by step (c) to generate merged images of each of the H&E stained whole slide images; and (e) using the hormone receptor information of step (a) to train a plurality of merged images respectively generated from step (d) to establish the model. In step (a) of the method, the hormone receptor information comprises the positive or negative expression of a hormone receptor, and the hormone receptor is selected from the group consisting of estrogen receptor (ER), progesterone receptor (PR) and / or a combination thereof.

[0008] According to some embodiments of the present disclosure, in step (e) of the method of the present invention, a plurality of merged images are trained by means of a column vector regularized complex matrix factorization (CMF) method. The vector regularized CMF method mainly comprises the following steps: (e-1) obtaining a complex matrix from the complex numbers of each merged image; (e-2) converting the complex matrix into a plurality of column vectors corresponding to each merged image; and (e-3) distinguishing each merged image as a positive expression or a negative expression of a hormone receptor based on the similarity between the plurality of column vectors obtained in step (e-2).

[0009] In some embodiments, step (e-3) is performed by executing a k-nearest neighbor (k-NN) algorithm.

[0010] According to certain embodiments of the present disclosure, steps (c), (d) and (e) of the method of the present invention may be performed with the aid of a deep learning algorithm.

[0011] According to certain embodiments of the present disclosure, the individual suffers from or is suspected of suffering from breast cancer.

[0012] Another aspect of the present disclosure is a method for determining hormone receptor status based on hematoxylin-eosin (H&E) stained whole slide images of individual biopsy tissue sections. The method comprises: (a) dividing the H&E stained whole slide image into a plurality of image blocks; (b) selecting image blocks showing abnormal H&E staining by means of block capture and merging them to generate images to be tested; and (c) processing the images to be tested generated in step (b) within the model established by the above method to determine hormone receptor status. In the present method, the hormone receptor status comprises a positive expression or a negative expression of a hormone receptor, wherein the hormone receptor is selected from the group consisting of estrogen receptor (ER), progesterone receptor (PR) and / or a combination thereof.

[0013] According to some embodiments of the present disclosure, in step (c) of the method of the present invention, the image to be tested is processed by means of a column vector regularized complex matrix factorization (CMF) method, and the vector regularized complex matrix factorization method includes: (c-1) obtaining a complex matrix from the complex numbers of the image to be tested; (c-2) converting the complex matrix into a number of column vectors of the image to be tested; and (c-3) based on the absolute distance between the several column vectors of the image to be tested obtained in step (c-2) and the several column vectors of the merged image in the model established by the aforementioned method, distinguishing whether the image to be tested is hormone receptor positive or negative.

[0014] In some embodiments, step (c-3) is performed by executing a k-nearest neighbor (k-NN) algorithm.

[0015] In a preferred embodiment, the hormone receptor status also includes the expression intensity of the hormone receptor.

[0016] Optionally or optionally, the vector regularized CMF method further comprises (c-4) determining the expression intensity of the hormone receptor in the image to be tested based on the ratio between several column vectors of the merged image corresponding to positive expression and negative expression, respectively, in the model established by the aforementioned method.

[0017] According to some embodiments of the present disclosure, steps (b) and (c) of the method may be performed with the aid of a deep learning algorithm.

[0018] According to certain embodiments of the present disclosure, the individual suffers from or is suspected of suffering from breast cancer.

[0019] Yet another aspect of the present disclosure is related to a system, which includes an image capture unit, a server, and a processor, and is configured to implement the aforementioned method of the present invention.

[0020] More specifically, the image capture unit is configured to collect one or more candidate hematoxylin-eosin (H&E) stained whole slide images of a biopsy tissue section from an individual. The server is configured to store the model established by the aforementioned method and receive one or more candidate H&E stained whole slide images transmitted to the image capture unit. In addition, the processor is programmed by instructions to execute a method for determining the hormone receptor status of the one or more candidate H&E stained whole slide images transmitted from the server, wherein the method comprises: (a) dividing each of the candidate H&E stained whole slide images into a plurality of image blocks; (b) selecting and merging the image blocks showing abnormal H&E staining by performing image block capture to form a test image; and (c) determining the hormone receptor status by processing the test image generated by step (b) by using the model stored in the server, wherein the hormone receptor status comprises a positive expression or a negative expression of a hormone receptor, wherein the hormone receptor is selected from the group consisting of estrogen receptor (ER), progesterone receptor (PR) and / or a combination thereof.

[0021] According to some embodiments of the present disclosure, in step (c) of the present invention, the image to be tested is processed by means of a column vector regularized complex matrix factorization (CMF) method, which includes: (c-1) obtaining a complex matrix from the complex numbers of the image to be tested; (c-2) converting the complex matrix into a number of column vectors of the image to be tested; and (c-3) based on the absolute distance between the number of column vectors of the image to be tested obtained in step (c-2) and the number of column vectors of the merged image in the model stored in the server, distinguishing whether the image to be tested is hormone receptor positive or negative.

[0022] In some specific embodiments, step (c-3) of the method of the present disclosure is performed by executing a k-nearest neighbor (k-NN) algorithm.

[0023] In some embodiments, hormone receptor status also includes the expression intensity of hormone receptors.

[0024] Optionally or optionally, the vector regularized CMF method further comprises step (c-4): determining the hormone receptor expression intensity of the image to be tested based on the ratio between a number of column vectors of the merged image corresponding to positive expression and negative expression, respectively, within the model stored in the server.

[0025] In some specific embodiments, steps (b) and (c) of the disclosed method may be performed with the aid of a deep learning algorithm.

[0026] In yet another aspect, the present disclosure relates to a method for determining and treating breast cancer in an individual in need thereof. The method comprises: (a) obtaining an H&E-stained whole slide image of a biopsy tissue section from the individual; (b) determining the hormone receptor status of the individual by using the aforementioned method; and (c) administering an anti-cancer therapy to the individual based on the hormone receptor status of step (b), wherein the hormone receptor status comprises positive or negative expression of a hormone receptor selected from the group consisting of estrogen receptor (ER), progesterone receptor (PR) and / or a combination thereof, and the intensity of the expression; and the anti-cancer therapy is selected from the group consisting of surgery, radiofrequency ablation, systemic chemotherapy, transarterial chemoembolization (TACE), immunotherapy, targeted drug therapy, hormone therapy, and a combination thereof.

[0027] According to certain embodiments of the present disclosure, the individual is a human.

[0028] Through the above configuration, the method and system for determining and identifying the hormone receptor status of an individual can be quickly executed, thereby improving the efficiency and accuracy of breast cancer diagnosis.

[0029] After referring to the following embodiments, those skilled in the art to which the present invention belongs may easily understand the basic spirit and other invention purposes of the present invention, as well as the technical means and implementation aspects adopted by the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] To make the above and other objects, features, advantages and embodiments of the present invention more clearly understood, the following are the descriptions of the accompanying drawings:

[0031] Figure 1 is a flow chart of a method 10 according to an embodiment of the present disclosure;

[0032] Figure 2 is a schematic diagram of a system 20 according to another embodiment of the present disclosure; and

[0033] Figure 3 FIG. 4 is a flow chart of a method 30 implemented by the system 20 according to another embodiment of the present disclosure.

[0034] According to the usual working method, the various components and features in the figure are not drawn to scale, and the drawing method is to present the specific features and components related to the present invention in the best way. In addition, the same or similar component symbols are used to refer to similar components / parts between different drawings. DETAILED DESCRIPTION

[0035] In order to make the description of the present disclosure more detailed and complete, the following is an illustrative description of the implementation aspects and specific embodiments of the present invention: However, this is not the only form of implementing or using the specific embodiments of the present invention. The implementation covers the features of multiple specific embodiments and the method steps and sequences used to construct and operate these specific embodiments. However, other specific embodiments can also be used to achieve the same or equivalent functions and step sequences.

[0036] 1. Definition

[0037] For ease of explanation, specific terms described in this specification, embodiments and claims are systematically described here. Unless otherwise defined herein, all technical and scientific terms herein have the same meaning as those commonly known to those skilled in the art in the art to which the present invention belongs.

[0038] As used herein, the singular forms "a," "an," and "the" include plural forms as well, unless the context clearly indicates otherwise.

[0039] As used herein, "hormone receptor information" refers to the expression status of one or more hormone receptors, including but not limited to estrogen receptor (ER), progesterone receptor (PR), and combinations thereof. According to the present disclosure, the expression status can be positive expression or negative expression, and / or the expression intensity of the hormone receptor.

[0040] The terms "biopsy", "biological specimen", "biopsy specimen" or "biological specimen" are used interchangeably herein to refer to tissue and / or cell samples taken from a subject or anywhere in the body (including normal and / or abnormal skin or organs). In practice, biopsies are usually used for pathological evaluation, and usually sufficient biopsy specimens are prepared and observed under a microscope. According to the present disclosure, a biopsy thus contains any specimen derived from a tumor or cancerous tissue (including breast cancer, preferably ER / PR-positive breast cancer and ER / PR-negative breast cancer).

[0041] The term "merged image" herein refers to the process of segmenting an image or picture into multiple image blocks for feature extraction, and then reorganizing the merged image after processing to remove featureless image blocks. According to one embodiment of the present disclosure, the merged image is used as a "reference image" to train the machine learning model, and the merged image obtained from an individual to identify the hormone receptor status of the individual is used as a "test image."

[0042] As used herein, the term "vector-regularized complex matrix factorization (CMF)" refers to a complex factorization method in the complex domain for image representation. Real data is transformed into the complex domain, and the complex matrix is ​​decomposed into a basis matrix and a coefficient matrix, which are usually derived from the solution of an unconstrained optimization problem in the complex domain. According to the present disclosure, vector-regularized complex matrix factorization can be used to simplify the complex matrix and ultimately extract features from real image data with high discrimination.

[0043] This article uses the term "complex number" or "complex value" interchangeably to refer to an element in the number system that extends a real number by a certain element, denoted by i, called the imaginary unit, and satisfies the equation i 2 =-1; every complex number can be expressed in the form of a+bi, where a and b are both real numbers.

[0044] As used herein, the terms "treatment," "therapy," and "anti-cancer treatment" are used interchangeably and encompass the partial or complete prevention, amelioration, alleviation and / or management of symptoms, secondary diseases or conditions associated with breast cancer. 2. Specific implementation methods

[0046] It is reported that tumor morphology captured in hematoxylin-eosin (hereinafter referred to as H&E) staining contains predictive signals of molecular marker status, and pattern recognition algorithms can be directly applied to determine the status of molecular markers from H&E stained images. In the field of pattern recognition, it is important to represent images in a way that emphasizes relevant information and transforms high-dimensional data space into a low-dimensional feature subspace. Different image representations produce different recognition results. Therefore, a suitable representation method can clearly express the potential structure of the data and reduce redundancy and computational cost. Based on the above, the present invention aims to provide an improvement and system that can solve the above problems. Furthermore, the present invention also aims to develop an improved complex matrix decomposition (CMF) method for pattern recognition of H&E stained images, so as to achieve the intensity of expression of quantitative hormone receptor status.

[0047] 2.1 Methods for establishing models that can measure hormone receptor status

[0048] The first aspect of the present disclosure is a method for establishing a model by measuring the hormone receptor status by whole slide imaging of hematoxylin-eosin (hereinafter referred to as H&E) stained biopsy tissue sections. Figure 1 .

[0049] Figure 1A flowchart of a method 10 implemented by a computer or a processor according to an embodiment of the present disclosure is presented. The method 10 comprises the following steps, which are respectively Figure 1 The components are represented by the symbols S101 to S105: S101: Acquire a plurality of H&E-stained whole-slide images of living tissue sections, wherein each of the whole-slide images contains hormone receptor information; S102: Divide each of the whole slide images in step S101 into a plurality of image patches; S103: distinguishing normal or abnormal H&E staining of each image block in step S102 by performing tiles extraction; S104: selecting and merging the image blocks showing abnormal H&E staining after being distinguished in step S103 to generate a merged image of each H&E-stained whole slide image; and S105: Using the hormone receptor information of step S101 to train a number of merged images respectively generated from step S104, thereby establishing the model of the present invention.

[0050] The biopsy of the method 10 of the present disclosure is usually a piece of tissue or cell sample obtained from the human body. According to an exemplary embodiment, the biopsy is a piece of breast tissue obtained from a healthy or diseased individual. In order to establish and train the model, multiple whole slide images derived from the individual and containing known hormone receptor information are used for the training method 10 of the present disclosure. Specifically, H&E stained whole slide images of multiple biopsy sections can be collected from the existing database of the medical center (S101). According to the present disclosure, the hormone receptor information includes positive or negative expression of hormone receptors, wherein the hormone receptors are selected from the group consisting of estrogen receptors (hereinafter referred to as ER), progesterone receptors (hereinafter referred to as PR) and / or the aforementioned combination. Diagnostic information corresponding to each individual (e.g., cancer stage) can also be collected additionally or selectively. Then, the whole slide image is automatically forwarded to a device and / or system (e.g., a computer or processor) that has built-in instructions for executing subsequent steps (S102 to S105). In steps S102 and S103, each whole slide image is divided into a number of image blocks (i.e., a small whole slide image), and each image block is subjected to block capture to distinguish whether the H&E staining presented in each image block is normal or abnormal. Block capture can be performed by an algorithm with the help of pathological criteria known in the art, so as to distinguish and select image blocks with abnormal H&E staining from those image blocks presenting normal H&E staining. The image blocks presenting abnormal H&E staining after classification are then merged to generate a merged image of each H&E stained whole slide image, that is, a merged image corresponding to each H&E stained whole slide image (S104). It should be noted that all image blocks in a merged image are derived from the same individual, so that each composite image has known hormone receptor information and clinical diagnosis information for reference when these merged images are subjected to the training procedure described in step S105.

[0051] Next, in step S105, with the aid of the hormone receptor information, a plurality of (or a plurality of) merged images are used to train a machine learning model built into a computer (eg, a processor), thereby establishing the model of the present invention.

[0052] According to certain embodiments of the present disclosure, the merged image can be trained by means of a vector regularized complex matrix decomposition (hereinafter referred to as CMF) method, the method comprising: Figure 1 The steps shown and marked with reference numerals S105a to S105c are: S105a: Obtain a complex matrix from the complex values ​​of each merged image; S105b: converting the complex matrix into a number of column vectors corresponding to each merged image; and S105c: Based on the similarities between the column vectors obtained in step S105b, distinguish each of the merged images as hormone receptor positive or negative.

[0053] Since each merged image is composed of multiple real numbers for each pixel and has a real data matrix X of the pixels, the ultimate purpose of step S105a is to normalize and convert the real data matrix X into complex numbers, thereby generating a complex matrix Z corresponding to the merged image. It should be noted that the complex values ​​of the merged image of the present invention can be obtained from Fourier transform, and Euler's formula can be used to convert a point from Cartesian coordinates to polar coordinates. In this scheme, the vector value of the pixel intensity is first normalized, and then converted to a unit sphere by mapping f from N-dimensional real space to N-dimensional complex space using Euler's formula with the help of equation (1), where x t represents an N-dimensional vector containing the expressing image X in lexicographic ordering t , x t (c)∈[0,1] and α∈R + .

[0054] According to the present disclosure, there are N image blocks (or images) that can be trained, and each image block contains M real pixels, which means there will be M complex values. By using equation (1), a column vector corresponding to the complex value M can be obtained (that is, the right side of equation (1) is represented by express).

[0055] Then proceed to step S105b, the purpose of which is to obtain a number of column vectors of each merged image from the complex matrix Z. Note that when there are N image blocks and M complex values, the number of column vectors Z of the merged image is represented by Z∈C N×M In order to minimize the objective function, the complex matrix Z is factorized into two sub-matrices W∈C N×K and V∈C K×M , where K represents a constant. Therefore, the sub-matrices W and V can be calculated by equation (2): Where V H LV denotes the complex graph regularization in the real domain (i.e., merged images), λ is the regularization parameter, The balance between the accuracy of the adjustment factor and the sparsity of the matrix V. By decomposing the complex matrix Z, the W and V sub-matrices can be learned and obtained after training multiple image patches (or images).

[0056] Finally, the relationship between the sub-matrices W and V and the complex matrix Z can be obtained, and is expressed in equations (3) and (4): Z=WV (3), z=Wv (4).

[0057] Note that equation (3) can be transformed into equation (4) by substituting z into equation (1), where v represents a number of column vectors of the image. Accordingly, the number of column vectors v of each merged image can eventually be transformed into a complex matrix Z or a submatrix V (S105b). In some preferred embodiments, the number of column vectors v are feature vectors of each merged image, thereby allowing the merged image to be further analyzed through the feature vectors of these merged images. In summary, by training N image blocks and decomposing the complex matrix Z, a number of column vectors v (i.e., feature vectors) of the merged image can be obtained.

[0058] Once a number of column vectors v are obtained, the vector regularized CMF method is then used to proceed to the classification step (S105c). In step S105c, each merged image has its own number of column vectors v and known hormone receptor information (including the expression status of the hormone receptor). Therefore, by comparing the similarities between multiple number of column vectors v, the hormone receptor expression pattern of the merged image can be classified into a positive expression group or a negative expression group. Alternatively, the current diagnostic information corresponding to the source of the biopsy can be added to ensure re-verification. Accordingly, with the help of the known hormone receptor information and / or diagnostic information in step S101, the image recognition and differentiation of the positive or negative expression of the hormone receptor can be trained, thereby jointly establishing a model for determining the hormone receptor status mainly based on H&E stained biopsy. It is worth noting that the algorithm applicable to step S105c of the method disclosed in the present invention can be any classification algorithm known in the present invention. In a specific embodiment, step S105c is performed by executing the k-nearest neighbor (k-NN) algorithm.

[0059] The training algorithm applicable to the method (particularly steps S103 to S105) or system of the present disclosure may be a deep learning algorithm. Examples of the deep learning algorithm include, but are not limited to, convolutional neural networks (CNNs), long short term memory networks (LSTMs), recurrent neural networks (RNNs), generative adversarial networks (GANs), radial basis function networks (RBFNs), multilayer perceptrons (MtLPs), self-organizing maps (SOMs), deep belief networks (DBNs), restricted Boltzmann machines (RBMs) and autoencoders.

[0060] By executing the above steps S101 to S105 , the present disclosure establishes a well-trained model that can directly determine the status of hormone receptors from H&E staining of living tissue sections.

[0061] 2.2 System and method for identifying individual hormone receptor status

[0062] A second aspect of the present disclosure is a method and system for determining hormone receptor status based on H&E stained whole slide images of biopsy tissue sections collected from an individual. Figure 2 and Figure 3 .

[0063] Figure 2 The system 20 is shown, which includes an image capture unit 210, a server 220 and a processor 230, wherein the image capture unit 210 and the server 220 are respectively coupled to the processor 230. According to the present disclosure, the image capture unit 210 is configured to capture and collect one or more candidate H&E stained whole slide images from the individual's biopsy. In a specific embodiment, the image capture unit 210 is a microscope camera or a whole slide scanner. The server 220 is configured to store the model 2201 established by the above method 10 (i.e., steps S101 to S105). The processor 230 is configured to perform the image recognition of the method of the present invention for identifying the hormone receptor state.

[0064] In some embodiments, the server 220 and the processor 230 are configured as two separate devices; in other alternative embodiments, the two can also be configured in the same hardware. In some embodiments, the server 220 is in communication with the image acquisition unit 210 and the processor 230, and is configured to store one or more candidate H&E stained whole slide images transmitted from the image acquisition unit 210 and to be analyzed by the processor 230. The processor 230 is programmed with instructions to perform a method for determining the hormone receptor status of the candidate H&E stained whole slide images by means of the model 2201 established in the server 220.

[0065] According to some embodiments of the present disclosure, the image capture unit 210, the server 220, and the processor 230 are communicatively connected to each other. Various technologies can be used to implement the communication between the image capture unit 210, the server 220, and the processor 230. For example, the server 220 of the present invention can be a cloud server that communicates with the image capture unit 210 and the processor 230 through a network (such as a local area network (LAN), a wide area network (WAN), a network, or a wireless network).

[0066] refer to Figure 3 , which depicts a flow chart of a method 30 executed on a processor 230, the method being operable to determine the hormone receptor status of a candidate H&E stained whole slide image of a biopsy collected from an individual suffering from or suspected of suffering from breast cancer. The method 30 comprises the following steps (see Figure 3 Component symbols S301 to S303 are shown, S301: Divide the candidate H&E stained whole slide image into a plurality of image blocks; S302: selecting and merging the image blocks showing abnormal H&E staining by performing image block extraction to form an image to be tested; and S303: Using the model 2201 established by the method 10 of the present invention, and processing the image to be tested generated in step S302, the hormone receptor status is determined.

[0067] According to the present disclosure, hormone receptor status comprises the positive or negative expression of hormone receptor. In some alternative embodiments, hormone receptor status also comprises the expression intensity of hormone receptor. Hormone receptors suitable for the disclosed method are selected from the group consisting of estrogen receptor (ER), progesterone receptor (PR) and / or a combination thereof.

[0068] Once the candidate H&E stained whole slide image is obtained, the processor 230 performs block capture, so that the candidate H&E stained whole slide image is divided into a plurality of image blocks, and then the image blocks showing abnormal H&E staining are selected and merged into a test image (steps S301-302). Similar to steps S102 and S103 of the aforementioned method 10, the strategies applied in steps S301 and S302 can also be implemented by algorithms known in the art and default pathological criteria, preferably deep learning algorithms, including but not limited to: convolutional neural networks (CNNs), long short-term memory networks (LSTMs), recursive neural networks (RNNs), generative adversarial networks (GANs), radial basis function networks (RBFNs), multi-layer perceptrons (MLPs), self-organizing maps (SOMs), deep belief networks (DBNs), restricted Boltzmann machines (RBMs) and autoencoders. In order to save space, this section will not repeat the description of steps S301 and S302.

[0069] Then proceeding to step S303, the image to be tested is processed and compared with the reference information stored in the model 2201, so as to measure the hormone receptor state of the image to be tested. According to one embodiment of the present disclosure, the image to be tested can be processed by implementing a vector regularized complex matrix decomposition method executed by the processor 230. Figure 3 As shown, the vector regularized complex matrix decomposition method specifically includes the following steps: (S303a) obtaining a complex matrix from the complex numbers of the image to be tested; (S303b) converting the complex matrix into a number of column vectors of the image to be tested; and (S303c) based on the absolute distance between the number of column vectors of the image to be tested obtained in step S303b and the number of column vectors of the merged image stored in the server 220 model 2201, distinguishing whether the hormone receptor on the image to be tested is positive or negative.

[0070] After obtaining a number of column vectors of the image to be tested, proceed to the classification step (S303c). Except for step S303c, the strategies used in steps S303a and S303b are similar to the strategies described in steps S105a and S105b of method 10, and the purpose is also to obtain the feature vectors (i.e., a number of column vectors) of the real image (i.e., the image to be tested) for image recognition, thereby providing correct recognition results. Converting the real data (data) into complex numbers has been described in detail in steps S105a to S105b, and will not be repeated in this paragraph for the sake of brevity.

[0071] The main difference between step S105c and step S303c lies in the difference in the classification strategy applied to the image to be tested. Unlike step S105, in step S303c, several column vectors of the image to be tested are used to compare with several column vectors of the merged image in model 2201, specifically by calculating the absolute distance between the two. Generally speaking, the closer the distance between several column vectors of two images, the more similar the two images are considered. Preferably, the calculation can be performed by executing the k-nearest neighbor (k-NN) algorithm. If several column vectors of the image to be tested are closer to several column vectors of the merged image that expresses positive than several column vectors of the merged image that expresses negative, then several column vectors of the image to be tested will be determined as positive expression vectors. On the contrary, if several column vectors of the image to be tested are closer to the merged image of negative expression, they will be determined as negative expression vectors. During specific execution, several column vectors of the image to be tested will be compared with several column vectors of all merged images in model 2201. Each comparison will generate an identification result, and when a test image completes all the comparisons, multiple identification results will be generated. If the number of column vectors corresponding to the positive expression of hormone receptors is greater than the number of column vectors corresponding to the negative expression, the test image is determined to have positive hormone receptor expression. On the contrary, if the number of column vectors (of the test image) corresponding to the negative expression is greater, the test image is determined to have negative hormone receptor expression. In this way, step S303c can accurately determine whether the hormone receptor expression of the test image from the individual is positive or negative.

[0072] In a preferred embodiment, the vector regularized complex matrix decomposition method used to process the image to be tested may also include a step of measuring the intensity of expression (step S303d). Since the comparison of step S303c has given the measured numbers of multiple positive and negative hormone receptor expressions respectively, the intensity of expression can be expressed by further calculating the ratio of these numbers, thereby expressing the proportion of positive expression or negative expression. For example, if there are a total of 15 merged images as reference images in the model, the comparison of these merged images with the image to be tested one by one will obtain 15 results, of which 10 are positive expressions and 5 are negative expressions. Accordingly, the hormone receptor state of the image to be tested is determined to be positive expression, and its expression intensity is represented by 10 / 15 (10 times are determined to be positive in 15 determinations). By this calculation, step S303d can further determine the hormone receptor expression intensity of the image to be tested from the individual.

[0073] 2.3 Methods for detecting and treating cancer

[0074] The purpose of the present disclosure is also to provide diagnosis and treatment for individuals suffering from or suspected of having breast cancer. In this regard, the aforementioned methods, models and systems can all be used to assist clinical personnel in accurately determining the state of hormone receptors. Therefore, the present disclosure includes another aspect of a method for determining and treating individual breast cancer.

[0075] According to some embodiments of the present disclosure, the method comprises: (a) H&E stained whole slide images obtained from biopsy tissue sections of the aforementioned individuals; (b) determining the hormone receptor status of the individual by using the aforementioned methods and systems; and (c) administering an anti-cancer treatment to the individual based on the hormone receptor status of step (b).

[0076] The disclosed method begins with obtaining H&E stained whole slide images from a biopsy of an individual, the individual being a mammal, for example, a human, mouse, rat, hamster, guinea pig, rabbit, dog, cat, cow, goat, sheep, monkey or horse. Preferably, the individual is a human. Suitable tools and / or processes may be implemented to obtain the biopsy and its whole slide image. In a specific embodiment, the biopsy is a breast biopsy stained with hematoxylin-eosin, and its whole slide image is captured and collected with the aid of an image capture device, such as the image capture unit 210 (e.g., a microscope camera or a whole slide scanner) of the disclosed system 20.

[0077] Next, the hormone receptor status of the individual is determined by the aforementioned method 30. According to the present disclosure, the hormone receptor status includes the positive or negative expression of hormone receptors selected from the group consisting of estrogen receptors (ER), progesterone receptors (PR) and / or combinations thereof, and the expression intensity thereof.

[0078] Once determined and confirmed (optionally), the hormone receptor status of an individual can be used as an indicator to determine whether an anti-cancer treatment should be administered to the individual. In some embodiments, when the whole slide image is determined to be positive for ER or PR expression, the individual is likely or at risk of developing ER / PR-positive breast cancer, and thus, the individual is administered an anti-cancer treatment that can prevent or ameliorate symptoms associated with ER / PR-positive breast cancer. In other embodiments, when the whole slide image is determined to be negative for ER or PR expression, the individual is likely or at risk of developing ER / PR-negative breast cancer, and thus, the individual is administered an anti-cancer treatment that can prevent or ameliorate symptoms associated with ER / PR-negative breast cancer.

[0079] Examples of anticancer treatments suitable for the methods of the present disclosure (i.e., for administration to individuals exhibiting positive or negative hormone receptor status) include, but are not limited to, surgery, radiofrequency ablation, systemic chemotherapy, transarterial chemoembolization (TACE), immunotherapy, targeted drug therapy, hormone therapy, and combinations thereof. Any clinical practitioner can select a treatment suitable for the methods of the present disclosure based on the specific condition being treated, the severity of the condition, the physical condition of the individual patient (including age, physical condition, body shape, sex and weight, duration of treatment, type of concurrently treated disease (if any)), the specific route of administration, and other similar factors, in conjunction with the general knowledge and expertise of health practitioners.

[0080] With the aid of the aforementioned technical features, the disclosed method does not require immunohistochemistry (IHC) staining, and can provide accurate measurement and identification of hormone receptor status mainly based on H&E stained whole slide images, thereby improving the accuracy and efficiency of breast cancer diagnosis, thereby allowing confirmed patients to receive appropriate treatment.

[0081] Example

[0082] Materials and Methods

[0083] Data Collection

[0084] A total of 166 H&E-stained whole slide images of estrogen receptor (ER) and 163 H&E-stained whole slide images of progesterone receptor (PR) were obtained from the Department of Breast Surgery of Mackay Memorial Hospital to construct the image recognition and verification model.

[0085] Image processing and tile extraction

[0086] Each whole slide image obtained from the database was rectified to a normalized pixel size of 8 times magnification and then divided into 256×256 image patches for further use in the deep learning procedure using a CNN model.

[0087] Vector regularized complex matrix factorization (CMF) for pattern recognition

[0088] The vector regularized CMF method of the present invention aims to directly transform a constrained optimization problem into an unconstrained optimization problem. Based on the principle of Euler's formula, the vector value of pixel intensity can be normalized and converted to a unit sphere by mapping f from N-dimensional real space to N-dimensional complex space, using the equation: where xt represents an N-dimensional vector containing an expressing image X in lexicographic ordering t , x t (c)∈[0,1] and α∈R + .

[0089] Next, given a matrix Z∈C N×M , and find two matrices W∈C N×K and V∈C K×M To minimize the objective function where λ is the regularization parameter, And the balance between the accuracy of the λ adjustment factor and the sparsity of the matrix V. Notice,

[0090] Example 1: Constructing the image recognition model disclosed herein

[0091] The purpose of this experiment is to provide a trained machine learning model for whole slide image recognition. To this end, two models were established according to the process described in the "Materials and Methods" section to identify estrogen receptors (ER) and progesterone receptors (PR). Specifically, for estrogen receptor (ER) identification (Model I), a total of 133 whole slide images were used, including 107 whole slide images with positive ER expression and 26 whole slide images with negative ER expression; and for progesterone receptor (PR) identification (Model II), a total of 130 whole slide images were used, including 91 positive PR expressions and 39 negative PR expressions.

[0092] Example 2: Verifying the image recognition model disclosed herein

[0093] Next, the image recognition efficiency of the training model and the method for determining hormone receptor status of Example 1 was evaluated. To this end, the vector regularized CMF method of the present disclosure was used to process 33 candidate whole slide images containing ER and PR expressions and input them into the model of the present disclosure (i.e., the aforementioned Model I and Model II). In equations (1) and (2), the values ​​of the two parameters α and λ were adjusted within the interval [0, 2) and 0.01, respectively.

[0094] The results showed that the recognition rates of ER recognition and PR recognition by the model of the present invention were as high as 86% and 81% respectively.

[0095] By using the disclosed method and system, pathological biopsy tissue sections obtained from patients can be automatically identified and interpreted without the need for additional IHC examination, thereby improving the efficiency and accuracy of breast cancer diagnosis.

[0096] It should be understood that the foregoing description of the embodiments is given only by way of example, and various modifications may be made by those skilled in the art. The above description, examples and experimental results provide a complete description of the structure and use of exemplary embodiments of the present invention. Although various specific embodiments of the present invention are disclosed in the above embodiments, they are not intended to limit the present invention. Those skilled in the art of the present invention may make various changes and modifications to the present invention without departing from the principles and spirit of the present invention. Therefore, the scope of protection of the present invention shall be subject to the definition of the claims. Description of Reference Numerals

[0097] 10,30 Method 20 Systems 210 Image Capture Unit 220 Server 2201 Model 230 processors Steps S101-S105, S105a-S105c, S301-S303, S303a-S303d

Claims

1. A method for establishing a model for determining hormone receptor status of an individual via whole slide imaging of hematoxylin-eosin (H&E) stained tissue sections of the individual, comprising: (a) obtaining a plurality of H&E-stained whole slide images of the biopsy tissue section, wherein each of the H&E-stained whole slide images contains hormone receptor information; (b) dividing each H&E stained whole slide image in step (a) into a plurality of image blocks; (c) distinguishing between normal and abnormal H&E staining in each of the image blocks of step (b) by performing tiles extraction; (d) selecting and merging the image blocks showing abnormal H&E staining after being differentiated in step (c) to generate a merged image of each H&E-stained whole slide image; and (e) using the hormone receptor information of step (a) to train the plurality of merged images respectively produced from step (d), thereby establishing the model, The hormone receptor information of step (a) comprises a positive expression or a negative expression of a hormone receptor, wherein the hormone receptor is selected from the group consisting of estrogen receptor (ER), progesterone receptor (PR) and / or a combination thereof.

2. The method of claim 1, wherein in step (e), the plurality of merged images are trained by executing a vector regularized complex matrix decomposition method, the vector regularized complex matrix decomposition method comprising: (e-1) obtaining a complex matrix from the complex values ​​of each of the merged images; (e-2) converting the complex matrix into a plurality of column vectors corresponding to each of the merged images; and (e-3) Based on the similarity between the column vectors obtained in step (e-2), determine whether each of the merged images is the positive expression or the negative expression of the hormone receptor.

3. The method of claim 2, wherein step (e-3) is performed by executing a k-nearest neighbors algorithm.

4. The method of claim 1, wherein steps (c), (d) and (e) are performed by executing a deep learning algorithm.

5. The method of claim 1, wherein the individual has or is suspected of having breast cancer.

6. A method for determining hormone receptor status based on H&E stained whole slide images of individual biopsy tissue sections, comprising: (a) Divide the H&E stained whole slide image into several image blocks; (b) selecting and merging the image blocks showing abnormal H&E staining by performing block extraction to form a test image; and (c) determining the hormone receptor status by processing the image to be tested within the model established by the method of claim 1, The hormone receptor status comprises a positive expression or a negative expression of a hormone receptor, wherein the hormone receptor is selected from the group consisting of estrogen receptor (ER), progesterone receptor (PR) and / or a combination thereof.

7. The method of claim 6, wherein in step (c), the image to be tested is processed by performing a vector regularization complex matrix decomposition method, comprising: (c-1) obtaining a complex matrix from the complex values ​​of the image to be measured; (c-2) converting the complex matrix into a number of column vectors corresponding to the image to be measured; and (c-3) Based on the absolute distance between the several column vectors of the image to be tested obtained in step (c-2) and the merged images in the model established by the method as described in claim 1, the image to be tested is distinguished as the positive expression or negative expression of the hormone receptor.

8. The method of claim 7, wherein step (c-3) is performed by executing a k-nearest neighbor (k-NN) algorithm.

9. The method of claim 8, wherein the hormone receptor status further comprises the expression intensity of the hormone receptor.

10. The method as claimed in claim 9, wherein the vector regularized complex matrix decomposition method further comprises, (c-4) determining the expression intensity of the hormone receptor in the image to be tested based on the ratio between the number of column vectors of the merged images that are positive in the model established according to the method as claimed in claim 1 and the number of column vectors of the merged images that are negative in the model established according to the method as claimed in claim 1.

11. The method of claim 6, wherein steps (b) and (c) are performed by executing a deep learning algorithm.

12. The method of claim 6, wherein the individual has or is suspected of having breast cancer.

13. A system for identifying the hormone receptor status of an individual, comprising: an image acquisition unit configured to collect one or more candidate H&E-stained whole slide images of a biopsy tissue section from the individual; A server configured to store the model established by the method of claim 1 and configured to receive the one or more candidate H&E-stained whole slide images transmitted from the image acquisition unit; and A processor programmed with instructions to execute a method for determining the hormone receptor status of the one or more candidate H&E stained whole slide images transmitted from the server, wherein the method comprises, (a) dividing each candidate H&E stained whole slide image into a number of image blocks; (b) selecting and merging the image blocks showing abnormal H&E staining by performing block extraction to form a test image; and (c) determining the hormone receptor state by means of the model stored in the server and by means of processing the image to be tested generated in step (b), wherein the hormone receptor state comprises a positive expression or a negative expression of a hormone receptor, wherein the hormone receptor is selected from the group consisting of estrogen receptor The invention relates to a group consisting of ER, progesterone receptor (PR) and / or a combination thereof.

14. The system of claim 13, wherein in step (c), the image to be tested is processed by executing a vector regularization complex matrix decomposition method, the method comprising: (c-1) obtaining a complex matrix from the complex values ​​of the image to be measured; (c-2) converting the complex matrix into a number of column vectors corresponding to the image to be measured; and (c-3) Based on a number of column vectors of the merged image stored in the model on the server and step (c-2) or the absolute distance between the number of column vectors of the image to be tested, the image to be tested is distinguished as the positive expression or the negative expression of the hormone receptor.

15. The system of claim 14, wherein step (c-3) is performed by executing a k-nearest neighbor (k-NN) algorithm.

16. The system of claim 14, wherein the hormone receptor status further comprises the expression intensity of the hormone receptor.

17. A system as described in claim 16, wherein the vector regularized complex matrix decomposition method further includes, (c-4) determining the expression intensity of the hormone receptor in the image to be tested based on the ratio between the number of column vectors corresponding to the positive expression merged images in the model stored in the server and the number of column vectors corresponding to the negative expression merged images in the model stored in the server.

18. The system of claim 13, wherein steps (b) and (c) are performed by executing a deep learning algorithm.

Images