Image processing device, operating method thereof, and program

Abstract

To provide a method of analyzing the state of unstained tissues at cell level.SOLUTION: A cancer determination device is provided, comprising an irradiation unit for irradiating a tissue with excitation light, a third harmonic image acquisition unit for acquiring a third harmonic image of the tissue based on light generated by third harmonic generation by an interaction between the tissue and the excitation light, and a cancer tissue determination unit for determining the possibility that the tissue is cancerous on the basis of the state of cell nuclei in the third harmonic image.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to a cancer diagnosis device. This application claims priority based on Japanese Patent Application Nos. 2019-192469 and 2019-192470, filed on October 23, 2019, the contents of which are incorporated herein by reference. [Background technology]

[0002] The morphology of cell nuclei carries important information in pathological diagnosis, and staining of cell nuclei is an essential procedure for pathological tissue specimens. Meanwhile, a technique for identifying cancerous tissue by analyzing images of biological tissue is known. When identifying cancerous tissue by image analysis, images of biological tissue that have been stained are used.

[0003] For example, a cancer screening device is known that captures an image of a group of biological cells coated with a dye that selectively stains cancer-related gene products in the biological cells in a chromatic color, and includes a judgment unit that judges the level of malignancy of the cancerous transformation of the group of biological cells based on the staining state of the group of biological cells in the image obtained (Patent Document 1). Previously, it was difficult to analyze the state of unstained tissue at the cellular level. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2017 / 200066 Summary of the Invention

[0005] One aspect of the present invention is a cancer diagnosis device comprising an irradiation unit that irradiates tissue with excitation light, a third harmonic image acquisition unit that acquires a third harmonic image of the tissue based on light generated by third harmonic generation caused by the interaction between the tissue and the excitation light, and a cancer tissue diagnosis unit that determines the likelihood that the tissue is cancerous tissue based on the state of cell nuclei in the third harmonic image. [Brief explanation of the drawings]

[0006] [Figure 1] 1 is a diagram showing an example of a uterine cancer determination system according to a first embodiment. [Figure 2] FIG. 2 is a diagram showing an example of a captured image of mouse skin tissue for explaining visualization of cell nuclei using third harmonic generation according to the first embodiment. [Figure 3] FIG. 2 is a diagram showing an example of a captured image of human uterine tissue according to the first embodiment. [Figure 4] 1 is a diagram illustrating an example of the configuration of a uterine cancer determination device according to a first embodiment. [Figure 5] FIG. 2 is a diagram showing an example of a nuclear region image according to the first embodiment. [Figure 6] FIG. 3 is a diagram illustrating an example of feature amounts according to the first embodiment. [Figure 7] FIG. 3 is a diagram showing an example of uterine cancer determination processing according to the first embodiment. [Figure 8] FIG. 4 is a diagram showing an example of a nucleus region image generation process according to the first embodiment. [Figure 9] FIG. 2 is a diagram showing an example of a Z-stack nuclear region image according to the first embodiment. [Figure 10] FIG. 4 is a diagram illustrating an example of a learning process according to the first embodiment. [Figure 11] FIG. 2 is a diagram showing an example of measurement of a cell nucleus according to the first embodiment. [Figure 12] FIG. 4 is a diagram illustrating an example of a determination process according to the first embodiment. [Figure 13] FIG. 10 is a diagram showing an example of a determination result according to the first embodiment. [Figure 14] FIG. 10 is a diagram showing an example of a uterine cancer determination device according to a second embodiment. [Figure 15] FIG. 10 is a diagram showing an example of deep learning used by a nucleus region image generating unit according to the second embodiment. [Figure 16] FIG. 10 is a diagram illustrating an example of core region learning processing according to the second embodiment. [Figure 17]FIG. 10 is a diagram illustrating an example of a core region determination process according to the second embodiment. [Figure 18] 10A and 10B are diagrams showing examples of a nucleus region image and a nucleus region annotation image according to the second embodiment. [Figure 19] FIG. 10 is a diagram showing an example of a result of a core region determination process according to the second embodiment. [Figure 20] FIG. 11 is a diagram showing an example of an F value for the entire Z-stack nuclear region image according to the second embodiment. [Figure 21] FIG. 10 is a diagram showing an example of a determination result according to the second embodiment. [Figure 22] FIG. 11 is a diagram showing an example of a second harmonic image according to the third embodiment. [Figure 23A] 10A and 10B are diagrams showing examples of cross sections of uterine tissues for each stage of tumor progression according to the third embodiment. [Figure 23B] 10A and 10B are diagrams showing examples of cross sections of uterine tissues for each stage of tumor progression according to the third embodiment. [Figure 24] FIG. 10 is a diagram showing an example of a uterine cancer determination device according to a third embodiment. [Figure 25] FIG. 11 is a diagram showing an example of uterine cancer determination processing according to the third embodiment. [Figure 26] FIG. 11 is a diagram showing an example of a pre-cancer / invasion cancer determination process according to the third embodiment. [Figure 27] FIG. 10 is a diagram showing an example of a fiber-like structure image according to the third embodiment. [Figure 28] FIG. 11 is a diagram illustrating an example of a detection pixel ratio according to the third embodiment. [Figure 29] FIG. 11 is a diagram showing an example of an average value of the detection pixel ratio for the shallow part of the uterine tissue according to the third embodiment. [Figure 30] FIG. 11 is a diagram showing an example of an ROC curve showing the determination rate of the cancer progression determination unit according to the third embodiment. [Figure 31] FIG. 11 is a diagram showing an example of the relationship between the determination threshold and the determination rate of the cancer progression determination unit according to the third embodiment. [Figure 32] FIG. 10 is a diagram showing an example of a system for determining the progression of uterine cancer according to a fourth embodiment. [Figure 33] FIG. 13 is a diagram showing an example of a second harmonic generation image according to the fourth embodiment. [Figure 34A] FIG. 13 is a diagram showing an example of cross sections of uterine tissues for each stage of tumor progression according to the fourth embodiment. [Figure 34B] FIG. 13 is a diagram showing an example of cross sections of uterine tissues for each stage of tumor progression according to the fourth embodiment. [Figure 35] FIG. 10 is a diagram showing an example of a cancer progression assessment device according to a fourth embodiment. [Figure 36] FIG. 13 is a diagram showing an example of cancer progression determination processing according to the fourth embodiment. [Figure 37] FIG. 10 is a diagram showing an example of a fiber-like structure image according to the fourth embodiment. [Figure 38] FIG. 13 is a diagram illustrating an example of a detection pixel ratio according to the fourth embodiment. [Figure 39] FIG. 13 is a diagram showing an example of an average value of the detection pixel ratio for the shallow part of the uterine tissue according to the fourth embodiment. [Figure 40] FIG. 13 is a diagram showing an example of an ROC curve showing the determination rate of the cancer progression determination unit according to the fourth embodiment. [Figure 41] FIG. 13 is a diagram showing an example of the relationship between the determination threshold and the determination rate of the cancer progression determination unit according to the fourth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0007] (First embodiment) The first embodiment will be described in detail below with reference to the drawings. FIG. 1 is a diagram showing an example of a uterine cancer determination system ST according to this embodiment. The uterine cancer determination system ST includes a uterine cancer determination device 1, a multiphoton microscope 2, and a display device 3. In the uterine cancer determination system ST, the uterine cancer determination device 1 analyzes a Z-stack captured image ZP1 of a subject's uterine tissue captured by the multiphoton microscope 2, and determines the possibility that the uterine tissue captured in the Z-stack captured image ZP1 is cancerous tissue. The uterine cancer determination device 1 displays the determination result on the display device 3.

[0008] In this embodiment, the uterine tissue is, for example, tissue of the cervix. Note that the uterine tissue may also include the uterine body. Furthermore, although the uterine tissue is described as an example in this embodiment, the tissue is not limited to this.

[0009] The Z-stack captured image ZP1 is a Z-stack image. A Z-stack image is a set of multiple images captured at different distances from the uterine tissue in the Z-axis direction. The Z-stack captured image ZP1 is made up of multiple images of the uterine tissue at each Z coordinate. Here, capturing an image of the uterine tissue at each Z coordinate means capturing an image of the uterine tissue at various distances between the objective lens and the uterine tissue. The Z-stack captured image ZP1 is a multiple image captured by the multiphoton microscope 2 at various distances between the uterine tissue and the lens.

[0010] In this embodiment, the Z axis is selected to point in the direction from the epithelial tissue of the uterine tissue toward the interior. In other words, the closer the imaging plane is to the uterine epithelial tissue, the smaller the Z coordinate value, and the deeper the imaging plane is into the uterine tissue, the larger the Z coordinate value. The origin of the Z axis is selected at a shallow position close to the uterine epithelial tissue. In this way, the Z-stack captured image ZP1 is a plurality of cross-sectional images perpendicular to the depth direction of the subject's uterine epithelial tissue, and is also simply referred to as a cross-sectional image. Note that the depth direction is the direction from the superficial layer to the basal layer, and the plurality of cross-sectional images do not need to be perfectly perpendicular to the depth direction of the subject's uterine epithelial tissue, and may be tilted by approximately ±5 degrees.

[0011] The multiphoton microscope 2 observes and images the subject's uterine tissue in an unstained state. The multiphoton microscope 2 images the uterine tissue using nonlinear optical phenomena. The nonlinear optical phenomena used by the multiphoton microscope 2 for imaging include second harmonic generation (SHG) and third harmonic generation (THG).

[0012] SHG is a phenomenon in which light with twice the frequency of the excitation light is generated by interaction with nonlinear optical crystals such as collagen fibers. THG is a phenomenon in which light with a frequency three times higher than the excitation light is generated by interactions with interfaces and layered structures.

[0013] In this embodiment, the multiphoton microscope 2 uses THG to image uterine tissue and generate a third harmonic image. The third harmonic image is an image of a cross section of the uterine tissue generated based on light generated by third harmonic generation due to interaction between excitation light irradiated from the irradiation unit of the multiphoton microscope 2 and the uterine tissue. The captured image included in the Z-stack captured image ZP1 is a third harmonic image, which is an image of the uterine tissue captured using THG. Hereinafter, the third harmonic image will be referred to as a THG image.

[0014] Hereinafter, the Z-stack captured image ZP1 will be referred to as the Z-stack THG image ZT1. The Z-stack THG image ZT1 is a plurality of THG images of the uterine tissue captured using THG at different distances from the uterine tissue in the Z-axis direction. In other words, the third harmonic image of the subject's uterine tissue is a plurality of cross-sectional images of the subject's uterine epithelial tissue. Furthermore, each of the multiple images included in the Z-stack THG image ZT1 is referred to as a THG image PTi (i=1, 2, . . . , N: N is the number of images included in the Z-stack THG image ZT1), etc.

[0015] The multiphoton microscope 2 uses THG to visualize cell nuclei in the subject's uterine tissue, which has been previously exposed to acetic acid. The ability to visualize cell nuclei using THG will now be described with reference to Figures 2 and 3. Figure 2 shows an example of a captured image of mouse skin tissue to illustrate the visualization of cell nuclei using THG.

[0016] THG image PA10, THG image PA11, THG image PA12, and THG image PA13 are images of mouse skin tissue captured using THG. Fluorescence image PA20, PA21, PA22, and PA23 are images of cell nuclei of mouse skin tissue stained with a fluorescent dye and captured using fluorescence. The fluorescent dye is Hoechst 33342. Composite image PA30, PA31, PA32, and PA33 are images composited of an image captured using THG and an image stained with Hoechst 33342 and captured using fluorescence.

[0017] The THG image PA10 and the fluorescence image PA20 are images of mouse skin tissue captured without the addition of fluorescent dye or acetic acid. The THG image PA10 and the fluorescence image PA20 capture a common portion of the skin tissue. The composite image PA30 is an image obtained by combining the THG image PA10 and the fluorescence image PA20.

[0018] The THG image PA11 and the fluorescent image PA21 are images of mouse skin tissue captured with a fluorescent dye added. The THG image PA11 and the fluorescent image PA21 capture a common portion of the skin tissue. The composite image PA31 is an image obtained by combining the THG image PA11 and the fluorescent image PA21.

[0019] The THG image PA12 and the fluorescence image PA22 are images of mouse skin tissue captured with a fluorescent dye and acetic acid added. The THG image PA12 and the fluorescence image PA22 capture a common portion of the skin tissue. The composite image PA32 is an image created by combining the THG image PA12 and the fluorescence image PA22.

[0020] The THG image PA13 and the fluorescence image PA23 are images of mouse skin tissue captured with acetic acid added. The THG image PA13 and the fluorescence image PA23 capture a common portion of the skin tissue. The composite image PA33 is an image obtained by combining the THG image PA13 and the fluorescence image PA23.

[0021] Fluorescence image PA22 shows the cell nucleus. Composite image PA32 shows that the image of the cell nucleus shown in THG image PA12 matches the image of the stained cell nucleus shown in fluorescence image PA22. The image of the cell nucleus captured using THG with the addition of a fluorescent dye and acetic acid shows the same cell nucleus as the image of the cell nucleus captured using fluorescence with the addition of acetic acid.

[0022] Next, comparing THG image PA13 with THG image PA12, it can be seen that the image of the cell nucleus is captured in THG image PA13 in the same way as THG image PA12. That is, it can be seen that the image captured using THG in the presence of acetic acid captures the cell nucleus in the same way as the image captured using THG in the presence of fluorescent dye and acetic acid.

[0023] Therefore, in THG image PA13, the cell nuclei are captured in the same manner as in fluorescence image PA22. In other words, in an image captured using THG in the presence of acetic acid, the cell nuclei are captured in the same manner as in an image captured using fluorescence in the presence of acetic acid. In this way, by using THG in the presence of acetic acid, images of cell nuclei can be captured without staining the cell nuclei.

[0024] Fig. 3 is a diagram showing an example of captured images of human uterine tissue according to the present embodiment. In the captured images shown in Fig. 3, cervical tissue from human uterine tissue is captured. THG image PB10 and THG image PB11 are images of normal human cervical tissue captured using THG. THG image PB20 and THG image PB21 are images of human cervical tissue, which is cancerous tissue, captured using THG. Stained image PB12 and stained image PB22 are images for comparison with the THG image. Stained image PB12 is an image of normal human cervical tissue stained with a dye. Stained image PB22 is an image of cancerous human cervical tissue stained with a dye. The dye is hematoxylin.

[0025] In THG image PB10 and THG image PB20, cervical tissue was imaged without adding acetic acid, while in THG image PB11 and THG image PB21, cervical tissue was imaged with adding acetic acid. Comparing THG image PB11 and THG image PB21 with THG image PB10 and THG image PB20, it can be seen that the cell nuclei are more clearly imaged in the THG image taken with acetic acid added than in the THG image taken without acetic acid added.

[0026] Comparing THG image PB11 and THG image PB21 with stained image PB12 and stained image PB22, it is clear that THG image PB11 and THG image PB21 are able to capture cell nuclei, just like stained image PB12 and stained image PB22.

[0027] In other words, in human cervical tissue, as in the mouse skin tissue shown in Figure 2, by using THG in the presence of acetic acid, it is possible to capture images of cell nuclei without staining them. Conventionally, in THG images taken by multiphoton microscopes, cell nuclei are rarely seen in tissues. In contrast, in the THG images taken by multiphoton microscope 2, cell nuclei of uterine tissue are seen.

[0028] (Configuration of the uterine cancer detection device) Next, the configuration of the uterine cancer determination device 1 will be described with reference to Fig. 4. Fig. 4 is a diagram showing an example of the configuration of the uterine cancer determination device 1 according to this embodiment. The uterine cancer determination device 1 is, for example, a computer.

[0029] The uterine cancer diagnosis device 1 includes a nuclear region image generation unit 10, a feature amount processing unit 11, a data division unit 12, a classification model generation unit 13, a cancer tissue determination unit 14, an output unit 15, and an operation input unit 16. The nuclear region image generation unit 10, the feature amount processing unit 11, the data division unit 12, the classification model generation unit 13, the cancer tissue determination unit 14, and the output unit 15 are each a module realized by a CPU (Central Processing Unit) reading a program from a ROM (Read Only Memory) and executing the processing.

[0030] The nuclear region image generating unit 10 generates a Z-stack nuclear region image ZN1 from the Z-stack THG image ZT1 captured by the multiphoton microscope 2. Here, the Z-stack nuclear region image ZN1 refers to a plurality of images in which regions showing cell nuclei of uterine tissue are determined for each of the THG images PTi (i=1, 2, . . ., N: N is the number of images included in the Z-stack THG image ZT1), which are a plurality of images included in the Z-stack THG image ZT1. Each of the multiple images included in the Z-stack nuclear region image ZN1 is called a nuclear region image PNi (i=1, 2, . . . , N: N is the number of images included in the Z-stack nuclear region image ZN1).

[0031] In the present embodiment, the nuclear region, which is a region indicating a cell nucleus, is determined, for example, by a user of the uterine cancer determination device 1. The user of the uterine cancer determination device 1 visually checks the Z-stack captured image ZP1 and determines the nuclear region based on knowledge and experience.

[0032] An example of the nuclear region image PNi will now be described with reference to Fig. 5. Fig. 5 is a diagram showing an example of the nuclear region image PNi according to this embodiment. In the nuclear region image PNn, a nuclear region is determined in the THG image PTi of normal uterine tissue, and the contour of the determined nuclear region is colored. In the nuclear region image PNc, a region showing a cell nucleus is determined in the THG image PTi of uterine tissue that is cancerous tissue, and the contour of the determined nuclear region is colored.

[0033] Returning to FIG. 4, the description of the configuration of the uterine cancer diagnosis device 1 will be continued. The nucleus region image generating unit 10 includes a THG image acquiring unit 100, a nucleus determination operation receiving unit 101, and an image processing unit . The THG image acquisition unit 100 acquires the Z-stack captured image ZP1 captured by the multiphoton microscope 2.

[0034] The nucleus determination operation receiving unit 101 receives a nucleus determination operation from a user of the uterine cancer determination device 1 via the operation input unit 16. Here, the nucleus determination operation is an operation for determining an area showing a cell nucleus of uterine tissue in each of the THG images PTi included in the Z-stack captured image ZP1. The image processing unit 102 generates a Z-stack nucleus region image ZN1 from the Z-stack captured image ZP1 based on the nucleus determination operation received by the nucleus determination operation receiving unit 101.

[0035] The feature processor 11 extracts feature values C1 from the Z-stack nucleus region image ZN1 and performs various processes on the extracted feature values C1. The feature values C1 are feature values that indicate the state of the cell nuclei and are extracted for each nucleus region image PNi. In this embodiment, the state of the cell nuclei is at least one selected from the group consisting of the area of the cell nucleus, the density of the cell nucleus, and the shape of the cell nucleus.

[0036] The feature amount processing unit 11 includes a nuclear region image acquisition unit 110 , a nuclear measurement unit 111 , a feature amount extraction unit 112 , a Z position correction unit 113 , and a feature amount scaling unit 114 .

[0037] The nuclear region image acquisition unit 110 acquires the Z-stack nuclear region image ZN1 supplied from the nuclear region image generation unit 10. The nucleus measurement unit 111 performs various measurements for each nucleus based on the image of the nucleus captured in the Z-stack nucleus region image ZN1, including, for example, area, circularity, and nearest neighbor distance.

[0038] The feature amount extraction unit 112 extracts a feature amount C1 for each nuclear region image PNi included in the Z-stack nuclear region image ZN1 based on the measurement results of the nuclear measurement unit 111.

[0039] The feature C1 will now be described in detail with reference to Fig. 6. Fig. 6 is a diagram showing an example of the feature C1 according to this embodiment. The feature C1 includes, for example, the area of the cell nucleus, the circularity of the cell nucleus, the nearest neighbor distance between the cell nuclei, and the number of cell nuclei. Of the feature C1, the area, circularity, and nearest neighbor distance are used by calculating the median and median absolute error.

[0040] Multiphoton imaging of normal cervical tissue shows distinct cell nuclei at different depths from the surface, which is consistent with pathological findings. Here, the depths from the surface are classified into upper, middle, and lower layers. Figure 6 shows the qualitative characteristics of the feature values for each depth from the surface, categorized as "large," "medium," and "small." In the uterine cancer determination device 1, a feature amount of a cell nucleus that is effective for determining the possibility that uterine tissue is cancerous tissue is selected in advance.

[0041] For example, the median area of cell nuclei in normal tissue is "small" in the upper layer, "medium" in the middle layer, and "medium" in the lower layer. In other words, the median area of normal tissue does not show "large" in any layer. On the other hand, the median area is "large" in the first cancer tissue, and since cancer tissue showing a "large" value has been confirmed, the "large" median area is at least a characteristic of the first cancer tissue, and is considered to be a characteristic that can distinguish cancer tissue from normal tissue. A "large" median area corresponds to the pathological feature of enlarged nuclei.

[0042] Similarly, a large median absolute error in the area of cell nuclei, a small median circularity of cell nuclei, a large median absolute error in the circularity of cell nuclei, a small median nearest-neighbor distance between cell nuclei, a large median absolute error in the nearest-neighbor distance between cell nuclei, and a large number of cell nuclei are each considered to be characteristics that can distinguish cancerous tissue from normal tissue.

[0043] A large median absolute error in the area of cell nuclei corresponds to the pathological feature of heterogeneity in nuclear size. A small median value of circularity of cell nuclei corresponds to the pathological feature of atypical cell nuclei. A large median absolute error in circularity of cell nuclei corresponds to the pathological feature of atypical cell nuclei. A small median value of nearest neighbor distance between cell nuclei corresponds to the pathological feature of high density of cell nuclei. A large median absolute error in nearest neighbor distance between cell nuclei corresponds to the pathological feature of heterogeneity in the distribution of cell nuclei in a certain area. A large number of cell nuclei corresponds to the pathological feature of high density of cell nuclei.

[0044] In summary, the cell nuclei of cancerous uterine tissue have the following characteristics (i) to (vi). (i) The average area of the cell nuclei is increased compared to normal uterine tissue; (ii) the variation in the area of the cell nuclei is increased compared to normal uterine tissue; (iii) the density of the cell nuclei is increased compared to normal uterine tissue; (iv) the variation in the density of the cell nuclei is increased compared to normal uterine tissue; (v) the irregularity in the shape of the cell nuclei is increased compared to normal uterine tissue; (vi) the variation in the shape of the cell nuclei is increased compared to normal uterine tissue.

[0045] Returning to FIG. 4, the description of the configuration of the uterine cancer diagnosis device 1 will be continued. The Z-position correction unit 113 performs correction in the Z-axis direction on the Z-stack nuclear region image ZN1. The feature amount scaling unit 114 performs scaling on the feature amount C1 extracted by the feature amount extraction unit 112 based on normalization and standardization.

[0046] The data dividing unit 12 divides the feature C1 extracted by the feature processing unit 11 into learning feature data CL1 used for learning and determination feature data CE1 used for determination. The classification model generation unit 13 generates the classification model M1 based on machine learning using the learning feature data CL1. Here, the machine learning used by the classification model generation unit 13 is, for example, a nonlinear support vector machine.

[0047] The cancer tissue determination unit 14 determines the possibility that the uterine tissue is cancer tissue based on the determination feature amount data CE1 and the classification model M1 generated by the classification model generation unit 13. Output unit 15 outputs determination result A of cancer tissue determination unit 14 to display device 3. Determination result A indicates whether the uterine tissue is cancer tissue or normal tissue. Determination result A according to the present embodiment is an example, and cancer tissue determination unit 14 may output a probability indicating whether the tissue is cancer tissue or normal tissue to display device 3, or may output this to display device 3 in parallel with learning feature data CL1 used for learning.

[0048] The operation input unit 16 accepts various operations from the user of the uterine cancer determination device 1. Examples of the operation input unit 16 include a touch panel, a mouse, and a keyboard. The display device 3 displays the determination result A obtained by the uterine cancer determination device 1. The display device 3 is, for example, a display.

[0049] (Processing of uterine cancer diagnosis system) Next, a uterine cancer detection process, which is a process of the uterine cancer detection system ST, will be described. Fig. 7 is a diagram showing an example of the uterine cancer detection process according to this embodiment.

[0050] Step S10: The multiphoton microscope 2 captures a Z-stack image ZP1 of the subject's uterine tissue. As described above, in this embodiment, the Z-stack image ZP1 is a Z-stack THG image ZT1. The multiphoton microscope 2 outputs the captured Z-stack image ZP1 to the uterine cancer assessment device 1.

[0051] Step S20: The uterine cancer determination device 1 performs a normal tissue / cancer tissue determination process, which is a process for determining whether the uterine tissue of the subject is normal tissue or cancer tissue, based on the Z-stack captured image ZP1 captured by the multiphoton microscope 2. Details of the normal tissue / cancer tissue determination process will be described later with reference to FIG.

[0052] Step S30: The output unit 15 of the uterine cancer determination device 1 performs processing based on the determination result A. If the uterine tissue of the subject is determined to be normal (step S30; YES), the output unit 15 outputs a result indicating that the uterine tissue of the subject is normal to the display device 3. Thereafter, the display device 3 performs the processing of step S40. On the other hand, if it is determined that the uterine tissue of the subject is not normal tissue, that is, is cancerous tissue (step S30; NO), the output unit 15 outputs a result indicating that the uterine tissue of the subject is cancerous tissue to the display device 3. Thereafter, the display device 3 executes the process of step S50.

[0053] Step S40: The display device 3 displays the result indicating that the uterine tissue of the subject is normal. Step S50: The display device 3 displays the result indicating that the uterine tissue of the subject is cancerous tissue. With this, the uterine cancer detection system ST ends the uterine cancer detection process.

[0054] (Processing of uterine cancer detection device 1) Next, we will explain the details of the normal tissue / cancer tissue determination process of the uterine cancer determination device 1. The normal tissue / cancer tissue determination process includes a nuclear region image generation process for generating a Z-stack nuclear region image ZN1, a learning process for generating a classification model M1, and a determination process for determining the possibility that the uterine tissue is cancer tissue based on the classification model M1.

[0055] FIG. 8 is a diagram showing an example of the nucleus region image generation process according to this embodiment. Step S110: The THG image acquiring unit 100 acquires a Z-stack captured image ZP1 captured by the multiphoton microscope 2. That is, the THG image acquiring unit 100 acquires a third harmonic image of the uterine tissue of the subject obtained by the multiphoton microscope 2. The THG image acquisition unit 100 supplies the acquired Z stack captured image ZP1 to the nucleus determination operation reception unit 101.

[0056] Here, the Z-stack nuclear region image ZN1 will be described in detail with reference to Fig. 9. Fig. 9 is a diagram showing an example of the Z-stack nuclear region image ZN1 according to this embodiment. In this embodiment, the Z-stack nuclear region image ZNZP1 is a plurality of images. The THG images PTi (i = 1, 2, ..., 317) included in the Z-stack captured image ZP1 are made up of 512 x 512 pixels.

[0057] The Z-stack image ZP1 consists of 157 THG images PTi of cancerous uterine tissue and 160 THG images PTi of normal uterine tissue. Nine specimens of uterine tissue were used to capture the THG images PTi of cancerous uterine tissue and the THG images PTi of normal uterine tissue.

[0058] As an example, the uterine cancer diagnosis device 1 uses an image of one specimen from among the THG images PTi in which uterine tissue, which is cancerous tissue, is imaged for the diagnosis process. The uterine cancer diagnosis device 1 uses images of 17 specimens, including the remaining THG images PTi in which uterine tissue, which is cancerous tissue, is imaged, which are not used for the diagnosis process, and the THG images PTi in which uterine tissue, which is normal tissue, is imaged, for the learning process.

[0059] The third harmonic images for training, which are the multiple THG images PTi used in the training process, are multiple cross-sectional images of uterine epithelial tissue.

[0060] Returning to FIG. 8, the description of the nuclear region image generation process will be continued. Step S120: The nucleus determination operation receiving unit 101 receives a nucleus determination operation from the user of the uterine cancer determination device 1 via the operation input unit 16. The nucleus determination operation receiving unit 101 supplies information indicating the received nucleus determination operation and the Z-stack captured image ZP1 to the image processing unit 102.

[0061] Here, the nucleus determination operation receiving unit 101 causes, for example, the Z-stack captured image ZP1 acquired by the THG image acquiring unit 100 to be displayed on the display device 3. A user of the uterine cancer determination device 1 determines a region in each THG image PTi that represents a cell nucleus of uterine tissue while checking the THG images PTi included in the Z-stack captured image ZP1 displayed on the display device 3. The user of the uterine cancer determination device 1 inputs an operation to determine the region as representing a cell nucleus via the operation input unit 16. As an example, the nucleus determination operation is performed by tracing the outline of the region determined to represent a cell nucleus with a mouse pointer via the operation input unit 16, which is a mouse, in the THG image PTi displayed on the display device 3, which is a display.

[0062] The nucleus determination operation may be performed by tracing the outline of the area determined to represent a cell nucleus using the operation input unit 16, which is a touch pen, on the THG image PTi displayed on the display device 3, which is a touch panel.

[0063] Step S130: The image processing unit 102 generates a Z-stack nucleus region image ZN1 from the Z-stack captured images ZP1 based on the nucleus determination operation received by the nucleus determination operation receiving unit 101. Here, the image processing unit 102 generates the Z-stack nucleus region image ZN1 by coloring the outline of the region indicating the cell nuclei of the uterine tissue in each of the THG images PTi based on the nucleus determination operation. Step S140: The image processing unit 102 outputs the generated Z-stack nuclear region image ZN1 to the feature amount processing unit 11. With this, the uterine cancer determination device 1 ends the nuclear region image generation process.

[0064] Next, the learning process of the uterine cancer determination device 1 will be described with reference to Fig. 10. Fig. 10 is a diagram showing an example of the learning process according to this embodiment.

[0065] Step S210: The nuclear region image acquisition unit 110 acquires the Z-stack nuclear region image ZN1 supplied from the nuclear region image generation unit 10. Here, the Z-stack nuclear region image ZN1 supplied from the nuclear region image generation unit 10 has been generated in the nuclear region image generation process described above, and the nuclear region image PNi in which uterine tissue, which is cancerous tissue, is imaged and the nuclear region image PNi in which uterine tissue, which is normal tissue, are identified in advance. The nuclear region image acquisition unit 110 supplies the acquired Z-stack nuclear region image ZN1 to the nuclear measurement unit 111.

[0066] Step S220: The nucleus measurement unit 111 measures the area, circularity, and nearest neighbor distance for each cell nucleus based on the image of the cell nucleus captured in the Z-stack nucleus region image ZN1 acquired by the nucleus region image acquisition unit 110. The nucleus measurement unit 111 supplies the measurement results to the feature extraction unit 112.

[0067] Here, the measurement of the cell nucleus performed by the nucleus measurement unit 111 will be described with reference to Fig. 11. Fig. 11 is a diagram showing an example of the measurement of the cell nucleus according to this embodiment. The nucleus measurement unit 111 measures the number of pixels in the cell nucleus region x the area per pixel. Here, the area per pixel depends on the magnification of the multiphoton microscope 2, and is, for example, 1 μm 2 .

[0068] The nucleus measurement unit 111 measures the circularity as 4π × (area) / (square of perimeter). The circularity can take a value between 0 and 1. The closer the outline of the cell nucleus is to a circle, the closer the circularity value is to 1. The nucleus measurement unit 111 associates labels with cell nuclei, and calculates the Euclidean distance from the center of gravity of the label to be measured to the center of gravity of another label that is closest to this center of gravity, as the nearest neighbor distance.

[0069] Returning to FIG. 10, the explanation of the learning process will be continued. Step S230: The feature amount extraction unit 112 extracts a feature amount C1 for each nuclear region image PNi included in the Z-stack nuclear region image ZN1 based on the measurement result by the nuclear measurement unit 111.

[0070] Step S240: The feature extraction unit 112 determines whether the number of cell nuclei for each nucleus region image PNi is greater than 0. As described above, the number of cell nuclei is included in the feature C1, and the feature extraction unit 112 makes the determination based on the feature C1.

[0071] An example of a case where the number of cell nuclei captured in the nuclear region image PNi is 0 is when the epithelium of the uterine tissue is tilted relative to the direction perpendicular to the Z-axis direction, resulting in imaging of the area near the epithelium without the uterine tissue being included in the imaging plane.

[0072] If the feature extraction unit 112 determines that the number of cell nuclei for each nucleus region image PNi is greater than 0 (step S240; YES), the feature processing unit 11 executes the process of step S250. On the other hand, if the feature extraction unit 112 determines that the number of cell nuclei for each nucleus region image PNi is 0 (step S240; NO), the feature processing unit 11 executes the process of step S2130.

[0073] Step S250: The Z-position correcting unit 113 corrects the Z-stack nucleus region image ZN1 in the Z-axis direction. Here, the Z-position correcting unit 113 corrects the Z-coordinate value so that, for example, the Z-coordinate of the imaging plane corresponding to the nucleus region image PNi with the smallest Z-coordinate among the nucleus region images PNi determined by the feature extracting unit 112 to have more than one cell nucleus becomes the origin of the Z-axis. It is preferable that the Z position correction unit 113 performs correction in the Z axis direction even when the number of nuclear region images included in the Z stack nuclear region image ZN1 is one.

[0074] Step S260: The feature amount scaling unit 114 calculates the average value and standard deviation for each type of feature amount C1 extracted by the feature amount extraction unit 112 for the nucleus region image PNi included in the Z-stack nucleus region image ZN1.

[0075] Step S270: The feature scaling unit 114 scales the feature C1 extracted by the feature extraction unit 112 using the average value and standard deviation calculated for each type. Here, the feature scaling unit 114 scales the feature C1 based on normalization or standardization, for example. For example, the feature scaling unit 114 standardizes the range of values that different types of feature included in the feature C1 can take by normalization. Alternatively, the feature scaling unit 114 standardizes the average value of different types of feature included in the feature C1 to 0 and the variance to 1.

[0076] Step S280: The data dividing unit 12 divides the feature C1 scaled by the feature scaling unit 114 into training feature data CL1 to be used for training and determination feature data CE1 to be used for determination. The data dividing unit 12 sets, among the feature C1, a feature extracted from an image of a certain specimen in the THG image PTi in which uterine tissue, which is cancerous tissue, is captured as determination feature data CE1. The data dividing unit 12 sets the remaining feature C1 as training feature data CL1.

[0077] As described above, in the Z-stack nuclear region images ZN1 acquired by the nuclear region image acquisition unit 110 in step S210, the nuclear region images PNi in which uterine tissue, which is cancerous tissue, is imaged and the nuclear region images PNi in which uterine tissue, which is normal tissue, are identified in advance. The data division unit 12 associates, for each data item of the learning feature data CL1, a label that distinguishes between the feature extracted from the nuclear region image PNi in which uterine tissue, which is cancerous tissue, is imaged and the feature extracted from the nuclear region image PNi in which uterine tissue, which is normal tissue, is imaged.

[0078] Step S290: The data dividing unit 12 outputs the learning feature data CL1 to the classification model generating unit 13. The data dividing unit 12 outputs the determination feature data CE1 to the cancer tissue determination unit . Step S2100: The classification model generation unit 13 acquires the learning feature data CL1 output from the data division unit 12.

[0079] Step S2110: The classification model generation unit 13 generates the classification model M1 based on machine learning using the acquired learning feature data CL1. Here, the classification model generation unit 13 generates the classification model M1 based on a nonlinear support vector machine, as an example. Note that the classification model generation unit 13 may generate the classification model M1 based on machine learning other than a nonlinear support vector machine.

[0080] As described above, the training feature data CL1 is extracted from the Z-stack nuclear region image ZN1 generated from the Z-stack captured image ZP1 consisting of the THG image PTi of cancerous uterine tissue and the THG image PTi of normal uterine tissue. Therefore, the classification model M1 is a classification model trained using training third harmonic images of normal uterine tissue obtained by a multiphoton microscope and / or training third harmonic images of uterine cancer tissue obtained by a multiphoton microscope.

[0081] Step S2120: The classification model generation unit 13 outputs the generated classification model M1 to the cancer tissue determination unit 14. With this, the uterine cancer determination device 1 ends the learning process.

[0082] Step S2130: The feature extraction unit 112 discards the nucleus region image PNi for which it is determined that the number of cell nuclei is 0. In other words, the feature extraction unit 112 does not use the nucleus region image PNi for which it is determined that the number of cell nuclei is 0 in the subsequent processes (the processes from step S250 to step S2130).

[0083] Next, the determination process of the uterine cancer determination device 1 will be described with reference to Fig. 12. Fig. 12 is a diagram showing an example of the determination process according to this embodiment.

[0084] Step S300: The cancer tissue determination unit 14 acquires the determination feature data CE1 output from the data division unit 12. Step S310: The cancer tissue determination unit 14 acquires the classification model M1 output from the classification model generation unit 13.

[0085] Step S320: The cancer tissue determination unit 14 determines the possibility that the uterine tissue captured in the Z-stack THG image ZT1 is cancerous tissue using the determination feature data CE1 and the classification model M1. As described above, the feature C1 is extracted for each nuclear region image PNi included in the Z-stack nuclear region image ZN1 generated from the Z-stack THG image ZT1. Therefore, using the determination feature data CE1, it is possible to determine the possibility that the uterine tissue captured in the THG image PTi corresponding to the nuclear region image PNi from which the determination feature data CE1 was extracted is cancerous tissue.

[0086] As described above, the feature C1 is a feature that indicates the state of the cell nuclei captured in the nuclear region image PNi included in the Z-stack nuclear region image ZN1. Therefore, the cancer tissue determination unit 14 determines the possibility that the subject's uterine tissue is cancerous tissue based on the state of the cell nuclei in the third harmonic image of the subject's uterine tissue.

[0087] Furthermore, as described above, the cancer tissue determination unit 14 determines the likelihood that the uterine tissue captured in the Z-stack THG image ZT1 is cancerous tissue based on the classification model M1. Furthermore, as described above, the classification model M1 is a classification model trained using training third harmonic images of normal uterine tissue obtained by a multiphoton microscope and / or training third harmonic images of uterine cancer tissue obtained by a multiphoton microscope. Therefore, the cancer tissue determination unit 14 refers to the classification model M1 trained using training third harmonic images of normal uterine tissue obtained by a multiphoton microscope and / or training third harmonic images of uterine cancer tissue obtained by a multiphoton microscope, and determines the likelihood that the subject's uterine tissue is cancerous tissue based on the state of cell nuclei in the third harmonic images of the subject's uterine tissue.

[0088] As described above, the classification model M1 is a model obtained as a result of machine learning using the feature C1, which includes the median and median absolute error of the area of the cell nuclei, the median and median absolute error of the circularity of the cell nuclei, and the median and median absolute error of the nearest neighbor distance between the cell nuclei. The cancer tissue determination unit 14 determines that the uterine tissue of the subject is highly likely to be cancer tissue if at least one of the following conditions is met: (i) the mean area of the cell nuclei is increased compared to normal uterine tissue; (ii) the variability of the area of the cell nuclei is increased compared to normal uterine tissue; (iii) the density of the cell nuclei is increased compared to normal uterine tissue; (iv) the variability of the density of the cell nuclei is increased compared to normal uterine tissue; (v) the irregularity of the shape of the cell nuclei is increased compared to normal uterine tissue; and (vi) the variability of the shape of the cell nuclei is increased compared to normal uterine tissue.

[0089] Here, as an example, the cancer tissue determination unit 14 calculates the possibility that the uterine tissue is cancerous tissue based on the ratio of the determination Z-stack captured images ZP1 determined to have a high possibility of being cancerous tissue to all the determination Z-stack captured images ZP1. First, the cancer tissue determination unit 14 determines the possibility that the uterine tissue captured for each of the multiple THG images PT1 included in the determination Z-stack captured images ZP1 is cancerous tissue. Next, the cancer tissue determination unit 14 calculates the possibility that the uterine tissue is cancerous tissue based on the ratio of the determination Z-stack captured images ZP1 determined to have a high possibility of being cancerous tissue to all the determination Z-stack captured images ZP1.

[0090] In other words, the cancer tissue determination unit 14 determines the possibility that each of the multiple third harmonic images of the subject's uterine tissue is cancerous tissue, and calculates the possibility that the subject's uterine tissue is cancerous tissue based on the proportion of third harmonic images that are determined to have a high possibility of being cancerous tissue to all third harmonic images.

[0091] Step S330: The cancer tissue determination unit 14 outputs the determination result A to the output unit 15. With this, the uterine cancer determination device 1 ends the determination process.

[0092] 13 is a diagram showing an example of a determination result A according to this embodiment. The determination accuracy is expressed as a percentage, which is the ratio of the number of determination Z-stack captured images ZP1 for which a correct determination result was calculated to the number of determination Z-stack captured images ZP1 used in the determination process. The specimen "q1607" in FIG. 13 was the specimen used for the determination, and the determination accuracy for specimen "q1607" was 83.3 percent.

[0093] 13 also shows the determination accuracy for each sample for the learning Z-stack captured images ZP1, which include THG images of cancerous tissue and THG images of normal tissue. In FIG. 13, the accuracy was calculated for a certain sample as a determination sample and the others as learning samples. The average determination accuracy for all Z-stack captured images ZP1 was 90.1 percent. The uterine cancer determination device 1 can determine the possibility that uterine tissue is cancerous with a determination accuracy of about 90 percent.

[0094] (summary) As described above, the uterine cancer determination device 1 according to this embodiment includes the THG image acquisition unit 100 and the cancer tissue determination unit 14. The THG image acquisition unit 100 acquires a third harmonic image of the subject's uterine tissue obtained by the multiphoton microscope 2 (in this example, a Z-stack THG image ZT1). The cancer tissue determination unit 14 determines the possibility that the subject's uterine tissue is cancerous tissue based on the state of cell nuclei in the third harmonic image of the subject's uterine tissue (in this example, Z-stack THG image ZT1).

[0095] With this configuration, the uterine cancer detection device 1 of this embodiment can determine the possibility that the subject's uterine tissue is cancerous tissue based on the state of cell nuclei in the third harmonic image of the subject's uterine tissue, and therefore can determine the possibility that the uterine tissue is cancerous tissue without staining the uterine tissue.

[0096] In the uterine cancer determination device 1 according to this embodiment, the state of the cell nucleus is at least one selected from the group consisting of the area of the cell nucleus, the density of the cell nucleus, the shape of the cell nucleus, and the state of the nucleolus. With this configuration, the uterine cancer diagnosis device 1 of this embodiment can determine the possibility that the subject's uterine tissue is cancerous tissue based on at least one selected from the group consisting of the area of the cell nucleus, the density of the cell nucleus, the shape of the cell nucleus, and the state of the nucleoli in the third harmonic image of the subject's uterine tissue, and therefore can determine the possibility that the uterine tissue is cancerous tissue with higher accuracy than when not based on at least one selected from this group.

[0097] Furthermore, in the uterine cancer determination device 1 according to this embodiment, the cancer tissue determination unit 14 determines that the uterine tissue of the subject is highly likely to be cancer tissue when at least one of the following conditions is met: (i) the mean area of the cell nuclei is increased compared to normal uterine tissue; (ii) the variability of the area of the cell nuclei is increased compared to normal uterine tissue; (iii) the density of the cell nuclei is increased compared to normal uterine tissue; (iv) the variability of the density of the cell nuclei is increased compared to normal uterine tissue; (v) the irregularity of the shape of the cell nuclei is increased compared to normal uterine tissue; and (vi) the variability of the shape of the cell nuclei is increased compared to normal uterine tissue.

[0098] With this configuration, the uterine cancer detection device 1 of this embodiment can determine the possibility that uterine tissue is cancerous tissue based on (i) to (vi), which are characteristics that coincide with pathological findings about cell nuclei in cancerous tissue in uterine tissue, and can therefore determine the possibility that uterine tissue is cancerous tissue with higher accuracy than when not based on (i) to (vi).

[0099] Furthermore, in the uterine cancer determination device 1 according to this embodiment, the uterine tissue of the subject is exposed to acetic acid in advance. With this configuration, the uterine cancer detection device 1 of this embodiment can capture cell nuclei in the third harmonic image of the subject's uterine tissue more clearly than when the uterine tissue has not been exposed to acetic acid in advance, and therefore can determine the possibility that the uterine tissue is cancerous tissue with higher accuracy than when the uterine tissue has not been exposed to acetic acid in advance.

[0100] Furthermore, in the uterine cancer diagnosis device 1 according to this embodiment, the cancer tissue diagnosis unit 14 refers to a classification model M1 trained using a learning third harmonic image of normal uterine tissue obtained by the multiphoton microscope 2 and / or a learning third harmonic image of uterine cancer tissue obtained by the multiphoton microscope 2, and determines the possibility that the subject's uterine tissue is cancerous tissue based on the state of the cell nuclei in the third harmonic image of the subject's uterine tissue (in this example, the Z-stack THG image ZT1).

[0101] With this configuration, the uterine cancer detection device 1 of this embodiment can determine the possibility that the subject's uterine tissue is cancerous tissue by referring to the classification model M1, and can therefore determine the possibility that the uterine tissue is cancerous tissue with higher accuracy than when not referring to the classification model M1.

[0102] In addition, in the uterine cancer determination device 1 according to this embodiment, the learning third harmonic images are a plurality of cross-sectional images of uterine epithelial tissue. With this configuration, the uterine cancer detection device 1 of this embodiment can perform learning using multiple cross-sectional images of uterine epithelial tissue, and can therefore determine the possibility that uterine tissue is cancerous tissue with higher accuracy than when a single cross-sectional image is used for learning.

[0103] Furthermore, in the uterine cancer determination device 1 according to this embodiment, the third harmonic image of the uterine tissue of the subject (in this example, the Z-stack THG image ZT1) is a plurality of cross-sectional images of the uterine epithelial tissue of the subject. With this configuration, the uterine cancer detection device 1 of this embodiment can determine the possibility that the uterine tissue is cancerous tissue for each depth direction of the subject's uterine epithelial tissue, and can therefore determine the possibility that the uterine tissue is cancerous tissue with higher accuracy than when a single cross-sectional image is used for the determination.

[0104] Furthermore, in the uterine cancer diagnosis device 1 according to this embodiment, the cancer tissue diagnosis unit 14 determines the possibility that each of multiple third harmonic images (in this example, Z-stack THG image ZT1) of the subject's uterine tissue is cancerous tissue, and calculates the possibility that the subject's uterine tissue is cancerous tissue based on the proportion of third harmonic images determined to have a high possibility of being cancerous tissue to all third harmonic images (in this example, Z-stack THG image ZT1).

[0105] With this configuration, the uterine cancer diagnosis device 1 of this embodiment can calculate the possibility that the subject's uterine tissue is cancerous tissue based on the proportion of third harmonic images that are determined to have a high probability of being cancerous tissue out of multiple third harmonic images of the subject's uterine tissue, and therefore can calculate the possibility that the subject's uterine tissue is cancerous tissue for the entire depth direction.

[0106] (Second embodiment) The second embodiment will be described in detail below with reference to the drawings. In the first embodiment, a case where a nucleus region image is generated from a THG image based on a nucleus determination operation received from a user in a uterine cancer detection device has been described. In the present embodiment, a case where a uterine cancer detection device determines a nucleus region among regions included in a THG image based on machine learning and generates a nucleus region image will be described. The uterine cancer diagnosis device according to this embodiment is referred to as a uterine cancer diagnosis device 1a.

[0107] (Configuration of the uterine cancer detection device) FIG. 14 is a diagram showing an example of a uterine cancer determination device 1a according to this embodiment. Comparing the uterine cancer determination device 1a according to this embodiment (FIG. 14) with the uterine cancer determination device 1 according to the first embodiment (FIG. 4), the difference is that a memory unit 17 is provided instead of the nuclear region image generation unit 10a and the operation input unit 16. Here, the functions of the other components (feature amount processing unit 11, data division unit 12, classification model generation unit 13, cancer tissue determination unit 14, and output unit 15) are the same as those of the first embodiment. Description of the same functions as those of the first embodiment will be omitted, and the second embodiment will mainly be described with reference to the parts that are different from the first embodiment.

[0108] The uterine cancer diagnosis device 1a includes a nuclear region image generation unit 10a, a feature amount processing unit 11, a data division unit 12, a classification model generation unit 13, a cancer tissue diagnosis unit 14, an output unit 15, and a storage unit 17.

[0109] The nuclear region image generating unit 10a generates a Z-stack nuclear region image ZN1a from the Z-stack captured image ZP1 based on machine learning. Here, the machine learning used by the nuclear region image generating unit 10a is, for example, deep learning.

[0110] Here, an overview of deep learning used by the nucleus region image generating unit 10a will be described with reference to Fig. 15. Fig. 15 is a diagram showing an example of deep learning used by the nucleus region image generating unit 10a according to this embodiment.

[0111] As an example, the nucleus region image generation unit 10a uses U-Net, a deep learning architecture, as the nucleus region extraction model M20. U-Net is a convolutional neural network (CNN) that has been reported to have high accuracy in segmentation of medical images. By using U-Net, it is possible to capture the characteristics of the entire image and minimize errors in the correct image (distribution of cell nuclei and non-cell nuclei) at multiple resolutions for the entire image. All layers of U-Net are composed of convolutional layers, and an up-conv layer is provided toward the output layer.

[0112] As a parameter of the nuclear region extraction model M20, the input size is 512×512, which is the image size of the Z-stack captured image ZP1.

[0113] The nucleus region image generating unit 10a performs learning based on the nucleus region extraction model M20 using the training Z-stack THG image LT and the nucleus region annotation image PA. Here, the nucleus region annotation image PA is an image showing the nucleus region for each of the multiple THG images included in the training Z-stack THG image LT.

[0114] The nucleus region image generation unit 10a generates a nucleus region extraction model M21 as a result of the learning. The nucleus region image generation unit 10a applies the generated nucleus region extraction model M21 to the determination Z-stack THG image ET, and outputs a nucleus region image PN1, which is an image in which the nucleus region of the determination Z-stack THG image ET has been determined.

[0115] Returning to FIG. 14, the description of the configuration of the uterine cancer determination device 1a will be continued. The nucleus region image generating unit 10a includes a THG image acquiring unit 100a, a nucleus region annotation image acquiring unit 103a, a nucleus region extraction model generating unit 140a, a nucleus region determining unit 105a, and an image processing unit 106a.

[0116] The THG image acquisition unit 100a acquires a Z-stack captured image ZP1. The THG image acquisition unit 100a includes a learning THG image acquisition unit 101a and a determination THG image acquisition unit 102a. The Z-stack captured image ZP1 acquired by the THG image acquisition unit 100a includes a learning Z-stack THG image LT and a determination Z-stack THG image ET. The learning THG image acquisition unit 101a acquires a learning Z-stack THG image LT. The determination-use THG image acquisition unit 102a acquires a determination-use Z-stack THG image ET.

[0117] The nucleus region annotation image acquisition unit 103 a acquires the nucleus region annotation image PA from the storage unit 17 . The nucleus region extraction model generation unit 140a uses the learning Z-stack THG image LT and the nucleus region annotation image PA to perform learning based on the nucleus region extraction model M20, and generates a nucleus region extraction model M21 as a result of the learning.

[0118] The nucleus region determination unit 105a determines the nucleus region for the determination-use Z-stack THG image ET based on the nucleus region extraction model M21 generated by the nucleus region extraction model generation unit 140a. The image processing unit 106a generates a Z-stack nuclear region image ZN1a based on the determination result of the nuclear region determination unit 105a.

[0119] The storage unit 17 stores various types of information. The various types of information include a nucleus region annotation image PA and threshold information TH. The threshold information TH is information indicating various thresholds used in image processing performed by the image processing unit 106a.

[0120] (Nuclear region image generation processing) Next, details of the nuclear region image generation process in which the uterine cancer detection device 1a generates the Z-stack nuclear region image ZN1a will be described. The nuclear region image generation process includes a nuclear region learning process for generating a nuclear region extraction model M21 and a nuclear region determination process for determining the nuclear region based on the nuclear region extraction model M21.

[0121] FIG. 16 is a diagram showing an example of the core region learning process according to this embodiment. Step S400: The training THG image acquisition unit 101a acquires a training Z-stack THG image LT. The training THG image acquisition unit 101a supplies the acquired training Z-stack THG image LT to the nucleus region extraction model generation unit 140a.

[0122] Step S410: The nucleus region annotation image acquiring unit 103a acquires the nucleus region annotation image PA from the storage unit 17. The nucleus region annotation image acquiring unit 103a supplies the acquired nucleus region annotation image PA to the nucleus region extraction model generating unit 140a.

[0123] Step S420: The nucleus region extraction model generation unit 140a performs learning based on the nucleus region extraction model M20 using the training Z-stack THG image LT supplied from the training THG image acquisition unit 101a and the nucleus region annotation image PA supplied from the nucleus region annotation image acquisition unit 103a, and generates a nucleus region extraction model M21 as the learning result. Here, the nucleus region extraction model generation unit 140a generates the nucleus region extraction model M21 by changing the weights between the nodes of the nucleus region extraction model M20 based on the training Z-stack THG image LT and the nucleus region annotation image PA.

[0124] Step S430: The nucleus region extraction model generating unit 140a outputs the generated nucleus region extraction model M21 to the nucleus region determining unit 105a.

[0125] Next, the nucleus region determination process will be described with reference to Fig. 17. Fig. 17 is a diagram showing an example of the nucleus region determination process according to this embodiment. Step S500: The determination-use THG image acquisition unit 102a acquires a determination-use Z-stack THG image ET The determination-use THG image acquisition unit 102a supplies the acquired determination-use Z-stack THG image ET to the nucleus region determination unit 105a.

[0126] Step S510: The nucleus region determination unit 105a acquires the nucleus region extraction model M21 output from the nucleus region extraction model generation unit 140a.

[0127] Step S520: The nucleus region determination unit 105a determines a nucleus region for the determination Z-stack THG image ET supplied from the determination THG image acquisition unit 102a based on the acquired nucleus region extraction model M21. Here, the nucleus region determination unit 105a determines a nucleus region for each pixel of the multiple THG images PTi included in the determination Z-stack THG image ET. As a determination result, the nucleus region determination unit 105a assigns a probability value indicating the possibility that each pixel of the multiple THG images PTi is a nucleus region. The nucleus region determination unit 105a supplies the determination result of the nucleus region to the image processing unit 106a.

[0128] Step S530: The image processing unit 106a generates a Z-stack nucleus region probability value image based on the nucleus region determination result supplied from the nucleus region determination unit 105a. The Z-stack nucleus region probability value image is an image in which a probability value indicating the possibility that each pixel of the THG image PTi is a nucleus region is displayed. Here, the probability value indicating the possibility that the pixel is a nucleus region is displayed, for example, in grayscale.

[0129] Step S540: The image processing unit 106a acquires the binarization threshold value from the storage unit 17. Here, the binarization threshold value is included in the threshold value information TH. Step S550: The image processing unit 106a performs binarization processing. The image processing unit 106a performs binarization processing on the generated Z-stack nucleus region probability value image based on the acquired binarization threshold. The image processing unit 106a generates a Z-stack binarized image from the Z-stack nucleus region probability value image through the binarization processing.

[0130] Here, for each pixel in the Z-stack nucleus region probability value image, if the probability value indicating the possibility that it is a nucleus region is equal to or greater than the binarization threshold, the image processing unit 106a assigns to this pixel, for example, one of the binary luminance values, brightness value 1 (for example, a brightness value corresponding to white). For each pixel in the Z-stack nucleus region probability value image, if the probability value indicating the possibility that it is a nucleus region is less than the binarization threshold, the image processing unit 106a assigns to this pixel the other of the binary luminance values, brightness value 2 (for example, a brightness value corresponding to black).

[0131] Step S560: The image processing unit 106a performs a hole filling process on the generated Z-stack binarized image. Here, the image processing unit 106a uses, for example, closing or opening as the hole filling process. The image processing unit 106a generates a Z-stack hole-filled image from the Z-stack binarized image through the hole filling process.

[0132] Step S570: The image processing unit 106a acquires the area threshold and the circularity threshold from the storage unit 17. Here, the area threshold and the circularity threshold are included in the threshold information TH. Step S580: The image processing unit 106a performs a filtering process The image processing unit 106a performs a filtering process on the generated Z-stack filled image based on the acquired area threshold and circularity threshold.

[0133] Here, the image processing unit 106a pairs adjacent pixels among the pixels included in the Z-stack filled image that have been assigned a brightness value of 1 (for example, pixels that have been assigned a white color), and determines these pixels as candidate regions that are candidates for the nucleus region. The image processing unit 106a also determines a candidate region when only one pixel is isolated.

[0134] The image processing unit 106a measures the area and circularity of each of the determined candidate regions. For a given candidate region, if the measured area is equal to or greater than the area threshold and the measured circularity is equal to or greater than the circularity threshold, the image processing unit 106a determines not to exclude the candidate region. On the other hand, for a given candidate region, if the measured area is less than the area threshold or the measured circularity is less than the circularity threshold, the image processing unit 106a determines to exclude the candidate region.

[0135] The image processing unit 106a changes the brightness values of pixels included in a candidate region determined to be excluded from among the candidate regions included in the Z-stack filled image to a brightness value of 2. For example, if the candidate region is white, the image processing unit 106a changes the color of the candidate region determined to be excluded to black.

[0136] Step S590: The image processing unit 106a performs segmentation processing. The image processing unit 106a performs segmentation processing on the filtered Z-stack filled-up image. The image processing unit 106a uses, for example, WaterShed as a segmentation method. The image processing unit 106a determines the outline of the nuclear region through the segmentation processing, and segments the filtered Z-stack filled-up image into a nuclear region and a region other than the nuclear region. The image processing unit 106a generates a Z-stack nuclear region image ZN1a as a result of the segmentation process.

[0137] Step S5100: The image processing unit 106a outputs the generated Z-stack nuclear region image ZN1a to the feature amount processing unit 11. With this, the nuclear region image generating unit 10a ends the nuclear region determination process.

[0138] The results of the core region determination process will be described with reference to FIGS. Fig. 18 is a diagram showing an example of a nucleus region image and a nucleus region annotation image according to this embodiment. Fig. 18 shows a nucleus region image PNj as an example of a nucleus region image obtained by the nucleus region determination process. The nucleus region annotation image PAj is the nucleus region annotation image used to generate this nucleus region image PNj.

[0139] Next, the determination accuracy of the core region determination process will be described with reference to FIG. 19 is a diagram showing an example of the results of the nucleus region determination process according to this embodiment. The nucleus region shown in the nucleus region image obtained by the nucleus region determination process includes a first region and a second region. The first region is a region that matches the nucleus region shown in the nucleus region annotation image. The second region is a region that is erroneously determined and does not match the nucleus region shown in the nucleus region annotation image. Meanwhile, the nucleus region shown in the nucleus region annotation image includes a third region that is not included in the nucleus region of the nucleus region image because it was not determined.

[0140] 19, the nuclear region image obtained by the nuclear region determination process is shown with the first and second regions distinguished from each other, and further with the third region added, resulting in images PC1 and PC2. Image PC1 is a nuclear region image of normal tissue. Image PC2 is a nuclear region image of cancer tissue.

[0141] Here, the F-value is used to evaluate the accuracy of the nucleus region determination process. The F-value is the harmonic mean of the precision and recall, and ranges from 0 to 1, with the closer to 1 the value is the higher the determination accuracy. The precision is the ratio of the number of first regions to the number of extracted nucleus regions (first and second regions). The recall is the ratio of the number of first regions to the number of nucleus regions (first and third regions) indicated by the nucleus region annotation image.

[0142] By using the F value, it is possible to calculate the determination accuracy that is not dependent on the ratio of the area occupied by the nuclear region to the area occupied by the background other than the nuclear region in the nuclear region image. The F-value of image PC1 was 0.932, and the F-value of image PC2 was 0.909.

[0143] Next, referring to Fig. 20, the F-value for the entire Z-stack nuclear region image ZN1a is shown. Fig. 20 is a diagram showing an example of the F-value for the entire Z-stack nuclear region image ZN1a according to this embodiment. The average F-value for the entire Z-stack nuclear region image ZN1a was 0.854. In the uterine cancer determination device 1a, nuclear regions were extracted with high accuracy from both the image PC1 and the image PC2.

[0144] Next, the determination result Aa of the uterine cancer determination process of the uterine cancer determination device 1a will be described with reference to Fig. 21. Fig. 21 is a diagram showing an example of the determination result Aa according to this embodiment. The uterine cancer determination process of the uterine cancer determination device 1a is a uterine cancer determination process when a Z-stack nuclear region image ZN1a is generated by the nuclear region determination process. Upon completion of the nuclear region determination process, the uterine cancer determination device 1a performs the uterine cancer determination process by performing processes similar to the learning process of Fig. 10 and the determination process of Fig. 12 described above.

[0145] The average value of the determination accuracy for all Z-stack captured images ZP1, which was calculated by using a certain sample as a determination sample and the others as learning samples and repeating this process for the number of samples, was 90.7%. According to the uterine cancer determination device 1a, even when the Z-stack nuclear region image ZN1a is generated by the nuclear region determination process, it is possible to determine the possibility that the uterine tissue is cancerous tissue with a determination accuracy of about 90%.

[0146] (summary) As described above, the uterine cancer determination device 1a according to this embodiment includes the nuclear region image generation unit 10a. The nuclear region image generation unit 10a generates Z-stack nuclear region images ZN1a, which are a plurality of images in which regions representing cell nuclei of uterine tissues are determined from the Z-stack captured images ZP1 based on machine learning. With this configuration, the uterine cancer determination device 1a according to this embodiment can generate a Z-stack nuclear region image ZN1a based on machine learning, thereby reducing the effort required to determine the region showing the cell nuclei of the uterine tissue from the Z-stack captured image ZP1.

[0147] (Third embodiment) The third embodiment will be described in detail below with reference to the drawings. In the first and second embodiments, the case where the uterine cancer determination device determines the possibility that the uterine tissue of a subject is cancerous tissue based on a THG image is described. In the present embodiment, the case where the uterine cancer determination device determines the stage of cancer based on a second harmonic image for uterine tissue determined to be highly likely to be cancerous tissue based on a THG image is described. The uterine cancer diagnosis system according to this embodiment is referred to as uterine cancer diagnosis system STb. The uterine cancer diagnosis device according to this embodiment is referred to as uterine cancer diagnosis device 1b, and the multiphoton microscope is referred to as multiphoton microscope 2b.

[0148] As a tumor progresses, fibrosis occurs in the surrounding tissue. The fibers produced by fibrosis are composed of various collagen-containing molecules. Some fibrous collagens generate SHG. In addition to THG images, the multiphoton microscope 2b of this embodiment uses SHG to capture images of uterine tissue and generate second harmonic images. The second harmonic images are images of a cross section of uterine tissue generated based on the excitation light irradiated from the irradiation unit of the multiphoton microscope 2b and the light generated by second harmonic generation due to interaction with the uterine tissue.

[0149] In this embodiment, the captured images included in the Z-stack captured image include a THG image and a second harmonic image, which is an image of uterine tissue captured using SHG. Hereinafter, the second harmonic image will be referred to as an SHG image. Also, the Z-stack captured image of the SHG image will be referred to as a Z-stack SHG image ZS1. In other words, the Z-stack SHG image ZS1 is a plurality of SHG images of uterine tissue captured using SHG at different distances from the uterine tissue in the Z-axis direction.

[0150] Furthermore, each of the multiple images included in the Z stack SHG image ZS1 is referred to as an SHG image PSi (i=1, 2, . . . , N: N is the number of images included in the Z stack SHG image ZS1), etc. The multiphoton microscope 2b captures a THG image and an SHG image for each depth of the uterine tissue. The SHG image PSi included in the Z-stack SHG image ZS1 and the THG image PTi included in the Z-stack THG image ZT1 correspond to each other because they are images captured at a common depth of the uterine tissue.

[0151] Here, referring to FIG. 22, an SHG image captured by the multiphoton microscope 2b will be described. FIG. 22 is a diagram showing an example of an SHG image PS0 according to this embodiment. For comparison, FIG. 22 shows a fiber-like structure image F1 together with the SHG image PS0. The fiber-like structure image F1 is an image extracted and detected from the SHG image PS0 by machine learning. It can be seen that the SHG image PS0 captured by the multiphoton microscope 2b captures fiber-like structures to the same extent as those captured in the fiber-like structure image F1.

[0152] As the tumor progresses, fibrosis spreads from deep to superficial uterine tissue.

[0153] The spread of fibrosis from deep to shallow parts of uterine tissue will now be described with reference to Fig. 23. Fig. 23 is a diagram showing an example of cross sections of uterine tissue for each stage of tumor progression according to this embodiment. In Figure 23A, fibrous structures are present in the deep part of the uterine tissue, while in Figure 23B, a cross section of the uterine tissue after tumor progression shows fibrosis in the superficial part.

[0154] In this embodiment, the Z axis is also set in a direction from the epithelial tissue toward the inside of the uterine tissue. As the tumor progresses, fibrosis spreads from areas with high Z coordinate values to areas with low Z coordinate values in the uterine tissue. In the shallow part of the uterine tissue, the depth at which fibrosis occurs is unknown in advance because it differs depending on the stage of the tumor. By using the Z-stack SHG image ZS1, the uterine cancer diagnosis device 1b can extract fibrous structures regardless of the depth at which fibrosis occurs in the shallow part of the tissue.

[0155] (Configuration of the uterine cancer detection device) FIG. 24 is a diagram showing an example of a uterine cancer determination device 1b according to this embodiment. Comparing the uterine cancer determination device 1b according to this embodiment (FIG. 24) with the uterine cancer determination device 1a according to the second embodiment (FIG. 14), the SHG image acquisition unit 18b and the cancer progression determination unit 19b are different. Here, the functions of the other components (nuclear region image generation unit 10a, feature amount processing unit 11, data division unit 12, classification model generation unit 13, cancer tissue determination unit 14, and output unit 15) are the same as those of the second embodiment. Description of the same functions as those of the second embodiment will be omitted, and the third embodiment will mainly be described with reference to the portions different from the second embodiment.

[0156] The uterine cancer assessment device 1b classifies the stage of tumor progression by extracting fibrous structural features from Z-stack SHG images ZS1 of the shallow part of the uterine tissue using a multiphoton microscope 2b and quantifying them as the amount of fibrosis. Here, the stage of tumor progression is classified into non-invasive and invasive, for example. Non-invasive corresponds to pre-cancer, and invasive corresponds to invasive cancer.

[0157] The uterine cancer determination device 1b includes a nuclear region image generation unit 10a, a feature amount processing unit 11, a data division unit 12, a classification model generation unit 13, a cancer tissue determination unit 14, an output unit 15, a storage unit 17, an SHG image acquisition unit 18b, and a cancer progression determination unit 19b. Note that the uterine cancer determination device 1b may include a nuclear region image generation unit 10 instead of the nuclear region image generation unit 10a, similar to the uterine cancer determination device 1 according to the first embodiment (FIG. 4).

[0158] The SHG image acquisition unit 18b acquires a Z-stack SHG image ZS1 captured by the multiphoton microscope 2b. When the cancer tissue determination unit 14 determines that the subject's uterine tissue is highly likely to be cancer tissue, the cancer progression determination unit 19b determines the progression of the cancer based on the state of the fibrous structure in the SHG image PSi contained in the Z-stack SHG image ZS1. The cancer progression determination unit 19b includes a fibrous structure determination unit 190b, a fibrous proliferation amount calculation unit 191b, and a precancer / invasive cancer determination unit 192b.

[0159] The fibrous structure determination unit 190b determines whether the Z-stack SHG image ZS1 has a fibrous structure. In this embodiment, the fibrous structure determination unit 190b determines whether the Z-stack SHG image ZS1 has a fibrous structure based on machine learning, as an example. The fibrous structure determination unit 190b generates a Z-stack fibrous structure image ZF1 from the Z-stack SHG image ZS1 based on the determination result. The Z-stack fibrous structure image ZF1 is a Z-stack image in which a fibrous structure is shown. Each of the multiple images included in the Z-stack fiber-like structure image ZF1 is referred to as a fiber-like structure image PFi (i=1, 2, . . . , N: N is the number of images included in the Z-stack fiber-like structure image ZF1).

[0160] The fibrous proliferation amount calculation unit 191b calculates the amount of fibrous proliferation in the shallow part of the uterine tissue captured as the Z-stack SHG image ZS1 based on the Z-stack fibrous structure image ZF1 generated by the fibrous structure determination unit 190b. The precancer / invasive cancer determination unit 192b determines whether the uterine tissue determined to be highly likely to be cancerous tissue is precancer or invasive cancer based on the amount of fibrosis calculated by the fibrosis amount calculation unit 191b.

[0161] (Processing of uterine cancer diagnosis system) Next, a uterine cancer detection process, which is a process performed by the uterine cancer detection system STb, will be described. Fig. 25 is a diagram showing an example of the uterine cancer detection process according to this embodiment. The processes in steps S610, S620, S630, and S640 are similar to the processes in steps S10, S20, S30, and S40 in FIG. 7, and therefore will not be described.

[0162] Step S650: The uterine cancer determination device 1b performs a precancer / invasive cancer determination process, which is a process for determining whether the uterine tissue of the subject is precancerous tissue or invasive cancer tissue, based on the Z-stack SHG image ZS1 captured by the multiphoton microscope 2b. The precancerous / invasive cancer determination process will be described in detail later with reference to FIG. 26.

[0163] Step S660: The output unit 15 of the uterine cancer determination device 1b performs processing based on the determination result Ab of the cancer progression determination unit 19b. The determination result Ab indicates whether the uterine tissue of the subject is precancerous or invasive cancer. If the uterine tissue of the subject is determined to be precancerous (step S660; YES), the output unit 15 outputs a result indicating that the uterine tissue of the subject is precancerous to the display device 3. Thereafter, the display device 3 performs the processing of step S670. On the other hand, if it is determined that the subject's uterine tissue is not precancerous, i.e., is invasive cancer (step S660; NO), the output unit 15 outputs to the display device 3 a result indicating that the subject's uterine tissue is invasive cancer.

[0164] Step S670: The display device 3 displays the result indicating that the uterine tissue of the subject is precancerous. Step S680: The display device 3 displays the result indicating that the subject's uterine tissue has invasive cancer. With this, the uterine cancer detection system STb ends the uterine cancer detection process.

[0165] Next, the precancerous / invasive cancer determination process will be described with reference to Fig. 26. Fig. 26 is a diagram showing an example of the precancerous / invasive cancer determination process according to this embodiment. Step S700: The SHG image acquiring unit 18b acquires a Z-stack SHG image ZS1 captured by the multiphoton microscope 2b. That is, the SHG image acquiring unit 18b acquires a second harmonic image of the uterine tissue of the subject obtained by the multiphoton microscope 2b. The SHG image acquisition unit 18b supplies the acquired Z stack SHG image ZS1 to the fibrous structure determination unit 190b.

[0166] Step S710: The fibrous structure determination unit 190b determines whether the Z-stack SHG image ZS1 has a fibrous structure. Based on the determination result, the fibrous structure determination unit 190b generates a Z-stack fibrous structure image ZF1 from the Z-stack SHG image ZS1. The fibrous structure determination unit 190b supplies the generated Z-stack fibrous structure image ZF1 to the fibrous augmentation amount calculation unit 191b.

[0167] Here, the fibrous structure determining unit 190b performs the determination based on machine learning, for example, using U-Net as the machine learning.

[0168] Here, the fiber-like structure image PF1 included in the Z-stack fiber-like structure image ZF1 generated by the fiber-like structure determination unit 190b will be described with reference to Fig. 27. Fig. 27 is a diagram showing an example of the fiber-like structure image PF1 according to this embodiment. The fibrous structure image PF1 is an image in which the fibrous structure is determined based on the SHG image PS1 included in the Z-stack SHG image ZS1. As an example, the SHG image PS1 is an SHG image of tissue at a depth of 15 μm from the epithelial tissue of the uterus tissue.

[0169] In the present embodiment, an example has been described in which the fibrous structure determination unit 190b determines the fibrous structure in the Z-stack SHG image ZS1 using machine learning, but the present invention is not limited to this. The determination of the fibrous structure may be performed by a user of the uterine cancer determination device 1b. When the determination of the fibrous structure is performed by a user of the uterine cancer determination device 1b, the cancer stage determination unit 19b, for example, receives an operation to determine the fibrous structure in the Z-stack SHG image ZS1 via the operation input unit 16. The cancer stage determination unit 19b generates a Z-stack fibrous structure image ZF1 based on the received operation to determine the fibrous structure.

[0170] Returning to FIG. 26, the description of the precancer / invasive cancer determination process will be continued. Step S720: The fibrous structure amount calculation unit 191b calculates the amount of fibrous structure in the shallow part of the uterine tissue captured as the Z-stack SHG image ZS1 based on the Z-stack fibrous structure image ZF1 generated by the fibrous structure determination unit 190b. The fibrous structure amount calculation unit 191b supplies the calculated amount of fibrous structure to the precancer / invasive cancer determination unit 192b.

[0171] Here, the fibrosis amount calculation unit 191b calculates the amount of fibrosis as the ratio of the area of the portion where the fibrous structure is imaged to the entire area of the Z-stack fibrous structure image ZFi. The fibrosis amount calculation unit 191b calculates the area based on the number of pixels. In other words, the fibrosis amount calculation unit 191b calculates the amount of fibrosis as the ratio of the number of pixels of the portion where the fibrous structure is imaged to the entire number of pixels of the Z-stack fibrous structure image ZFi. Hereinafter, this ratio will be referred to as the detected pixel ratio.

[0172] The detected pixel ratio calculated by the fibrosis amount calculation unit 191b will now be described with reference to Fig. 28 and Fig. 29. Fig. 28 is a diagram showing an example of the detected pixel ratio according to this embodiment. In Fig. 28, the detected pixel ratio for each of 16 specimens of invasive cancerous uterine tissue, 5 specimens of precancerous uterine tissue, and 15 specimens of normal uterine tissue is shown relative to the Z axis of the Z stack image. As mentioned above, the pre-cancer / invasive cancer determination process is performed after the uterine tissue is determined to be cancerous tissue. However, in Figure 28, in addition to the cancerous tissue (invasive cancer and pre-cancer), the detection pixel ratio calculated using Z-stack SHG images of normal tissue is also shown for comparison.

[0173] As the Z-axis coordinate value increases, the depth of the imaged uterine tissue from the epithelial tissue in the shallow part increases. The Z-coordinate value of the epithelial tissue is 0. The Z-coordinate value of the boundary between the shallow and deep parts of the uterine tissue is 50.

[0174] FIG. 29 is a diagram showing an example of the average value of the detected pixel ratio for the shallow part of uterine tissue according to this embodiment. The average value for the shallow part of uterine tissue is the average for the Z-axis value of the Z-stack image. The three graphs shown in FIG. 29 show the average value of the detected pixel ratio for the shallow part of uterine tissue, for each number indicating each specimen. Graph G1 shows the average value of the detected pixel ratio for invasive cancerous uterine tissue. Graph G2 shows the average value of the detected pixel ratio for precancerous uterine tissue. Graph G3 shows the average value of the detected pixel ratio for normal uterine tissue.

[0175] Graphs G1, G2, and G3 show that the detected pixel ratio is higher in cancerous tissue than in normal tissue. Graphs G1 and G2 also show that the detected pixel ratio is higher in invasive cancer than in precancer. The detected pixel ratio increases as the cancer progresses. Here, the detected pixel ratio corresponds to the amount of fibrosis in the superficial part of the uterine tissue. Therefore, the amount of fibrosis in the superficial part of the uterine tissue increases as the cancer progresses.

[0176] Returning to FIG. 26, the description of the precancer / invasive cancer determination process will be continued. Step S730: The precancer / invasive cancer determination unit 192b determines whether the uterine tissue determined to be highly likely to be cancerous tissue is precancer or invasive cancer based on the detected pixel ratio calculated by the fibrosis amount calculation unit 191b. The precancer / invasive cancer determination unit 192b supplies the determination result as a determination result Ab to the output unit 15.

[0177] Here, the precancer / invasive cancer determination unit 192b determines that the specimen is invasive cancer when the average value of the detected pixel ratio is higher than a predetermined determination threshold.On the other hand, the precancer / invasive cancer determination unit 192b determines that the specimen is precancer when the average value of the detected pixel ratio is lower than a predetermined determination threshold. With the above, the cancer progression determination unit 19b ends the precancer / invasive cancer determination process. In this embodiment, the explanation has been given using the average value of the detection pixel ratio for the shallow part of the uterine tissue, but this is not limited to this, and for example, the maximum value of the detection pixel ratio for the shallow part may be replaced with the average value to make the judgment.

[0178] Here, the determination rate of the cancer stage determination unit 19b will be described using an ROC (Receiver Operating Characteristic) curve with reference to Fig. 30 and Fig. 31. Fig. 30 is a diagram showing an example of an ROC curve indicating the determination rate of the cancer stage determination unit 19b according to this embodiment. Fig. 31 is a diagram showing an example of the relationship between the determination threshold and the determination rate of the cancer stage determination unit 19b according to this embodiment.

[0179] The ROC curve in Figure 30 shows the rate of detecting invasive cancer as sensitivity against the rate of detecting precancer as specificity. In Figure 30, the evaluation is performed by corresponding invasive cancer to a positive result and precancer to a negative result. In other words, the rate of detecting precancer corresponds to the true negative rate, and the rate of detecting invasive cancer corresponds to the true positive rate. In the ROC curve in Figure 30, the AUC (Area under the ROC curve) value is 0.88.

[0180] As shown in Fig. 31, when the judgment threshold used by the precancer / invasive cancer judgment unit 192b was changed, a judgment rate of 81.3% sensitivity and 80.0% specificity was obtained when the judgment threshold was 0.0025. The AUC value of the ROC curve in Fig. 30 is the value when the judgment threshold is 0.0025.

[0181] In this embodiment, an example has been described in which the cancer progression determination unit 19b determines whether the stage of cancer is precancer or invasive cancer, but the present invention is not limited to this. The cancer progression determination unit 19b may determine whether the stage of cancer is any of the following: (i) Precancerous; (b) Invasive cancer; (c) mild dysplasia (CIN1), moderate dysplasia (CIN2), or severe dysplasia / carcinoma in situ (CIN3); (ii) Microinvasive squamous cell carcinoma or squamous cell carcinoma; or (e) Cancer requiring treatment or cancer not requiring treatment.

[0182] (summary) As described above, the uterine cancer diagnosing device 1b according to this embodiment includes a second harmonic image acquiring section (in this example, an SHG image acquiring section 18b) and a cancer progression determining section 19b. The second harmonic image acquisition unit (in this example, the SHG image acquisition unit 18b) acquires a second harmonic image (in this example, a Z-stack SHG image ZS1) of the subject's uterine tissue obtained by the multiphoton microscope 2b. When the cancer tissue determination unit 14 determines that the subject's uterine tissue is highly likely to be cancer tissue, the cancer progression determination unit 19b determines the progression of the cancer (in this example, classified as precancer and invasive cancer) based on the state of the fibrous structure in the second harmonic image (in this example, Z-stack SHG image ZS1).

[0183] With this configuration, the uterine cancer determination device 1b according to this embodiment can determine the stage of cancer based on the state of the fibrous structure in the second harmonic image of the shallow part of the uterine tissue of the subject, and therefore can determine the stage of cancer in the uterine tissue without staining the uterine tissue. In other words, the uterine cancer determination device 1b can determine the stage of cancer in the uterine tissue without examining the deep part of the uterine tissue and without staining the uterine tissue.

[0184] Furthermore, in the uterine cancer determination device 1b according to this embodiment, the cancer progression determination unit 19b determines whether the progression of cancer falls under any of the following conditions. (i) Precancerous; (b) Invasive cancer; (c) mild dysplasia (CIN1), moderate dysplasia (CIN2), or severe dysplasia / carcinoma in situ (CIN3); (ii) Microinvasive squamous cell carcinoma or squamous cell carcinoma; or (e) Cancer requiring treatment or cancer not requiring treatment.

[0185] With this configuration, the uterine cancer diagnosis device 1b of this embodiment can determine whether the cancer progresses according to the above items (i) to (e) based on the state of the fibrous structure in the second harmonic image of the shallow part of the subject's uterine tissue, and therefore can determine whether the cancer progresses according to the above items (i) to (e) without staining the uterine tissue.

[0186] (Fourth embodiment) The fourth embodiment will be described in detail below with reference to the drawings. There are two types of uterine cancer: cervical cancer, which is cancer of the tissues of the cervix, and endometrial cancer, which is cancer of the tissues of the uterine body. Cervical cancer progresses from cervical intraepithelial carcinoma, in which cancer develops in the epithelial tissue, to invasive cancer, depending on the stage of the cancer. As the cancer progresses, fibrosis is known to occur in the surrounding tissues. On the other hand, a technique for identifying cancer tissue by analyzing captured images of biological tissue is known. When identifying cancer tissue by image analysis, images of stained biological tissue are used.

[0187] For example, a cancer screening device is known that captures an image of a group of biological cells coated with a dye that selectively stains cancer-related gene products in the biological cells in a chromatic color, and includes a judgment unit that judges the level of malignancy of the cancerous transformation of the group of biological cells based on the staining state of the group of biological cells in the image obtained (Patent Document 1). Previously, it was difficult to analyze the state of unstained tissue at the cellular level.

[0188] 32 is a diagram showing an example of a uterine cancer progress determination system STc according to this embodiment. The uterine cancer progress determination system STc includes a cancer progress determination device 1c, a multiphoton microscope 2c, and a display device 3c. In the uterine cancer progress determination system STc, the cancer progress determination device 1c analyzes Z-stack captured images ZP1c of the subject's uterine tissue captured by the multiphoton microscope 2c, and determines the progress of the subject's uterine cancer. In this embodiment, the uterine tissue is, for example, tissue of the cervix and tissue of the uterine body.

[0189] The Z-stack image ZP1c is a Z-stack image. A Z-stack image is a set of multiple images captured at different distances from the uterine tissue in the Z-axis direction. The Z-stack image ZP1c is made up of multiple images of the uterine tissue at each Z coordinate. Here, capturing an image of the uterine tissue at each Z coordinate means capturing an image of the uterine tissue at various distances between the objective lens and the uterine tissue. The Z-stack image ZP1c is a multiple image captured by the multiphoton microscope 2c at various distances between the uterine tissue and the lens.

[0190] In this embodiment, the Z axis is selected to point in the direction from the epithelial tissue of the uterine tissue toward the interior. In other words, the closer the imaging plane is to the uterine epithelial tissue, the smaller the Z coordinate value, and the deeper the imaging plane is into the uterine tissue, the larger the Z coordinate value. The origin of the Z axis is selected at a shallow position close to the uterine epithelial tissue. In this way, the Z-stack captured image ZP1c is a plurality of cross-sectional images perpendicular to the depth direction of the subject's uterine epithelial tissue, and is also simply referred to as a cross-sectional image. Note that the depth direction is the direction from the superficial layer to the basal layer, and the plurality of cross-sectional images do not need to be perfectly perpendicular to the depth direction of the subject's uterine epithelial tissue, but may be tilted by approximately ±5 degrees.

[0191] The multiphoton microscope 2c observes and images the subject's uterine tissue in an unstained state. The multiphoton microscope 2c images the uterine tissue using nonlinear optical phenomena. The nonlinear optical phenomena used by the multiphoton microscope 2c for imaging include second harmonic generation (SHG). SHG is a phenomenon in which light with twice the frequency of the excitation light is generated by interaction with nonlinear optical crystals such as collagen fibers.

[0192] As a tumor progresses, fibrosis occurs in the surrounding tissue. The fibers produced by fibrosis are composed of various collagen-containing molecules. Some fibrous collagens generate SHG. In this embodiment, the multiphoton microscope 2c uses SHG to image the uterine tissue and generate a second harmonic image. The second harmonic image is an image of a cross section of the uterine tissue generated based on the excitation light irradiated from the irradiation unit of the multiphoton microscope 2c and the light generated by second harmonic generation due to the interaction between the excitation light and the uterine tissue. Hereinafter, the second harmonic image will be referred to as the SHG image.

[0193] In this embodiment, since the Z-stack captured image ZP1c is a Z-stack captured image of an SHG image, hereinafter, the Z-stack captured image ZP1c will be referred to as the Z-stack SHG image ZS1c. In other words, the Z-stack SHG image ZS1c is a plurality of SHG images of the uterine tissue captured using SHG at different distances from the uterine tissue in the Z-axis direction. In other words, the Z-stack SHG image ZS1c is a plurality of cross-sectional images of the uterine epithelial tissue captured at multiple depths. Here, these cross-sectional images are second harmonic generation images obtained by the multiphoton microscope 2c.

[0194] Furthermore, each of the multiple images included in the Z stack SHG image ZS1c is referred to as an SHG image PSic (i=1, 2, . . . , N: N is the number of images included in the Z stack SHG image ZS1c), for example. The multiphoton microscope 2c captures SHG images at each depth of the uterine tissue.

[0195] Here, an SHG image captured by the multiphoton microscope 2c will be described with reference to Fig. 33. Fig. 33 is a diagram showing an example of an SHG image PS0c according to this embodiment. For comparison, Fig. 33 shows a fiber-like structure image F1c together with the SHG image PS0c. It can be seen that the SHG image PS0c captured by the multiphoton microscope 2c captures fiber-like structures to the same extent as those captured in the fiber-like structure image F1c.

[0196] As the tumor progresses, fibrosis spreads from deep to superficial uterine tissue.

[0197] The spread of fibrosis from deep to shallow parts of uterine tissue will now be described with reference to Fig. 34. Fig. 34 is a diagram showing an example of cross sections of uterine tissue for each stage of tumor progression according to this embodiment. In Figure 34A, fibrous structures are present in the deep part of the uterine tissue, while in Figure 34B, a cross section of the uterine tissue after tumor progression shows fibrosis in the superficial part.

[0198] In this embodiment, the Z axis is also set in a direction from the epithelial tissue toward the inside of the uterine tissue. As the tumor progresses, fibrosis spreads from areas with high Z coordinate values to areas with low Z coordinate values in the uterine tissue. The depth at which fibrosis occurs in the shallow part of the uterine tissue is unknown in advance because it varies depending on the stage of the tumor. By using the Z-stack SHG image ZS1c, the cancer progression assessment device 1c can extract fibrous structures regardless of the depth at which fibrosis occurs in the shallow part of the tissue.

[0199] (Configuration of the cancer progression assessment device) Next, the configuration of the cancer progression determination device 1c will be described with reference to Fig. 35. Fig. 35 is a diagram showing an example of the cancer progression determination device 1c according to this embodiment. The cancer progression determination device 1c is, for example, a computer.

[0200] The cancer progression assessment device 1c includes a cross-sectional image acquisition unit 10c, a fiber pixel extraction unit 11c, a fibrosis amount calculation unit 12c, a cancer progression assessment unit 13c, an output unit 14c, and a storage unit 15c. The cross-sectional image acquisition unit 10c, the fiber pixel extraction unit 11c, the fibrosis amount calculation unit 12c, the cancer progression assessment unit 13c, and the output unit 14c are each modules implemented by a CPU (Central Processing Unit) reading a program from a ROM (Read Only Memory) and executing the process.

[0201] The cross-sectional image acquisition unit 10c acquires a Z-stack SHG image ZS1c captured by the multiphoton microscope 2.

[0202] The fiber pixel extraction unit 11c extracts fiber pixels, which are pixels in which a fiber-like structure is captured, from among the pixels in each SHG image PSic included in the Z-stack SHG image ZS1c. In this embodiment, the fiber pixel extraction unit 11c extracts fiber pixels based on machine learning, as an example. The fiber pixel extraction unit 11c generates a Z-stack fiber-like structure image ZF1c from the Z-stack SHG image ZS1c based on the extracted fiber pixels. The Z-stack fiber-like structure image ZF1c is a Z-stack image in which a fiber-like structure is displayed. Each of the multiple images included in the Z-stack fiber-like structure image ZF1c is referred to as a fiber-like structure image PFic (i=1, 2, . . . , N: N is the number of images included in the Z-stack fiber-like structure image ZF1c), etc.

[0203] The fibrous structure amount calculation unit 12c calculates the amount of fibrous structure captured in the Z-stack fibrous structure image ZF1c generated by the fiber pixel extraction unit 11c. Hereinafter, the amount of fibrous structure captured in the Z-stack SHG image ZS1c may be referred to as the fibrous structure amount. The cancer progression determination unit 13c determines the progression of uterine cancer in the subject based on the amount of fibrosis calculated by the fibrosis amount calculation unit 12c. In this embodiment, the progression of uterine cancer is classified in order of decreasing degree of progression, for example, into non-cancer, pre-cancer, or invasive cancer.

[0204] The output unit 14c outputs the determination result Ac of the cancer progression determination unit 13c to the display device 3c. The determination result Ac indicates the progression stage of the uterine cancer in the subject. The display device 3c displays the determination result Ac determined by the cancer progression stage determination device 1c. The display device 3c is, for example, a display.

[0205] The memory unit 15c stores various types of information. This information includes, for example, threshold information THc calculated from case images of the subject. The threshold information THc is information indicating a threshold for the amount of fibrosis, and includes a first threshold TH1c and a second threshold TH2c. The first threshold TH1c indicates the boundary between the amount of precancerous fibrosis and the amount of non-cancerous fibrosis. The second threshold TH2c indicates the boundary between the amount of precancerous fibrosis and the amount of invasive cancer fibrosis. The second threshold TH2c is greater than the first threshold TH1c.

[0206] (Cancer progression assessment process) Next, a cancer stage determination process in which the cancer stage determination device 1c determines the stage of uterine cancer in a subject will be described. Fig. 36 is a diagram showing an example of the cancer stage determination process according to this embodiment.

[0207] Step S810: The cross-sectional image acquisition unit 10c acquires a Z-stack image ZP1c captured by the multiphoton microscope 2c. That is, the cross-sectional image acquisition unit 10c acquires a cross-sectional image of the uterine epithelial tissue of the subject.

[0208] Step S820: The fiber pixel extraction unit 11c extracts fiber pixels, which are pixels in which a fiber-like structure is captured, from the pixels in each SHG image PSic included in the Z-stack SHG image ZS1c. The fiber pixel extraction unit 11c generates a Z-stack fiber-like structure image ZF1c from the Z-stack SHG image ZS1c based on the extraction results. The fiber pixel extraction unit 11c supplies the generated Z-stack fiber-like structure image ZF1c to the fiber augmentation amount calculation unit 12c.

[0209] Here, the fiber pixel extraction unit 11c determines whether or not a fiber-like structure is imaged at each pixel based on predetermined criteria, and extracts fiber pixels based on the determination result. For example, the fiber pixel extraction unit 11c performs the determination based on machine learning. That is, the fiber pixel extraction unit 11c uses criteria generated by machine learning as predetermined criteria for determining fiber-like structures.

[0210] The fiber pixel extraction unit 11c uses, for example, Conditional GAN (GAN: Generative Adversarial Networks) as machine learning. Conditional GAN is composed of two networks: a network called a generation unit and a network called a classification unit. The generation unit learns to generate fake images that cannot be detected by the classification unit. On the other hand, the classification unit learns a classifier that can detect fake images generated by the generation unit. In this embodiment, the criterion generated by machine learning is that the identification unit of the Conditional GAN determines that a fibrous structure is imaged in the pixel. Note that the machine learning used by the fiber pixel extraction unit 11c may be other than Conditional GAN.

[0211] Here, a fiber-like structure image PF1c included in the Z-stack fiber-like structure image ZF1c generated by the fiber pixel extraction unit 11c will be described with reference to Fig. 37. Fig. 37 is a diagram showing an example of the fiber-like structure image PF1c according to this embodiment. The fibrous structure image PF1c is an image in which the fibrous structure is determined based on the SHG image PS1c included in the Z-stack SHG image ZS1c. As an example, the SHG image PS1c is an SHG image of tissue at a depth of 15 μm from the epithelial tissue of the uterus tissue.

[0212] As described above, in this embodiment, the fiber pixel extraction unit 11c extracts fiber pixels based on the Z-stack captured image ZP1c acquired by the cross-sectional image acquisition unit 10c and criteria generated by machine learning.

[0213] In the present embodiment, an example has been described in which the fiber pixel extraction unit 11c uses machine learning to determine whether a pixel contains a fiber-like structure in the Z-stack SHG image ZS1c, but this is not limiting. The fiber pixel extraction unit 11c may determine whether a pixel contains a fiber-like structure based on a predetermined criterion, for example, whether the pixel is included in a region of continuous pixels that has a shape equal to or smaller than a predetermined width and equal to or greater than a predetermined length. Furthermore, the Z-stack SHG image ZS1c may be configured without including any images classified as non-cancerous.

[0214] In other words, the fiber pixel extraction unit 11c extracts fiber pixels, which are pixels in which fiber-like structures are imaged, from the pixels of the Z stack image ZP1c acquired by the cross-sectional image acquisition unit 10c based on the Z stack image ZP1c acquired by the cross-sectional image acquisition unit 10c and predetermined criteria.

[0215] Furthermore, although the present embodiment has described an example in which fiber pixels are determined by the fiber pixel extraction unit 11c, this is not limiting. The fiber pixel determination may be performed by a user of the cancer stage assessment device 1c. When the fiber pixel determination is performed by a user of the cancer stage assessment device 1c, the cancer stage assessment device 1c includes an operation input unit, and the fiber pixel extraction unit 11c receives, for example, an operation for determining a fiber-like structure in the Z-stack SHG image ZS1c via the operation input unit. The fiber pixel extraction unit 11c generates a Z-stack fiber-like structure image ZF1c based on the received operation for determining a fiber-like structure.

[0216] Returning to FIG. 36, the cancer progression determination process will be continued. Step S830: The fibrosis amount calculation unit 12c calculates the amount of fibrosis in the shallow part of the uterine tissue captured as the Z-stack SHG image ZS1c based on the Z-stack fibrous structure image ZF1c generated by the fiber pixel extraction unit 11c. Here, the Z-stack fibrous structure image ZF1c generated by the fiber pixel extraction unit 11c is a Z-stack image in which fibrous structures are shown in the Z-stack SHG image ZS1c acquired by the cross-sectional image acquisition unit 10c. In other words, the fibrosis amount calculation unit 12c calculates the amount of fibrous structures captured in the Z-stack SHG image ZS1c acquired by the cross-sectional image acquisition unit 10c. The fibrosis amount calculation unit 12c supplies the calculated amount of fibrosis to the cancer progression assessment unit 13c.

[0217] Here, the fibrous volume calculation unit 12c calculates the amount of fibrous structure (fibrous volume) captured in the Z-stack SHG image ZS1c based on the fiber pixels extracted by the fiber pixel extraction unit 11c. The fibrous volume calculation unit 12c calculates the ratio of the area of the portion where the fibrous structure is captured to the entire area of the Z-stack fibrous structure image ZFic as the amount of fibrous volume. The fibrous volume calculation unit 12c calculates these areas based on the number of pixels. In other words, the fibrous volume calculation unit 12c calculates the amount of fibrous volume as the ratio of the number of pixels of the portion where the fibrous structure is captured to the entire number of pixels of the Z-stack fibrous structure image ZFic. Hereinafter, this ratio will be referred to as the detected pixel ratio.

[0218] In this embodiment, an example has been described in which the fibrous proliferation amount calculation unit 12c calculates the amount of fibrous proliferation as the ratio of the area of the portion in which the fibrous structure is imaged to the total area of the Z-stack fibrous structure image ZFic, but this is not limited to this. The fibrosis amount calculation unit 12c may calculate the area of the fibrous structure captured in the Z-stack fibrous structure image ZFic as the amount of fibrosis, or may calculate the length or number of fibrous structures as the amount of fibrosis. Furthermore, the fibrosis amount calculation unit 12c may perform predetermined preprocessing on the Z-stack fibrous structure image ZFic in order to calculate the amount of fibrosis. One example of the predetermined preprocessing is binarization. This allows the amount of fibrous structure captured in the Z-stack SHG image ZS1c to be calculated more reliably. Furthermore, the preprocessing is not limited to binarization, and smoothing and morphology processing may also be used.

[0219] Furthermore, the fibrosis amount calculation unit 12c may calculate, as the amount of fibrosis, the total amount or average amount of fibrous structures captured in the multiple Z-stack fibrous structure images ZFic included in the Z-stack SHG image ZS1c, or the ratio of the number of pixels of the fibrous structures to the total number of pixels of the cross-sectional image. When the total amount or average amount of fibrous structures captured in the multiple Z-stack fibrous structure images ZFic, or the ratio of the number of pixels of the fibrous structures to the total number of pixels of the cross-sectional image, is calculated, it is possible to prevent a decrease in the accuracy of determining the stage of uterine cancer when there is variation in the amount of fibrosis among the SHG images ZSic included in the Z-stack SHG image ZS1c.

[0220] The detected pixel ratio calculated by the fibrosis amount calculation unit 12c will now be described with reference to Fig. 38 and Fig. 39. Fig. 38 is a diagram showing an example of the detected pixel ratio according to this embodiment. In Fig. 38, the detected pixel ratio for each of 16 specimens of invasive cancerous uterine tissue, 3 specimens of precancerous uterine tissue, and 13 specimens of normal uterine tissue is shown relative to the Z axis of the Z stack image. The larger the Z-axis coordinate value, the deeper the imaged uterine tissue is from the superficial epithelial tissue. The Z-coordinate value of the surface of the epithelial tissue is 0. The Z-coordinate value of the boundary between the superficial and deep uterine tissue is 50.

[0221] FIG. 39 is a diagram showing an example of the average value of the detected pixel ratio for the shallow part of uterine tissue according to this embodiment. The average value for the shallow part of uterine tissue is the average for the Z-axis value of the Z-stack image. The three graphs shown in FIG. 39 show the average value of the detected pixel ratio for the shallow part of uterine tissue against the number indicating each specimen. Graph G1c shows the average value of the detected pixel ratio for invasive cancerous uterine tissue. Graph G2c shows the average value of the detected pixel ratio for precancerous uterine tissue. Graph G3c shows the average value of the detected pixel ratio for normal uterine tissue.

[0222] Graphs G1c, G2c, and G3c show that the detected pixel ratio is higher in cancerous tissue than in normal tissue. Graphs G1c and G2c also show that the detected pixel ratio is higher in invasive cancer than in precancer. The detected pixel ratio increases as the stage of cancer progresses. Here, the detected pixel ratio corresponds to the amount of fibrosis in the superficial part of the uterine tissue. Therefore, the amount of fibrosis in the superficial part of the uterine tissue increases as the stage of cancer progresses.

[0223] Returning to FIG. 36, the cancer progression determination process will be continued. Step S840: The cancer progression determination unit 13c determines the progression of uterine cancer in the subject based on the detected pixel ratio calculated by the fibrosis amount calculation unit 12c and the first threshold TH1c and second threshold TH2c indicated by the threshold information THc. Here, the cancer progression determination unit 13c reads out the threshold information THc from the storage unit 15c. The cancer progression determination unit 13c supplies the determination result as a determination result Ac to the output unit 14c.

[0224] Here, the cancer progression determination unit 13c determines that the uterine epithelial tissue is non-cancerous tissue when the amount of fibrous structures is equal to or less than the first threshold TH1c, whereas the cancer progression determination unit 13c determines that the subject has or is likely to have uterine cancer when the amount of fibrous structures exceeds the first threshold TH1c. If the amount of fibrous structures exceeds the first threshold TH1c and is equal to or smaller than the second threshold TH2c that is greater than the first threshold TH1c, the cancer progression determination unit 13c determines that the subject's uterine cancer is in a precancerous state.If the amount of fibrous structures exceeds the second threshold TH2c, the cancer progression determination unit 13c determines that the subject's uterine cancer is invasive cancer.

[0225] Step S850: The output unit 14c outputs the determination result Ac to the display device 3c. With the above, the cancer stage determination device 1c ends the cancer stage determination process.

[0226] Here, the determination rate of the cancer stage determination unit 13c will be described using an ROC (Receiver Operating Characteristic) curve with reference to Fig. 40 and Fig. 41. Fig. 40 is a diagram showing an example of an ROC curve indicating the determination rate of the cancer stage determination unit 13c according to this embodiment. Fig. 41 is a diagram showing an example of the relationship between the determination threshold and the determination rate of the cancer stage determination unit 13c according to this embodiment.

[0227] The ROC curve in Figure 40 shows the rate of detecting invasive cancer as sensitivity against the rate of detecting precancer as specificity. In Figure 40, the evaluation is performed by corresponding invasive cancer to positive and precancer to negative. In other words, the rate of detecting precancer corresponds to the true negative rate, and the rate of detecting invasive cancer corresponds to the true positive rate. In the ROC curve in Figure 40, the AUC (Area under an ROC curve) value is 0.88.

[0228] As shown in Fig. 41, when the judgment threshold used by the cancer progression determination unit 13c was changed, a determination rate of 81.3% sensitivity and 80.0% specificity was obtained when the judgment threshold was 0.0025. The AUC value of the ROC curve in Fig. 40 is the value when the judgment threshold is 0.0025.

[0229] In this embodiment, an example has been described in which the cancer progression determination unit 13c determines whether the stage of cancer is precancer or invasive cancer, but the present invention is not limited to this. The cancer progression determination unit 13c may determine whether the stage of cancer is any of the following: (i) Precancerous; (b) Invasive cancer; (c) mild dysplasia (CIN1), moderate dysplasia (CIN2), or severe dysplasia / carcinoma in situ (CIN3); (ii) Microinvasive squamous cell carcinoma or squamous cell carcinoma; or (e) Cancer requiring treatment or cancer not requiring treatment.

[0230] As described above, the cancer progression assessment device 1c according to this embodiment includes a cross-sectional image acquisition unit 10c, a calculation unit (in this example, a fibrosis amount calculation unit 12c), and a cancer progression assessment unit 13c. The cross-sectional image acquisition unit 10c acquires a cross-sectional image (in this example, an SHG image PSic) of the uterine epithelial tissue of the subject. The calculation unit (in this example, the fibrosis amount calculation unit 12c) calculates the amount of fibrous structures captured in the cross-sectional image (in this example, the SHG image PSic) acquired by the cross-sectional image acquisition unit 10c. The cancer progression determination unit 13c determines the progression of the subject's uterine cancer (in this example, non-cancerous, pre-cancerous, or invasive cancer) based on the amount of fibrous structure calculated by the calculation unit (in this example, the fibrosis amount calculation unit 12c).

[0231] With this configuration, the cancer progression assessment device 1c of this embodiment can assess the progression of a subject's uterine cancer (in this example, non-cancerous, pre-cancerous, or invasive cancer) based on the amount of fibrous structures captured in a cross-sectional image of the subject's uterine epithelial tissue (in this example, SHG image PSic), and therefore can assess the progression of uterine cancer (in this example, non-cancerous, pre-cancerous, or invasive cancer) without staining the uterine tissue.

[0232] The cancer progression stage assessment device 1c according to this embodiment also includes an extraction unit (in this example, a fiber pixel extraction unit 11c). The extraction unit (in this example, the fiber pixel extraction unit 11c) extracts fiber pixels, which are pixels in which fiber-like structures are imaged, from the pixels of the cross-sectional image (in this example, the SHG image PSic) acquired by the cross-sectional image acquisition unit 10c, based on a predetermined criterion (in this example, a criterion generated by machine learning). In addition, the calculation unit (in this example, the fiber proliferation amount calculation unit 12c) calculates the amount of fiber-like structure captured in the cross-sectional image (in this example, the SHG image PSic) based on the fiber pixels extracted by the extraction unit (in this example, the fiber pixel extraction unit 11c).

[0233] With this configuration, the cancer progression assessment device 1c of this embodiment can extract fiber pixels, which are pixels in which fiber-like structures are captured, from the pixels of a cross-sectional image (in this example, SHG image PSic), thereby reducing the effort required to extract fiber pixels from the pixels of a cross-sectional image (in this example, SHG image PSic).

[0234] Furthermore, in the cancer progression assessment device 1c according to this embodiment, the calculation unit (in this example, the fibrosis amount calculation unit 12c) calculates the area of the fibrous structure captured in the cross-sectional image (in this example, the SHG image PSic) as the amount of the fibrous structure. With this configuration, the cancer progression assessment device 1c of this embodiment can calculate the area of the fibrous structure captured in the cross-sectional image (in this example, SHG image PSic) as the amount of fibrous structure, and therefore can assess the progression of uterine cancer (in this example, non-cancerous, pre-cancerous, or invasive cancer) based on the area of the fibrous structure captured in the cross-sectional image (in this example, SHG image PSic) without staining the uterine tissue.

[0235] In addition, in the cancer progression assessment device 1c according to this embodiment, the cross-sectional image (in this example, the SHG image PSic) is a second harmonic generation image obtained by the multiphoton microscope 2c. With this configuration, the cancer progression assessment device 1c according to this embodiment can use a cross-sectional image (in this example, an SHG image PSic) of a fibrous structure that occurs as a tumor progresses, which is imaged using second harmonic generation, and therefore can assess the progression of uterine cancer (in this example, non-cancerous, pre-cancerous, or invasive cancer) using the fibrous structure imaged based on second harmonic generation without staining the uterine tissue. Assessment can be made without staining the uterine tissue.

[0236] Furthermore, in the cancer progression assessment device 1c according to this embodiment, the cross-sectional image (in this example, SHG image PSic) is a plurality of cross-sectional images (in this example, Z-stack SHG image ZS1c) taken at each of a plurality of depths of the uterine epithelial tissue of the subject, and the calculation unit (in this example, fibrosis amount calculation unit 12c) calculates the total amount, average amount, or ratio of the fibrous structures imaged in the plurality of cross-sectional images (in this example, Z-stack SHG image ZS1c) to the number of pixels of the cross-sectional image.

[0237] With this configuration, the cancer progression assessment device 1c of this embodiment can assess the progression of uterine cancer (in this example, non-cancer, pre-cancer, or invasive cancer) based on the total amount, average amount, or ratio of fibrous structures captured in multiple cross-sectional images (in this example, Z-stack SHG image ZS1c) to the number of pixels in the cross-sectional image, thereby preventing a decrease in the accuracy of assessing the progression of uterine cancer (in this example, non-cancer, pre-cancer, or invasive cancer) when there is variation in the amount of fibrous structures between multiple cross-sectional images (in this example, Z-stack SHG image ZS1c).

[0238] Furthermore, in the cancer progression determination device 1c according to this embodiment, the cancer progression determination unit 13c determines whether the progression of uterine cancer falls under any of the following categories. (i) Precancerous; (b) Invasive cancer; (c) mild dysplasia (CIN1), moderate dysplasia (CIN2), or severe dysplasia / carcinoma in situ (CIN3); (ii) Microinvasive squamous cell carcinoma or squamous cell carcinoma; or (e) Cancer requiring treatment or cancer not requiring treatment.

[0239] With this configuration, the cancer progression assessment device 1c of this embodiment can determine whether the above-mentioned items (i) to (e) apply to the progression of uterine cancer based on the amount of fibrous structures in the cross-sectional image (in this example, the SHG image PSic), and therefore can determine whether the above-mentioned items (i) to (e) apply to the progression of uterine cancer without staining the uterine tissue.

[0240] Furthermore, in the cancer progression assessment device 1c according to this embodiment, the cancer progression assessment unit 13c determines that the uterine epithelial tissue is non-cancerous tissue if the amount of fibrous structures is equal to or less than the first threshold value TH1c, and determines that the subject has or is likely to have uterine cancer if the amount of fibrous structures exceeds the first threshold value TH1c. With this configuration, the cancer progression determination device 1c of this embodiment can determine the progression of uterine cancer as non-cancerous tissue, uterine cancer, or high possibility of uterine cancer, based on the amount of fibrous structures in the cross-sectional image (in this example, SHG image PSic).

[0241] Furthermore, in the cancer progression assessment device 1c according to this embodiment, the cross-sectional image is an image of precancer or invasive cancer, and as the progression of uterine cancer, the cancer progression assessment unit 13c assesses that the subject has precancer if the amount of fibrous structures exceeds a first threshold TH1c and is equal to or less than a second threshold TH2c that is greater than the first threshold TH1c, and assesses that the subject has invasive cancer if the amount exceeds the second threshold TH2c. With this configuration, the cancer progression assessment device 1c of this embodiment can determine the progression of uterine cancer, whether the subject has precancer or invasive cancer, based on the amount of fibrous structures in the cross-sectional image (in this example, the SHG image PSic). Furthermore, in the cancer progression assessment device 1c according to this embodiment, the cross-sectional image is a cross-sectional image of precancer or invasive cancer, and the cancer progression assessment unit 13c determines that the progression of the subject's uterine cancer is precancer if the amount of fibrous structures is smaller than the first threshold value TH1c when the progression of the subject's uterine cancer is precancer, and determines that the progression of the subject's uterine cancer is invasive cancer if the amount of fibrous structures exceeds the first threshold value TH1c. With this configuration, in the cancer progression determination device of this embodiment, the cross-sectional image (Z-stack SHG image ZS1c) is a cross-sectional image of precancer or invasive cancer, and by determining from the image that the subject's uterine epithelial tissue is precancer or invasive cancer, precancer can be accurately determined.

[0242] Note that parts of the uterine cancer assessment device 1, uterine cancer assessment device 1a, uterine cancer assessment device 1b, and cancer stage assessment device 1c in the above-described embodiments, such as the nuclear region image generation unit 10, feature amount processing unit 11, data division unit 12, classification model generation unit 13, cancer tissue assessment unit 14, output unit 15, operation input unit 16, nuclear region image generation unit 10a, SHG image acquisition unit 18b, cancer stage assessment unit 19b, cross-sectional image acquisition unit 10c, fiber pixel extraction unit 11c, fibrosis amount calculation unit 12c, cancer stage assessment unit 13c, and output unit 14c, may be implemented by a computer. In this case, a program for implementing these control functions may be recorded on a computer-readable recording medium, and the program may be read and executed by a computer system. Note that the "computer system" referred to here refers to a computer system built into the uterine cancer assessment device 1, uterine cancer assessment device 1a, uterine cancer assessment device 1b, or cancer stage assessment device 1c, and includes hardware such as an OS and peripheral devices. Furthermore, "computer-readable recording media" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, as well as storage devices such as hard disks built into computer systems. Furthermore, "computer-readable recording media" may also include devices that dynamically store programs for a short period of time, such as communication lines used when transmitting programs over networks like the Internet or communication lines like telephone lines, or devices that store programs for a fixed period of time, such as volatile memory within computer systems that serve as servers or clients in such cases. Furthermore, the above-mentioned programs may be programs that realize some of the aforementioned functions, or may be programs that can realize the aforementioned functions in combination with programs already stored in the computer system. Furthermore, the uterine cancer determination device 1, the uterine cancer determination device 1a, the uterine cancer determination device 1b, and the cancer stage determination device 1c in the above-described embodiments may be partly or entirely realized as an integrated circuit such as an LSI (Large Scale Integration). Each functional block of the uterine cancer determination device 1, the uterine cancer determination device 1a, the uterine cancer determination device 1b, and the cancer stage determination device 1c may be individually implemented as a processor, or partly or entirely integrated into a processor. Furthermore, the integrated circuit implementation method is not limited to LSI, and may be implemented using a dedicated circuit or a general-purpose processor. Furthermore, if an integrated circuit implementation technology that can replace LSI emerges due to advances in semiconductor technology, an integrated circuit based on this technology may be used.

[0243] One embodiment of the present invention has been described in detail above with reference to the drawings, but the specific configuration is not limited to that described above, and various design changes and the like are possible within the scope that does not deviate from the gist of the present invention. [Explanation of symbols]

[0244] 1, 1a, 1b...uterine cancer diagnosis device, 100...THG image acquisition unit, 14...cancer tissue diagnosis unit, ZT1...Z-stack THG image, 1c...cancer progression diagnosis device, 10c...cross-sectional image acquisition unit, 11c...fiber pixel extraction unit, 12c...fibrous proliferation amount calculation unit, 13c...cancer progression diagnosis unit, PSic...SHG image, ZS1c...Z-stack SHG image

Claims

[Claim 1] an irradiation unit that irradiates the tissue with excitation light; a third harmonic image acquisition unit that acquires a third harmonic image of the tissue based on light generated by third harmonic generation caused by an interaction between the tissue and the excitation light; a cancer tissue determination unit that determines the possibility that the tissue is cancer tissue based on the state of the cell nuclei in the third harmonic image; A cancer diagnosis device comprising:

Images