IMAGE PROCESSING APPARATUS, OPERATION METHOD OF IMAGE PROCESSING APPARATUS, AND PROGRAM

Abstract

This cancer assessment device comprises: an irradiation unit that irradiates tissue with excitation light; a third high-frequency image acquisition unit that acquires a third high-frequency image of the tissue on the basis of light generated through third high-frequency generation occurring due to interaction between the tissue and the excitation light; and a cancer tissue assessment unit that assesses the possibility that the tissue is cancer tissue on the basis of the state of cell nuclei in the third high-frequency image.

Description

[Technical field]

[0001] The present invention relates to an image processing device, Method for operating an image processing device , and programs. This application claims priority based on Japanese Patent Application Nos. 2019-192469 and 2019-192470, filed on October 23, 2019, the contents of which are incorporated herein by reference. [Background technology]

[0002] The morphology of cell nuclei carries important information in pathological diagnosis, and staining of cell nuclei is an essential procedure for pathological tissue specimens. On the other hand, a technique for determining cancer tissue by analyzing images of biological tissue is known. When determining cancer tissue by image analysis, images of biological tissue that have been stained and captured are used.

[0003] For example, a cancer screening device is known that captures an image of a group of biological cells coated with a dye that selectively stains cancer-related gene products in the biological cells in chromatic colors, and includes a judgment unit that judges the malignancy level of the cancer of the group of biological cells based on the stained state of the group of biological cells in the obtained image (Patent Document 1). Previously, it was difficult to analyze the state of unstained tissue at the cellular level. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2017 / 200066 Summary of the Invention

[0005] One aspect of the present invention is a non-stained uterus An irradiation unit that irradiates the tissue with excitation light; uterus The third harmonic generation is generated by the interaction of the tissue with the excitation light. uterus a third harmonic image acquisition unit for acquiring a third harmonic image of tissue; a cancer tissue determination unit that determines the possibility that the uterine tissue is cancerous tissue based on the state of the cell nuclei in the third harmonic image; a second harmonic image acquisition unit that acquires a second harmonic image based on light generated by second harmonic generation caused by an interaction between the uterine tissue and the excitation light; and a cancer progression determination unit that determines the progression of cancer based on the state of a fibrous structure in the second harmonic image when the cancer tissue determination unit determines that the uterine tissue is highly likely to be cancerous tissue. Equipped with the second harmonic image and the third harmonic image are a plurality of cross-sectional images captured at different depths in a direction from the epithelium of the uterine tissue toward the inside, the depth being common to both the second harmonic image and the third harmonic image; the cancer tissue determination unit performs a first determination to determine whether the uterine tissue is cancer tissue or normal tissue based on the feature amount of the cell nucleus corresponding to the depth from the third harmonic image; and the cancer progression determination unit performs a second determination to determine whether the uterine tissue is precancer or invasive cancer based on the amount of the fibrous structure corresponding to the depth from the second harmonic image for the uterine tissue determined to be cancer tissue in the first determination. An image processing device.

[0006] One aspect of the present invention is A method for operating an image processing device including an irradiation unit, a third harmonic image acquisition unit, a cancer tissue assessment unit, a second harmonic image acquisition unit, and a cancer progression assessment unit, the method comprising the steps of: an irradiation step of irradiating an excitation light onto unstained uterine tissue; The third harmonic image acquisition unit, a third harmonic image acquisition process for acquiring a third harmonic image of the uterine tissue based on light generated by third harmonic generation caused by an interaction between the uterine tissue and the excitation light; The cancer tissue determination section a cancer tissue determination process for determining the possibility that the uterine tissue is cancer tissue based on the state of the cell nuclei in the third harmonic image; The second harmonic image acquisition unit, a second harmonic image acquisition step of acquiring a second harmonic image based on light generated by second harmonic generation caused by an interaction between the uterine tissue and the excitation light; The cancer progression assessment unit, a cancer progression determination process for determining a stage of cancer based on a state of a fibrous structure in the second harmonic image when the cancer tissue determination process determines that the uterine tissue is highly likely to be cancer tissue, wherein the second harmonic image and the third harmonic image are a plurality of cross-sectional images captured at different depths in a direction from the epithelium of the uterine tissue toward the inside, the depths being common to both the second harmonic image and the third harmonic image, and the cancer tissue determination process In , The cancer tissue determination section a first determination as to whether the uterine tissue is cancer tissue or normal tissue based on the feature amount of the cell nucleus corresponding to the depth from the third harmonic image; In , The cancer progression assessment unit, performing a second determination on the uterine tissue determined to be cancerous tissue in the first determination, in which the uterine tissue is determined to be pre-cancer or invasive cancer based on the amount of the fibrous structure corresponding to the depth from the second harmonic image; Method for operating an image processing device It is.

[0007] One aspect of the present invention is to provide a computer uterus An irradiation step of irradiating a tissue with excitation light; uterus The third harmonic generation is generated by the interaction of the tissue with the excitation light. uterusA third harmonic image acquisition step of acquiring a third harmonic image of tissue; a cancer tissue determination step of determining the possibility that the uterine tissue is cancerous tissue based on the state of the cell nuclei in the third harmonic image; a second harmonic image acquisition step of acquiring a second harmonic image based on light generated by second harmonic generation caused by interaction between the uterine tissue and the excitation light; and a cancer progression determination step of determining the progression of cancer based on the state of a fibrous structure in the second harmonic image when the cancer tissue determination step determines that the uterine tissue is likely to be cancerous tissue. A program that executes wherein the second harmonic image and the third harmonic image are a plurality of cross-sectional images captured at different depths in a direction from an epithelial tissue of the uterine tissue toward the inside, the depths being common to the second harmonic image and the third harmonic image, the cancer tissue determination step performs a first determination to determine whether the uterine tissue is cancer tissue or normal tissue based on a feature amount of the cell nucleus corresponding to the depth from the third harmonic image, and the cancer progression determination step performs a second determination to determine whether the uterine tissue is precancer or invasive cancer based on an amount of the fibrous structure corresponding to the depth from the second harmonic image for the uterine tissue determined to be cancer tissue in the first determination. It is. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram showing an example of a uterine cancer determination system according to a first embodiment. [Diagram 2] FIG. 2 is a diagram showing an example of a captured image of mouse skin tissue for illustrating visualization of cell nuclei using third harmonic generation according to the first embodiment. [Diagram 3] FIG. 2 is a diagram showing an example of a captured image of human uterine tissue according to the first embodiment. [Figure 4] 1 is a diagram showing an example of the configuration of a uterine cancer diagnosis device according to a first embodiment. [Diagram 5] FIG. 2 is a diagram showing an example of a nuclear region image according to the first embodiment. [Figure 6] FIG. 4 is a diagram illustrating an example of a feature amount according to the first embodiment. [Figure 7] FIG. 4 is a diagram showing an example of uterine cancer determination processing according to the first embodiment. [Figure 8] FIG. 4 is a diagram showing an example of a core region image generating process according to the first embodiment. [Figure 9] FIG. 2 is a diagram showing an example of a Z-stack nuclear region image according to the first embodiment. [Figure 10] FIG. 4 is a diagram illustrating an example of a learning process according to the first embodiment. [Figure 11] FIG. 2 is a diagram showing an example of measurement of a cell nucleus according to the first embodiment. [Figure 12] FIG. 4 is a diagram illustrating an example of a determination process according to the first embodiment. [Figure 13] FIG. 4 is a diagram showing an example of a determination result according to the first embodiment. [Figure 14] FIG. 11 is a diagram showing an example of a uterine cancer diagnosing device according to the second embodiment. [Figure 15] FIG. 11 is a diagram showing an example of deep learning used by a nucleus region image generating unit according to the second embodiment. [Figure 16] FIG. 11 is a diagram illustrating an example of core region learning processing according to the second embodiment. [Figure 17] 13A to 13C are diagrams illustrating an example of a core region determination process according to the second embodiment. [Figure 18] 13 is a diagram showing an example of a nucleus region image and a nucleus region annotation image according to the second embodiment. FIG. [Figure 19] 13A to 13C are diagrams illustrating an example of a result of a core region determination process according to the second embodiment. [Figure 20] FIG. 13 is a diagram showing an example of an F value for the entire Z-stack nuclear region image according to the second embodiment. [Figure 21] FIG. 11 is a diagram showing an example of a determination result according to the second embodiment. [Figure 22] FIG. 13 is a diagram showing an example of a second harmonic image according to the third embodiment. [Figure 23A] FIG. 13 is a diagram showing an example of cross sections of uterine tissue for each stage of tumor progression according to the third embodiment. [Figure 23B] FIG. 13 is a diagram showing an example of cross sections of uterine tissue for each stage of tumor progression according to the third embodiment. [Figure 24] FIG. 11 is a diagram showing an example of a uterine cancer diagnosing device according to the third embodiment. [Diagram 25] FIG. 13 is a diagram showing an example of uterine cancer determination processing according to the third embodiment. [Figure 26] FIG. 13 is a diagram showing an example of a precancer / invasion cancer determination process according to the third embodiment. [Figure 27] FIG. 13 is a diagram showing an example of a fiber-like structure image according to the third embodiment. [Figure 28] FIG. 13 is a diagram illustrating an example of a detection pixel ratio according to the third embodiment. [Figure 29] FIG. 13 is a diagram showing an example of an average value of detection pixel ratios for a shallow part of uterine tissue according to the third embodiment. [Diagram 30] FIG. 13 is a diagram showing an example of an ROC curve showing the diagnosis rate of the cancer progression determination section according to the third embodiment. [Diagram 31] FIG. 13 is a diagram showing an example of the relationship between the determination threshold and the determination rate of the cancer progression determination unit according to the third embodiment. [Diagram 32] FIG. 13 is a diagram showing an example of a uterine cancer progression determination system according to a fourth embodiment. [Diagram 33] FIG. 13 is a diagram showing an example of a second harmonic generation image according to the fourth embodiment. [Figure 34A] FIG. 13 is a diagram showing an example of cross sections of uterine tissue for each stage of tumor progression according to the fourth embodiment. [Figure 34B] FIG. 13 is a diagram showing an example of cross sections of uterine tissue for each stage of tumor progression according to the fourth embodiment. [Diagram 35] FIG. 13 is a diagram showing an example of a cancer progression assessment device according to a fourth embodiment. [Diagram 36] FIG. 13 is a diagram showing an example of cancer progression determination processing according to the fourth embodiment. [Figure 37] FIG. 13 is a diagram showing an example of a fiber-like structure image according to the fourth embodiment. [Figure 38] FIG. 13 is a diagram showing an example of a detection pixel ratio according to the fourth embodiment. [Figure 39] FIG. 13 is a diagram showing an example of an average value of detection pixel ratios for a shallow part of uterine tissue according to the fourth embodiment. [Diagram 40] FIG. 13 is a diagram showing an example of an ROC curve showing the diagnosis rate of the cancer progression determination section according to the fourth embodiment. [Diagram 41] FIG. 13 is a diagram showing an example of the relationship between the determination threshold and the determination rate of the cancer progression determination unit according to the fourth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] (First embodiment) Hereinafter, the first embodiment will be described in detail with reference to the drawings. FIG. 1 is a diagram showing an example of a uterine cancer determination system ST according to this embodiment. The uterine cancer determination system ST includes a uterine cancer determination device 1, a multiphoton microscope 2, and a display device 3. In the uterine cancer determination system ST, the uterine cancer determination device 1 analyzes a Z-stack captured image ZP1, which is a uterine tissue of a subject captured by the multiphoton microscope 2, and determines the possibility that the uterine tissue captured in the Z-stack captured image ZP1 is a cancerous tissue. The uterine cancer determination device 1 causes the display device 3 to display the determination result.

[0010] In the present embodiment, the uterine tissue is, for example, tissue of the cervix. Note that the uterine tissue may include the uterine body. In addition, although the present embodiment describes the uterine tissue as an example, the tissue is not limited thereto.

[0011] The Z-stack captured image ZP1 is a Z-stack image. A Z-stack image is a set of multiple images captured at different distances from the uterine tissue in the Z-axis direction. The Z-stack captured image ZP1 is made up of multiple images of the uterine tissue at each Z coordinate. Capturing an image of the uterine tissue at each Z coordinate means capturing an image of the uterine tissue at various distances between the objective lens and the uterine tissue. The Z-stack captured image ZP1 is a set of multiple images captured by the multiphoton microscope 2 at different distances between the uterine tissue and the lens.

[0012] In this embodiment, the Z axis is selected in the direction from the epithelium of the uterine tissue toward the inside. In other words, the closer the imaging plane is to the uterine epithelium, the smaller the Z coordinate value is, and the deeper the imaging plane is into the uterine tissue, the larger the Z coordinate value is. The origin of the Z axis is selected at a shallow position close to the uterine epithelium. In this way, the Z-stack captured image ZP1 is a plurality of cross-sectional images perpendicular to the depth direction of the uterine epithelial tissue of the subject, and is also simply called a cross-sectional image. Note that the depth direction is the direction from the surface layer to the basal layer, and the plurality of cross-sectional images do not need to be completely perpendicular to the depth direction of the uterine epithelial tissue of the subject, and may be inclined by about ±5 degrees.

[0013] The multiphoton microscope 2 observes and images the subject's uterine tissue in an unstained state. The multiphoton microscope 2 images the uterine tissue using nonlinear optical phenomena. The nonlinear optical phenomena that the multiphoton microscope 2 uses for imaging include second harmonic generation (SHG) and third harmonic generation (THG).

[0014] SHG is a phenomenon in which light with twice the frequency of the excitation light is generated by interaction with nonlinear optical crystals such as collagen fibers. THG is a phenomenon in which light with a frequency three times that of the excitation light is generated. THG occurs due to interactions with interfaces and layered structures.

[0015] In this embodiment, the multiphoton microscope 2 uses THG to image the uterine tissue to generate a third harmonic image. The third harmonic image is an image of a cross section of the uterine tissue generated based on the excitation light irradiated from the irradiation unit of the multiphoton microscope 2 and the light generated by third harmonic generation due to the interaction between the excitation light and the uterine tissue. The captured image included in the Z-stack captured image ZP1 is a third harmonic image, which is an image of the uterine tissue captured using THG. Hereinafter, the third harmonic image will be referred to as a THG image.

[0016] Hereinafter, the Z-stack captured image ZP1 will be referred to as the Z-stack THG image ZT1. The Z-stack THG image ZT1 is a plurality of THG images of the uterine tissue captured using THG at different distances from the uterine tissue in the Z-axis direction. In other words, the third harmonic image of the subject's uterine tissue is a plurality of cross-sectional images of the subject's uterine epithelial tissue. Furthermore, each of the multiple images included in the Z-stack THG image ZT1 is referred to as a THG image PTi (i=1, 2, . . . , N: N is the number of images included in the Z-stack THG image ZT1), etc.

[0017] The multiphoton microscope 2 uses THG to visualize cell nuclei in the subject's uterine tissue, which has been previously exposed to acetic acid. Here, the fact that cell nuclei can be visualized by using THG will be described with reference to Figures 2 and 3. Figure 2 is a diagram showing an example of a captured image of mouse skin tissue for explaining the visualization of cell nuclei by using THG.

[0018] THG image PA10, THG image PA11, THG image PA12, and THG image PA13 are images of mouse skin tissue captured using THG. Fluorescent image PA20, fluorescent image PA21, fluorescent image PA22, and fluorescent image PA23 are images of cell nuclei of mouse skin tissue stained with a fluorescent dye and captured using fluorescence. The fluorescent dye is Hoechst33342. Composite image PA30, composite image PA31, composite image PA32, and composite image PA33 are images composited of an image captured using THG and an image stained with Hoechst33342 and captured using fluorescence.

[0019] The THG image PA10 and the fluorescent image PA20 are images of mouse skin tissue captured without the addition of fluorescent dye or acetic acid. The THG image PA10 and the fluorescent image PA20 capture a common portion of the skin tissue. The composite image PA30 is an image obtained by combining the THG image PA10 and the fluorescent image PA20.

[0020] The THG image PA11 and the fluorescent image PA21 are images of mouse skin tissue with a fluorescent dye added. The THG image PA11 and the fluorescent image PA21 are images of a common part of the skin tissue. The composite image PA31 is an image obtained by combining the THG image PA11 and the fluorescent image PA21.

[0021] The THG image PA12 and the fluorescent image PA22 are images of mouse skin tissues captured with a fluorescent dye and acetic acid added. The THG image PA12 and the fluorescent image PA22 capture a common portion of the skin tissue. The composite image PA32 is an image obtained by combining the THG image PA12 and the fluorescent image PA22.

[0022] The THG image PA13 and the fluorescent image PA23 are images of mouse skin tissue captured with acetic acid added. The THG image PA13 and the fluorescent image PA23 capture a common portion of the skin tissue. The composite image PA33 is an image obtained by combining the THG image PA13 and the fluorescent image PA23.

[0023] In the fluorescent image PA22, the cell nucleus is captured. In the composite image PA32, it can be seen that the image of the cell nucleus shown in the THG image PA12 matches the image of the stained cell nucleus shown in the fluorescent image PA22. It can be seen that the image of the cell nucleus captured using THG in a state where a fluorescent dye and acetic acid have been added captures the cell nucleus in the same way as the image of the cell nucleus captured using fluorescence in a state where acetic acid has been added.

[0024] Next, when THG image PA13 is compared with THG image PA12, it is found that an image of a cell nucleus is captured in THG image PA13, similar to THG image PA12. That is, it is found that an image captured using THG in a state where acetic acid is added, an image of a cell nucleus is captured similarly to an image captured using THG in a state where a fluorescent dye and acetic acid are added.

[0025] Therefore, in the THG image PA13, the cell nucleus is captured in the same manner as in the fluorescent image PA22. That is, in the image captured using THG in the presence of added acetic acid, the cell nucleus is captured in the same manner as in the image captured using fluorescence in the presence of added acetic acid. In this way, by using THG in the presence of acetic acid, it is possible to capture images of cell nuclei without staining the nuclei.

[0026] Fig. 3 is a diagram showing an example of a captured image of human uterine tissue according to the present embodiment. In the captured image shown in Fig. 3, cervical tissue of human uterine tissue is captured. THG image PB10 and THG image PB11 are images of normal human cervical tissue captured using THG. THG image PB20 and THG image PB21 are images of human cervical tissue, which is cancerous tissue, captured using THG. Stained image PB12 and stained image PB22 are images for comparison with the THG image. Stained image PB12 is an image of normal human cervical tissue stained with a staining agent and captured. Stained image PB22 is an image of cancerous human cervical tissue stained with a staining agent. The staining agent is hematoxylin.

[0027] In the THG image PB10 and the THG image PB20, the cervical tissue is imaged without the addition of acetic acid, whereas in the THG image PB11 and the THG image PB21, the cervical tissue is imaged with the addition of acetic acid. Comparing THG image PB11 and THG image PB21 with THG image PB10 and THG image PB20, it can be seen that the cell nuclei are more clearly depicted in the THG image captured when acetic acid is added than in the THG image captured when acetic acid is not added.

[0028] Comparing THG image PB11 and THG image PB21 with stained image PB12 and stained image PB22, it is found that THG image PB11 and THG image PB21 are able to capture cell nuclei, similar to stained image PB12 and stained image PB22.

[0029] In other words, in human cervical tissue, as in the mouse skin tissue shown in Figure 2, by using THG in the presence of acetic acid, it is possible to capture images of cell nuclei without staining the cell nuclei. Conventionally, in THG images captured by multiphoton microscopes, there are few tissues in which cell nuclei are captured. In contrast, in the THG images captured by multiphoton microscope 2, cell nuclei of uterine tissue are captured.

[0030] (Configuration of the uterine cancer diagnosis device) Next, the configuration of the uterine cancer diagnosis device 1 will be described with reference to Fig. 4. Fig. 4 is a diagram showing an example of the configuration of the uterine cancer diagnosis device 1 according to this embodiment. The uterine cancer diagnosis device 1 is, for example, a computer.

[0031] The uterine cancer diagnosis device 1 includes a nuclear region image generation unit 10, a feature amount processing unit 11, a data division unit 12, a classification model generation unit 13, a cancer tissue diagnosis unit 14, an output unit 15, and an operation input unit 16. The nuclear region image generation unit 10, the feature amount processing unit 11, the data division unit 12, the classification model generation unit 13, the cancer tissue diagnosis unit 14, and the output unit 15 are each a module realized by a CPU (Central Processing Unit) reading a program from a ROM (Read Only Memory) and executing processing.

[0032] The nuclear region image generating unit 10 generates a Z-stack nuclear region image ZN1 from the Z-stack THG image ZT1 captured by the multiphoton microscope 2. Here, the Z-stack nuclear region image ZN1 refers to a plurality of images in which a region showing a cell nucleus of the uterine tissue is determined for each of the THG images PTi (i=1, 2, . . ., N: N is the number of images included in the Z-stack THG image ZT1), which are a plurality of images included in the Z-stack THG image ZT1. Each of the multiple images included in the Z-stack nuclear region image ZN1 is called a nuclear region image PNi (i=1, 2, . . . , N: N is the number of images included in the Z-stack nuclear region image ZN1).

[0033] In the present embodiment, the nuclear region, which is a region indicating a cell nucleus, is determined, for example, by a user of the uterine cancer determination device 1. The user of the uterine cancer determination device 1 visually checks the Z-stack captured image ZP1 and determines the nuclear region based on knowledge and experience.

[0034] Here, an example of the nuclear region image PNi will be described with reference to Fig. 5. Fig. 5 is a diagram showing an example of the nuclear region image PNi according to this embodiment. In the nuclear region image PNn, a nuclear region is determined in the THG image PTi of normal uterine tissue, and the contour of the determined nuclear region is colored. In the nuclear region image PNc, a region showing a cell nucleus is determined in the THG image PTi of uterine tissue that is cancerous tissue, and the contour of the determined nuclear region is colored.

[0035] Returning to FIG. 4, the description of the configuration of the uterine cancer diagnosing device 1 will be continued. The nucleus region image generating unit 10 includes a THG image acquiring unit 100, a nucleus determination operation receiving unit 101, and an image processing unit . The THG image acquisition section 100 acquires a Z-stack captured image ZP 1 captured by the multiphoton microscope 2 .

[0036] The nucleus determination operation receiving unit 101 receives a nucleus determination operation from a user of the uterine cancer determination device 1 via the operation input unit 16. Here, the nucleus determination operation is an operation of determining an area showing a cell nucleus of a uterine tissue in each of the THG images PTi included in the Z-stack captured image ZP1. The image processing unit 102 generates a Z-stack nucleus region image ZN1 from the Z-stack captured images ZP1 based on the nucleus determination operation received by the nucleus determination operation receiving unit 101.

[0037] The feature amount processor 11 extracts a feature amount C1 from the Z-stack nucleus region image ZN1 and performs various processes on the extracted feature amount C1. The feature amount C1 is a feature amount indicating the state of the cell nucleus, and is extracted for each nucleus region image PNi. In this embodiment, the state of the cell nucleus is at least one selected from the group consisting of the area of ​​the cell nucleus, the density of the cell nucleus, and the shape of the cell nucleus.

[0038] The feature amount processing unit 11 includes a nucleus region image acquisition unit 110 , a nucleus measurement unit 111 , a feature amount extraction unit 112 , a Z position correction unit 113 , and a feature amount scaling unit 114 .

[0039] The nuclear region image acquisition unit 110 acquires the Z-stack nuclear region image ZN1 supplied from the nuclear region image generation unit 10. The nucleus measurement unit 111 performs various measurements for each cell nucleus based on the image of the cell nucleus captured in the Z-stack nuclear region image ZN1. Examples of the various measurements include area, circularity, and nearest neighbor distance.

[0040] The feature amount extraction unit 112 extracts a feature amount C1 for each nuclear region image PNi included in the Z-stack nuclear region image ZN1 based on the result of measurement by the nuclear measurement unit 111.

[0041] Here, the feature C1 will be described in detail with reference to Fig. 6. Fig. 6 is a diagram showing an example of the feature C1 according to this embodiment. The feature C1 includes, by way of example, the area of ​​the cell nucleus, the circularity of the cell nucleus, the nearest distance between the cell nuclei, and the number of cell nuclei. Of the feature C1, the area, the circularity, and the nearest distance are used by calculating the median and the median absolute error.

[0042] Multiphoton imaging of normal cervical tissue shows prominent cell nuclei according to the depth from the surface, which is consistent with pathological findings. Here, the depth from the surface is classified into upper, middle, and lower layers. In Figure 6, the qualitative characteristics of the feature quantities for each depth from the surface are classified into "large," "medium," and "small" values. In the uterine cancer determination device 1, a feature amount of a cell nucleus that is effective for determining the possibility that the uterine tissue is a cancerous tissue is selected in advance.

[0043] For example, the median area of ​​cell nuclei in the upper layer of normal tissue is "small," in the middle layer, and in the lower layer is "medium." In other words, the median area of ​​normal tissue is not "large" in any layer. On the other hand, the median area of ​​the first cancer tissue is "large," and since cancer tissue showing a "large" area has been confirmed, the "large" median area is at least a characteristic of the first cancer tissue, and is considered to be a characteristic that can distinguish cancer tissue from normal tissue. A "large" median area corresponds to the pathological feature of enlarged nuclei.

[0044] Similarly, a large median absolute error in nuclei area, a small median nuclei circularity, a large median absolute error in nuclei circularity, a small median nearest neighbor distance between nuclei, a large median absolute error in nearest neighbor distance between nuclei, and a large number of nuclei are each considered to be features that can distinguish cancer tissue from normal tissue.

[0045] A large median absolute error in the area of ​​cell nuclei corresponds to the pathological feature of heterogeneity in nuclear size. A small median circularity of cell nuclei corresponds to the pathological feature of atypical cell nuclei. A large median absolute error in the circularity of cell nuclei corresponds to the pathological feature of atypical cell nuclei. A small median nearest neighbor distance between cell nuclei corresponds to the pathological feature of high density of cell nuclei. A large median absolute error in the nearest neighbor distance between cell nuclei corresponds to the pathological feature of heterogeneity in the distribution of cell nuclei in a certain area. A large number of cell nuclei corresponds to the pathological feature of high density of cell nuclei.

[0046] In summary, the characteristics of cell nuclei of cancerous uterine tissue include the following characteristics (i) to (vi). (i) the average nuclear area is increased compared to normal uterine tissue; (ii) the variance in nuclear area is increased compared to normal uterine tissue; (iii) the density of nuclear cells is increased compared to normal uterine tissue; (iv) the variance in the density of nuclear cells is increased compared to normal uterine tissue; (v) the irregularity in the shape of nuclear cells is increased compared to normal uterine tissue; (vi) the variance in the shape of nuclear cells is increased compared to normal uterine tissue.

[0047] Returning to FIG. 4, the description of the configuration of the uterine cancer diagnosing device 1 will be continued. The Z position correction unit 113 performs correction in the Z-axis direction on the Z-stack nuclear region image ZN1. The feature amount scaling unit 114 performs scaling on the feature amount C1 extracted by the feature amount extraction unit 112 based on normalization or standardization.

[0048] The data division unit 12 divides the feature amount C1 extracted by the feature amount processing unit 11 into learning feature amount data CL1 used for learning and determination feature amount data CE1 used for determination. The classification model generation unit 13 generates the classification model M1 based on machine learning using the learning feature data CL1. Here, the machine learning used by the classification model generation unit 13 is, for example, a nonlinear support vector machine.

[0049] The cancer tissue determination unit 14 determines the possibility that the uterine tissue is cancer tissue, based on the determination feature amount data CE1 and the classification model M1 generated by the classification model generation unit 13. The output unit 15 outputs the determination result A of the cancer tissue determination unit 14 to the display device 3. The determination result A indicates whether the uterine tissue is cancer tissue or normal tissue. The determination result A according to the present embodiment is only an example, and the cancer tissue determination unit 14 may output the probability indicating whether the tissue is cancer tissue or normal tissue to the display device 3, or may output this to the display device 3 in parallel with the learning feature data CL1 used for learning.

[0050] The operation input unit 16 accepts various operations from a user of the uterine cancer diagnosis device 1. Examples of the operation input unit 16 include a touch panel, a mouse, and a keyboard. The display device 3 displays the determination result A obtained by the uterine cancer determination device 1. The display device 3 is, for example, a display.

[0051] (Processing of uterine cancer diagnosis system) Next, a uterine cancer diagnosis process, which is a process of the uterine cancer diagnosis system ST, will be described. Fig. 7 is a diagram showing an example of the uterine cancer diagnosis process according to this embodiment.

[0052] Step S10: The multiphoton microscope 2 captures a Z-stack image ZP1 of the uterine tissue of the subject. As described above, in this embodiment, the Z-stack image ZP1 is a Z-stack THG image ZT1. The multiphoton microscope 2 outputs the captured Z-stack image ZP1 to the uterine cancer assessment device 1.

[0053] Step S20: The uterine cancer determination device 1 performs a normal tissue / cancer tissue determination process, which is a process for determining whether the uterine tissue of the subject is normal tissue or cancer tissue, based on the Z-stack captured image ZP1 captured by the multiphoton microscope 2. The normal tissue / cancer tissue determination process will be described in detail later with reference to FIG.

[0054] Step S30: The output unit 15 of the uterine cancer determination device 1 performs processing based on the determination result A. If the uterine tissue of the subject is determined to be normal (step S30; YES), the output unit 15 outputs a result indicating that the uterine tissue of the subject is normal to the display device 3. After that, the display device 3 executes the processing of step S40. On the other hand, if it is determined that the uterine tissue of the subject is not normal tissue, that is, is cancerous tissue (step S30; NO), the output unit 15 outputs a result indicating that the uterine tissue of the subject is cancerous tissue to the display device 3. After that, the display device 3 executes the process of step S50.

[0055] Step S40: The display device 3 displays the result indicating that the uterine tissue of the subject is normal tissue. Step S50: The display device 3 displays the result indicating that the uterine tissue of the subject is cancerous tissue. With the above, the uterine cancer diagnosis system ST ends the uterine cancer diagnosis process.

[0056] (Processing of uterine cancer diagnosis device 1) Next, we will explain the details of the normal tissue / cancer tissue determination process of the uterine cancer determination device 1. The normal tissue / cancer tissue determination process includes a nuclear region image generation process for generating a Z-stack nuclear region image ZN1, a learning process for generating a classification model M1, and a determination process for determining the possibility that the uterine tissue is cancer tissue based on the classification model M1.

[0057] FIG. 8 is a diagram showing an example of the nucleus region image generating process according to this embodiment. Step S110: The THG image acquiring unit 100 acquires a Z-stack captured image ZP1 captured by the multiphoton microscope 2. That is, the THG image acquiring unit 100 acquires a third harmonic image of the uterine tissue of the subject obtained by the multiphoton microscope 2. The THG image acquisition unit 100 supplies the acquired Z-stack captured image ZP1 to the nucleus determination operation reception unit 101.

[0058] Here, the Z-stack nucleus region image ZN1 will be described in detail with reference to Fig. 9. Fig. 9 is a diagram showing an example of the Z-stack nucleus region image ZN1 according to this embodiment. In this embodiment, the Z-stack nucleus region image ZNZP1 is a plurality of images. The THG images PTi (i = 1, 2, ..., 317) included in the Z-stack captured image ZP1 are made up of 512 x 512 pixels.

[0059] The Z-stack captured image ZP1 is composed of 157 THG images PTi of uterine tissue, which is cancerous tissue, and 160 THG images PTi of uterine tissue, which is normal tissue. Nine specimens of uterine tissue were used to capture the THG images PTi of uterine tissue, which is cancerous tissue, and the THG images PTi of uterine tissue, which is normal tissue.

[0060] As an example, the uterine cancer diagnosis device 1 uses an image of one specimen among the THG images PTi in which uterine tissue, which is cancerous tissue, is imaged, for the diagnosis process. The uterine cancer diagnosis device 1 uses images of 17 specimens, including the remaining THG images PTi in which uterine tissue, which is cancerous tissue, is imaged, which are not used for the diagnosis process, and the THG images PTi in which uterine tissue, which is normal tissue, for the learning process.

[0061] The third harmonic images for training, which are the multiple THG images PTi used in the training process, are multiple cross-sectional images of uterine epithelial tissue.

[0062] Returning to FIG. 8, the explanation of the nuclear region image generation process will be continued. Step S120: The nucleus determination operation receiving unit 101 receives a nucleus determination operation from a user of the uterine cancer determination device 1 via the operation input unit 16. The nucleus determination operation receiving unit 101 supplies information indicating the received nucleus determination operation and the Z-stack captured image ZP1 to the image processing unit 102.

[0063] Here, the nucleus determination operation receiving unit 101 displays, for example, the Z-stack captured image ZP1 acquired by the THG image acquiring unit 100 on the display device 3. The user of the uterine cancer determination device 1 determines a region indicating a cell nucleus of uterine tissue in each of the THG images PTi while checking the THG images PTi included in the Z-stack captured image ZP1 displayed on the display device 3. The user of the uterine cancer determination device 1 inputs an operation to determine the region indicating a cell nucleus via the operation input unit 16. As an example, the nucleus determination operation is performed by tracing the outline of the region determined to be a region indicating a cell nucleus with a mouse pointer via the operation input unit 16, which is a mouse, in the THG image PTi displayed on the display device 3, which is a display.

[0064] The nucleus determination operation may be performed by tracing the outline of a region determined to represent a cell nucleus using the operation input unit 16, which is a touch pen, on the THG image PTi displayed on the display device 3, which is a touch panel.

[0065] Step S130: The image processing unit 102 generates a Z-stack nucleus region image ZN1 from the Z-stack captured images ZP1 based on the nucleus determination operation received by the nucleus determination operation receiving unit 101. Here, the image processing unit 102 generates the Z-stack nucleus region image ZN1 by coloring the outline of the region indicating the cell nucleus of the uterine tissue in each of the THG images PTi based on the nucleus determination operation. Step S140: The image processing unit 102 outputs the generated Z-stack nuclear region image ZN1 to the feature amount processing unit 11. With the above, the uterine cancer diagnosing device 1 ends the nuclear region image generating process.

[0066] Next, the learning process of the uterine cancer determination device 1 will be described with reference to Fig. 10. Fig. 10 is a diagram showing an example of the learning process according to this embodiment.

[0067] Step S210: The nuclear region image acquisition unit 110 acquires the Z-stack nuclear region image ZN1 supplied from the nuclear region image generation unit 10. Here, the Z-stack nuclear region image ZN1 supplied from the nuclear region image generation unit 10 is generated in the nuclear region image generation process described above, and the nuclear region image PNi in which the uterine tissue, which is cancerous tissue, is imaged and the nuclear region image PNi in which the uterine tissue, which is normal tissue, is imaged are previously determined. The nuclear region image acquisition unit 110 supplies the acquired Z-stack nuclear region image ZN1 to the nuclear measurement unit 111.

[0068] Step S220: The nucleus measurement unit 111 measures the area, circularity, and nearest neighbor distance for each cell nucleus based on the image of the cell nucleus captured in the Z-stack nucleus region image ZN1 acquired by the nucleus region image acquisition unit 110. The nucleus measurement unit 111 supplies the measurement results to the feature extraction unit 112.

[0069] Here, the measurement of the cell nucleus performed by the nucleus measurement unit 111 will be described with reference to Fig. 11. Fig. 11 is a diagram showing an example of the measurement of the cell nucleus according to this embodiment. The nucleus measurement unit 111 measures the number of pixels in the cell nucleus region x the area per pixel. Here, the area per pixel depends on the magnification of the multiphoton microscope 2, and is, for example, 1 μm 2 .

[0070] The nucleus measurement unit 111 measures the circularity as 4π×(area) / (square of circumference). The circularity can take a value between 0 and 1. The circularity takes a value closer to 1 as the outline of the cell nucleus becomes closer to a circle. The nucleus measurement unit 111 associates a label with a cell nucleus, and calculates the Euclidean distance from the center of gravity of the label to be measured to the center of gravity of another label that is closest to the center of gravity of the label.

[0071] Returning to FIG. 10, the explanation of the learning process will be continued. Step S230: The feature amount extraction unit 112 extracts a feature amount C1 for each nuclear region image PNi included in the Z-stack nuclear region image ZN1 based on the result of measurement by the nuclear measurement unit 111.

[0072] Step S240: The feature extraction unit 112 judges whether the number of cell nuclei for each nucleus region image PNi is greater than 0. As described above, the number of cell nuclei is included in the feature C1, and the feature extraction unit 112 makes the judgment based on the feature C1.

[0073] The number of cell nuclei captured in the nuclear region image PNi is zero when, for example, the epithelium of the uterine tissue is tilted relative to the direction perpendicular to the Z-axis direction, and imaging of the area near the epithelium is performed without including the uterine tissue in the imaging plane.

[0074] When the feature extraction unit 112 determines that the number of cell nuclei is greater than 0 for each nucleus region image PNi (step S240; YES), the feature processing unit 11 executes the process of step S250. On the other hand, when the feature extraction unit 112 determines that the number of cell nuclei is 0 for each nucleus region image PNi (step S240; NO), the feature processing unit 11 executes the process of step S2130.

[0075] Step S250: The Z-position correction unit 113 performs correction in the Z-axis direction for the Z-stack nucleus region image ZN1. Here, the Z-position correction unit 113 corrects the value of the Z coordinate so that, for example, the Z coordinate of the imaging plane corresponding to the nucleus region image PNi with the smallest Z coordinate among the nucleus region images PNi determined by the feature amount extraction unit 112 to have more than one cell nucleus becomes the origin of the Z axis. In addition, it is preferable that the Z position correction unit 113 performs correction in the Z axis direction even when the number of nuclear region images included in the Z-stack nuclear region image ZN1 is one.

[0076] Step S260: The feature amount scaling unit 114 calculates the average value and standard deviation for each type of feature amount C1 extracted by the feature amount extracting unit 112 for the nucleus region image PNi included in the Z-stack nucleus region image ZN1.

[0077] Step S270: The feature amount scaling unit 114 scales the feature amount C1 extracted by the feature amount extraction unit 112 using the average value and standard deviation calculated for each type. Here, as one example, the feature amount scaling unit 114 scales the feature amount C1 based on normalization or standardization. For example, the feature amount scaling unit 114 unifies the range of values ​​that different types of feature amounts included in the feature amount C1 can take by normalization. Alternatively, the feature amount scaling unit 114 sets the average value of different types of feature amounts included in the feature amount C1 to 0 and the variance to 1 by standardization.

[0078] Step S280: The data division unit 12 divides the feature C1 scaled by the feature scaling unit 114 into learning feature data CL1 used for learning and determination feature data CE1 used for determination. The data division unit 12 sets, among the feature C1, a feature extracted from an image of a certain specimen in a THG image PTi in which uterine tissue, which is cancerous tissue, is imaged, as determination feature data CE1. The data division unit 12 sets the remaining feature of the feature C1 as learning feature data CL1.

[0079] As described above, in the Z-stack nuclear region image ZN1 acquired by the nuclear region image acquisition unit 110 in step S210, the nuclear region image PNi in which the uterine tissue, which is cancerous tissue, is imaged and the nuclear region image PNi in which the uterine tissue, which is normal tissue, are identified in advance. The data division unit 12 associates a label for distinguishing between the feature extracted from the nuclear region image PNi in which the uterine tissue, which is cancerous tissue, is imaged and the feature extracted from the nuclear region image PNi in which the uterine tissue, which is normal tissue, is imaged, for each data of the learning feature data CL1.

[0080] Step S290: The data division unit 12 outputs the learning feature data CL1 to the classification model generation unit 13. The data division unit 12 outputs the determination feature data CE1 to the cancer tissue determination unit . Step S2100: The classification model generation unit 13 acquires the learning feature data CL1 output from the data division unit 12.

[0081] Step S2110: The classification model generation unit 13 generates the classification model M1 based on machine learning using the acquired learning feature data CL1. Here, the classification model generation unit 13 generates the classification model M1 based on a nonlinear support vector machine, as an example. Note that the classification model generation unit 13 may generate the classification model M1 based on machine learning other than the nonlinear support vector machine.

[0082] As described above, the learning feature data CL1 is extracted from the Z-stack nuclear region image ZN1 generated from the Z-stack captured image ZP1 consisting of the THG image PTi of the cancerous uterine tissue and the THG image PTi of the normal uterine tissue. Therefore, the classification model M1 is a classification model trained using the learning third harmonic image of the normal uterine tissue obtained by a multiphoton microscope and / or the learning third harmonic image of the uterine cancer tissue obtained by a multiphoton microscope.

[0083] Step S2120: The classification model generation unit 13 outputs the generated classification model M1 to the cancer tissue determination unit 14. With the above, the uterine cancer diagnosing device 1 ends the learning process.

[0084] Step S2130: The feature extraction unit 112 discards the nucleus region image PNi in which the number of cell nuclei is determined to be 0. In other words, the feature extraction unit 112 does not use the nucleus region image PNi in which the number of cell nuclei is determined to be 0 in the subsequent processes (the processes from step S250 to step S2130).

[0085] Next, the determination process of the uterine cancer determination device 1 will be described with reference to Fig. 12. Fig. 12 is a diagram showing an example of the determination process according to the present embodiment.

[0086] Step S300: The cancer tissue determination unit 14 acquires the determination feature data CE1 output from the data division unit 12. Step S310: The cancer tissue determination unit 14 acquires the classification model M1 output from the classification model generation unit 13.

[0087] Step S320: The cancer tissue determination unit 14 uses the determination feature data CE1 and the classification model M1 to determine the possibility that the uterine tissue captured in the Z-stack THG image ZT1 is cancer tissue. As described above, the feature C1 is extracted for each nucleus region image PNi included in the Z-stack nucleus region image ZN1 generated from the Z-stack THG image ZT1. Therefore, using the determination feature data CE1, it is possible to determine the possibility that the uterine tissue captured in the THG image PTi corresponding to the nucleus region image PNi from which the determination feature data CE1 is extracted is cancer tissue.

[0088] As described above, the feature C1 is a feature indicating the state of the cell nucleus captured in the nuclear region image PNi included in the Z-stack nuclear region image ZN1. Therefore, the cancer tissue determination unit 14 determines the possibility that the subject's uterine tissue is cancerous tissue based on the state of the cell nucleus in the third harmonic image of the subject's uterine tissue.

[0089] As described above, the cancer tissue determination unit 14 determines the possibility that the uterine tissue captured in the Z-stack THG image ZT1 is cancer tissue based on the classification model M1. As described above, the classification model M1 is a classification model trained using a learning third harmonic image of normal uterine tissue obtained by a multiphoton microscope and / or a learning third harmonic image of uterine cancer tissue obtained by a multiphoton microscope. Therefore, the cancer tissue determination unit 14 refers to the classification model M1 trained using a learning third harmonic image of normal uterine tissue obtained by a multiphoton microscope and / or a learning third harmonic image of uterine cancer tissue obtained by a multiphoton microscope, and determines the possibility that the subject's uterine tissue is cancer tissue based on the state of the cell nucleus in the third harmonic image of the subject's uterine tissue.

[0090] As described above, the classification model M1 is a model obtained by executing machine learning using the feature C1. The feature C1 includes the median and median absolute error of the area of ​​the cell nuclei, the median and median absolute error of the circularity of the cell nuclei, and the median and median absolute error of the nearest neighbor distance between the cell nuclei. The cancer tissue determination unit 14 determines that the uterine tissue of the subject is highly likely to be cancer tissue when at least one of the following conditions is met: (i) the average nuclear area is increased compared to normal uterine tissue; (ii) the variability in nuclear area is increased compared to normal uterine tissue; (iii) the density of nuclear cells is increased compared to normal uterine tissue; (iv) the variability in the density of nuclear cells is increased compared to normal uterine tissue; (v) the irregularity in the shape of nuclear cells is increased compared to normal uterine tissue; and (vi) the variability in the shape of nuclear cells is increased compared to normal uterine tissue.

[0091] Here, the cancer tissue determination unit 14 calculates the possibility that the uterine tissue is cancer tissue based on the ratio of the Z-stack image ZP1 for determination that is determined to be highly likely to be cancer tissue to all the Z-stack image ZP1 for determination. First, the cancer tissue determination unit 14 determines the possibility that the uterine tissue captured for each of the multiple THG images PT1 included in the Z-stack image ZP1 for determination is cancer tissue. Next, the cancer tissue determination unit 14 calculates the possibility that the uterine tissue is cancer tissue based on the ratio of the Z-stack image ZP1 for determination that is determined to be highly likely to be cancer tissue to all the Z-stack image ZP1 for determination.

[0092] In other words, the cancer tissue determination unit 14 determines the possibility that each of the multiple third harmonic images of the subject's uterine tissue is cancerous tissue, and calculates the possibility that the subject's uterine tissue is cancerous tissue based on the proportion of third harmonic images that are determined to have a high possibility of being cancerous tissue to all third harmonic images.

[0093] Step S330: The cancer tissue determination unit 14 outputs the determination result A to the output unit 15. With the above, the uterine cancer determination device 1 ends the determination process.

[0094] Fig. 13 is a diagram showing an example of a determination result A according to the present embodiment. The determination accuracy is expressed as a percentage as the ratio of the number of Z-stack captured images ZP1 for determination for which a correct determination result was calculated to the number of Z-stack captured images ZP1 for determination used in the determination process. The specimen "q1607" in Fig. 13 is the specimen used for determination, and the determination accuracy for the specimen "q1607" was 83.3%.

[0095] In addition, Fig. 13 also shows the determination accuracy for each sample for the learning Z-stack captured image ZP1 including the THG image of the cancer tissue and the THG image of the normal tissue. In Fig. 13, the average value of the determination accuracy for all the Z-stack captured images ZP1 obtained by repeating the calculation of accuracy for the number of samples by using a certain sample as a determination sample and the rest as learning samples was 90.1%. According to the uterine cancer determination device 1, it is possible to determine the possibility that the uterine tissue is cancerous tissue with a determination accuracy of about 90%.

[0096] (summary) As described above, the uterine cancer diagnosing device 1 according to this embodiment includes the THG image acquiring section 100 and the cancer tissue diagnosing section . The THG image acquisition section 100 acquires a third harmonic image of the subject's uterine tissue obtained by the multiphoton microscope 2 (in this example, a Z-stack THG image ZT1). The cancer tissue determination unit 14 determines the possibility that the subject's uterine tissue is cancer tissue based on the state of cell nuclei in the third harmonic image of the subject's uterine tissue (in this example, Z-stack THG image ZT1).

[0097] With this configuration, the uterine cancer diagnosis device 1 of this embodiment can determine the possibility that a subject's uterine tissue is cancerous tissue based on the state of cell nuclei in a third harmonic image of the subject's uterine tissue, and can therefore determine the possibility that the uterine tissue is cancerous tissue without staining the uterine tissue.

[0098] In the uterine cancer diagnosis device 1 according to this embodiment, the state of the cell nucleus is at least one selected from the group consisting of the area of ​​the cell nucleus, the density of the cell nucleus, the shape of the cell nucleus, and the state of the nucleolus. With this configuration, the uterine cancer diagnosis device 1 of this embodiment can determine the possibility that the subject's uterine tissue is cancerous tissue based on at least one selected from the group consisting of the area of ​​the cell nucleus, the density of the cell nucleus, the shape of the cell nucleus, and the state of the nucleolus in the third harmonic image of the subject's uterine tissue, and therefore can determine the possibility that the uterine tissue is cancerous tissue with higher accuracy than when not based on at least one selected from this group.

[0099] Furthermore, in the uterine cancer determination device 1 according to this embodiment, the cancer tissue determination unit 14 determines that the uterine tissue of the subject is highly likely to be cancer tissue when at least one of the following conditions is met: (i) the average nuclear area is increased compared to normal uterine tissue; (ii) the variability in nuclear area is increased compared to normal uterine tissue; (iii) the density of nuclear cells is increased compared to normal uterine tissue; (iv) the variability in the density of nuclear cells is increased compared to normal uterine tissue; (v) the irregularity in the shape of nuclear cells is increased compared to normal uterine tissue; and (vi) the variability in the shape of nuclear cells is increased compared to normal uterine tissue.

[0100] With this configuration, the uterine cancer diagnosis device 1 of this embodiment can determine the possibility that uterine tissue is cancerous tissue based on (i) to (vi), which are characteristics that coincide with pathological findings about cell nuclei in cancerous tissue in uterine tissue, and can therefore determine the possibility that uterine tissue is cancerous tissue with higher accuracy than when not based on (i) to (vi).

[0101] Moreover, in the uterine cancer diagnosing device 1 according to this embodiment, the uterine tissue of the subject is exposed to acetic acid in advance. With this configuration, the uterine cancer diagnosis device 1 of this embodiment can image cell nuclei in the third harmonic image of the subject's uterine tissue more clearly than when the uterine tissue has not been exposed to acetic acid in advance, and therefore can determine the possibility that the uterine tissue is cancerous tissue with higher accuracy than when the uterine tissue has not been exposed to acetic acid in advance.

[0102] Furthermore, in the uterine cancer diagnosis device 1 of this embodiment, the cancer tissue diagnosis unit 14 refers to the classification model M1 trained using a learning third harmonic image of normal uterine tissue obtained by the multiphoton microscope 2 and / or a learning third harmonic image of uterine cancer tissue obtained by the multiphoton microscope 2, and determines the possibility that the subject's uterine tissue is cancerous tissue based on the state of cell nuclei in the third harmonic image of the subject's uterine tissue (in this example, the Z-stack THG image ZT1).

[0103] With this configuration, the uterine cancer diagnosis device 1 of this embodiment can determine the possibility that the subject's uterine tissue is cancerous tissue by referring to the classification model M1, and can therefore determine the possibility that the uterine tissue is cancerous tissue with greater accuracy than when not referring to the classification model M1.

[0104] Moreover, in the uterine cancer diagnosing device 1 according to this embodiment, the learning third harmonic images are a plurality of cross-sectional images of the uterine epithelial tissue. With this configuration, the uterine cancer diagnosis device 1 of this embodiment can perform learning using multiple cross-sectional images of uterine epithelial tissue, and can therefore determine the possibility that uterine tissue is cancerous tissue with higher accuracy than when a single cross-sectional image is used for learning.

[0105] Moreover, in the uterine cancer diagnosing device 1 according to this embodiment, the third harmonic image of the uterine tissue of the subject (in this example, the Z-stack THG image ZT1) is a plurality of cross-sectional images of the uterine epithelial tissue of the subject. With this configuration, the uterine cancer diagnosis device 1 of this embodiment can determine the possibility that the uterine tissue is cancerous tissue for each depth direction of the subject's uterine epithelial tissue, thereby enabling the possibility that the uterine tissue is cancerous tissue to be determined with greater accuracy than when a single cross-sectional image is used.

[0106] Furthermore, in the uterine cancer diagnosis device 1 of this embodiment, the cancer tissue diagnosis unit 14 determines the possibility that each of multiple third harmonic images (in this example, Z-stack THG image ZT1) of the subject's uterine tissue is cancerous tissue, and calculates the possibility that the subject's uterine tissue is cancerous tissue based on the proportion of third harmonic images determined to have a high possibility of being cancerous tissue to all third harmonic images (in this example, Z-stack THG image ZT1).

[0107] With this configuration, the uterine cancer diagnosis device 1 of this embodiment can calculate the possibility that the subject's uterine tissue is cancerous tissue based on the proportion of third harmonic images that are determined to have a high possibility of being cancerous tissue among multiple third harmonic images of the subject's uterine tissue, and can therefore calculate the possibility that the tissue is cancerous tissue for the entire depth direction of the subject's uterine tissue.

[0108] Second embodiment The second embodiment will be described in detail below with reference to the drawings. In the above-mentioned first embodiment, the case where the uterine cancer determination device generates the nuclear region image from the THG image based on the nuclear determination operation received from the user is described. In the present embodiment, the case where the uterine cancer determination device determines the nuclear region among the regions included in the THG image based on machine learning and generates the nuclear region image is described. The uterine cancer diagnosis device according to this embodiment is referred to as a uterine cancer diagnosis device 1a.

[0109] (Configuration of the uterine cancer diagnosis device) FIG. 14 is a diagram showing an example of a uterine cancer diagnosis device 1a according to this embodiment. Comparing the uterine cancer diagnosis device 1a according to this embodiment (FIG. 14) with the uterine cancer diagnosis device 1 according to the first embodiment (FIG. 4), it is different in that a memory unit 17 is provided instead of the nuclear region image generating unit 10a and the operation input unit 16. Here, the functions of the other components (feature amount processing unit 11, data dividing unit 12, classification model generating unit 13, cancer tissue determining unit 14, and output unit 15) are the same as those of the first embodiment. The description of the same functions as those of the first embodiment will be omitted, and in the second embodiment, the parts different from the first embodiment will be mainly described.

[0110] The uterine cancer diagnosing device 1a includes a nuclear region image generating unit 10a, a feature amount processing unit 11, a data dividing unit 12, a classification model generating unit 13, a cancer tissue diagnosing unit 14, an output unit 15, and a storage unit 17.

[0111] The nuclear region image generating unit 10a generates the Z-stack nuclear region image ZN1a from the Z-stack captured image ZP1 based on machine learning. Here, the machine learning used by the nuclear region image generating unit 10a is, for example, deep learning.

[0112] Here, an overview of deep learning used by the nucleus region image generating unit 10a will be described with reference to Fig. 15. Fig. 15 is a diagram showing an example of deep learning used by the nucleus region image generating unit 10a according to the present embodiment.

[0113] As an example, the nucleus region image generating unit 10a uses U-Net, which is known as a deep learning architecture, as the nucleus region extraction model M20. U-Net is a convolutional neural network (CNN), and has been reported to have high accuracy in segmentation of medical images. By using U-Net, it is possible to capture the characteristics of the entire image, and to minimize the error of the correct image (distribution of cell nuclei and non-cell nuclei) at multiple resolutions for the entire image. In U-Net, all layers are composed of convolution layers, and an up-conv layer is provided toward the output layer.

[0114] As a parameter of the nuclear region extraction model M20, the input size is 512×512, which is the image size of the Z-stack captured image ZP1.

[0115] The nucleus region image generating unit 10a performs learning based on the nucleus region extraction model M20 using the learning Z-stack THG image LT and the nucleus region annotation image PA. Here, the nucleus region annotation image PA is an image showing the nucleus region for each of the multiple THG images included in the learning Z-stack THG image LT.

[0116] The nucleus region image generating unit 10a generates a nucleus region extraction model M21 as a result of executing learning. The nucleus region image generating unit 10a applies the generated nucleus region extraction model M21 to the determination Z-stack THG image ET, and outputs a nucleus region image PN1, which is an image in which the nucleus region of the determination Z-stack THG image ET is determined.

[0117] Returning to FIG. 14, the description of the configuration of the uterine cancer diagnosing device 1a will be continued. The nucleus region image generating unit 10a includes a THG image acquiring unit 100a, a nucleus region annotation image acquiring unit 103a, a nucleus region extraction model generating unit 140a, a nucleus region determining unit 105a, and an image processing unit 106a.

[0118] The THG image acquisition unit 100a acquires a Z-stack captured image ZP1. The THG image acquisition unit 100a includes a learning THG image acquisition unit 101a and a judgment THG image acquisition unit 102a. The Z-stack captured image ZP1 acquired by the THG image acquisition unit 100a includes a learning Z-stack THG image LT and a judgment Z-stack THG image ET. The learning THG image acquisition unit 101a acquires a learning Z-stack THG image LT. The determination-use THG image acquisition unit 102a acquires a determination-use Z-stack THG image ET.

[0119] The nucleus region annotation image acquisition unit 103 a acquires the nucleus region annotation image PA from the storage unit 17 . The nucleus region extraction model generating unit 140a executes learning based on the nucleus region extraction model M20 using the learning Z-stack THG image LT and the nucleus region annotation image PA, and generates a nucleus region extraction model M21 as a learning result.

[0120] The nucleus region determination unit 105a performs nucleus region determination for the determination-use Z-stack THG image ET based on the nucleus region extraction model M21 generated by the nucleus region extraction model generation unit 140a. The image processing unit 106a generates a Z-stack nucleus region image ZN1a based on the determination result of the nucleus region determination unit 105a.

[0121] The storage unit 17 stores various types of information. The various types of information include a nucleus region annotation image PA and threshold information TH. The threshold information TH is information indicating various thresholds used in image processing performed by the image processing unit 106a.

[0122] (Nuclear region image generation processing) Next, the nuclear region image generating process in which the uterine cancer diagnosis device 1a generates the Z-stack nuclear region image ZN1a will be described in detail. The nuclear region image generating process includes a nuclear region learning process for generating a nuclear region extraction model M21, and a nuclear region determination process for determining a nuclear region based on the nuclear region extraction model M21.

[0123] FIG. 16 is a diagram showing an example of the core region learning process according to the present embodiment. Step S400: The learning THG image acquisition unit 101a acquires a learning Z-stack THG image LT. The learning THG image acquisition unit 101a supplies the acquired learning Z-stack THG image LT to the nucleus region extraction model generation unit 140a.

[0124] Step S410: The nucleus region annotation image acquisition unit 103a acquires the nucleus region annotation image PA from the storage unit 17. The nucleus region annotation image acquisition unit 103a supplies the acquired nucleus region annotation image PA to the nucleus region extraction model generation unit 140a.

[0125] Step S420: The nucleus region extraction model generating unit 140a executes learning based on the nucleus region extraction model M20 using the learning Z-stack THG image LT provided from the learning THG image acquiring unit 101a and the nucleus region annotation image PA provided from the nucleus region annotation image acquiring unit 103a, and generates a nucleus region extraction model M21 as a learning result. Here, the nucleus region extraction model generating unit 140a changes the weight between the nodes of the nucleus region extraction model M20 based on the learning Z-stack THG image LT and the nucleus region annotation image PA to generate the nucleus region extraction model M21.

[0126] Step S430: The nucleus region extraction model generating unit 140a outputs the generated nucleus region extraction model M21 to the nucleus region determining unit 105a.

[0127] Next, the core region determination process will be described with reference to Fig. 17. Fig. 17 is a diagram showing an example of the core region determination process according to this embodiment. Step S500: The determination-use THG image acquisition unit 102a acquires a determination-use Z-stack THG image ET The determination-use THG image acquisition unit 102a supplies the acquired determination-use Z-stack THG image ET to the nucleus region determination unit 105a.

[0128] Step S510: The nucleus region determination unit 105a acquires the nucleus region extraction model M21 output from the nucleus region extraction model generation unit 140a.

[0129] Step S520: The nucleus region determination unit 105a determines the nucleus region for the determination Z-stack THG image ET supplied from the determination THG image acquisition unit 102a based on the acquired nucleus region extraction model M21. Here, the nucleus region determination unit 105a determines the nucleus region for each pixel of the multiple THG images PTi included in the determination Z-stack THG image ET. As a determination result, the nucleus region determination unit 105a assigns a probability value indicating the possibility of being a nucleus region to each pixel of the multiple THG images PTi. The nucleus region determination unit 105a supplies the determination result of the nucleus region to the image processing unit 106a.

[0130] Step S530: The image processing unit 106a generates a Z-stack nucleus region probability value image based on the nucleus region determination result provided by the nucleus region determination unit 105a. The Z-stack nucleus region probability value image is an image in which a probability value indicating the possibility that each pixel of the THG image PTi is a nucleus region is indicated. Here, the probability value indicating the possibility that each pixel is a nucleus region is indicated by, for example, a gray scale.

[0131] Step S540: The image processing unit 106a acquires a binarization threshold value from the storage unit 17. Here, the binarization threshold value is included in the threshold value information TH. Step S550: The image processing unit 106a executes a binarization process. The image processing unit 106a executes a binarization process on the generated Z-stack nucleus region probability value image based on the acquired binarization threshold. The image processing unit 106a generates a Z-stack binarized image from the Z-stack nucleus region probability value image by the binarization process.

[0132] Here, when the probability value indicating the possibility of each pixel being a nuclear region in the Z-stack nuclear region probability value image is equal to or greater than the binarization threshold, the image processing unit 106a assigns, for example, one of the binary luminance values, brightness value 1 (for example, a brightness value corresponding to white), to this pixel. When the probability value indicating the possibility of each pixel being a nuclear region in the Z-stack nuclear region probability value image is less than the binarization threshold, the image processing unit 106a assigns, for example, the other of the binary luminance values, brightness value 2 (for example, a brightness value corresponding to black), to this pixel.

[0133] Step S560: The image processing unit 106a performs a filling process on the generated Z-stack binary image. Here, the image processing unit 106a uses, for example, closing or opening as the filling process. The image processing unit 106a generates a Z-stack filled image from the Z-stack binary image by the filling process.

[0134] Step S570: The image processing unit 106a acquires the area threshold and the circularity threshold from the storage unit 17. Here, the area threshold and the circularity threshold are included in the threshold information TH. Step S580: The image processing unit 106a performs a filtering process The image processing unit 106a performs a filtering process on the generated Z-stack filled image based on the acquired area threshold and circularity threshold.

[0135] Here, the image processing unit 106a pairs adjacent pixels among the pixels (e.g., pixels assigned a white color) included in the Z-stack filled image that are assigned a luminance value of 1, and determines them as candidate regions that are candidates for the nucleus region. The image processing unit 106a also determines a region as a candidate region when only one pixel is isolated.

[0136] The image processing unit 106a measures the area and circularity of each of the determined candidate regions. When the measured area of ​​a candidate region is equal to or greater than the area threshold and the measured circularity is equal to or greater than the circularity threshold, the image processing unit 106a determines not to exclude the candidate region. On the other hand, when the measured area of ​​a candidate region is less than the area threshold or the measured circularity is less than the circularity threshold, the image processing unit 106a determines to exclude the candidate region.

[0137] The image processing unit 106a changes the luminance value of the pixels included in the candidate region determined to be excluded from among the candidate regions included in the Z-stack filled image to a luminance value of 2. For example, when the candidate region is white, the image processing unit 106a changes the color of the candidate region determined to be excluded to black.

[0138] Step S590: The image processing unit 106a performs a segmentation process. The image processing unit 106a performs a segmentation process on the filtered Z-stack filled-in image. The image processing unit 106a uses, for example, WaterShed as a segmentation method. The image processing unit 106a determines the contour of the nuclear region by the segmentation process, and segments the region of the filtered Z-stack filled-in image into a nuclear region and a region other than the nuclear region. The image processing unit 106a generates a Z-stack nuclear region image ZN1a as a result of the division process.

[0139] Step S5100: The image processing unit 106a outputs the generated Z-stack nuclear region image ZN1a to the feature amount processing unit 11. With the above, the nucleus region image generating unit 10a ends the nucleus region determination process.

[0140] The results of the core region determination process will be described with reference to FIGS. Fig. 18 is a diagram showing an example of a nucleus region image and a nucleus region annotation image according to the present embodiment. In Fig. 18, a nucleus region image PNj is shown as an example of a nucleus region image obtained by the nucleus region determination process. The nucleus region annotation image PAj is the nucleus region annotation image used to generate this nucleus region image PNj.

[0141] Next, the determination accuracy of the core region determination process will be described with reference to FIG. 19 is a diagram showing an example of the result of the nucleus region determination process according to the present embodiment. The nucleus region shown in the nucleus region image obtained by the nucleus region determination process includes a first region and a second region. The first region is a region that matches the nucleus region shown in the nucleus region annotation image. The second region is a region that is erroneously determined and does not match the nucleus region shown in the nucleus region annotation image. Meanwhile, the nucleus region shown in the nucleus region annotation image includes a third region that is not included in the nucleus region of the nucleus region image because it has not been determined.

[0142] In Fig. 19, the nuclear region image obtained by the nuclear region determination process is shown with the first and second regions distinguished from each other, and further with the third region added, as images PC1 and PC2. Image PC1 is a nuclear region image of normal tissue. Image PC2 is a nuclear region image of cancer tissue.

[0143] Here, the F value is used to evaluate the accuracy of the nucleus region determination process. The F value is the harmonic mean of the precision rate and the recall rate, and takes a value between 0 and 1, with the closer to 1 the value is the higher the determination accuracy. The precision rate is the ratio of the number of first regions to the number of extracted nucleus regions (first and second regions). The recall rate is the ratio of the number of first regions to the number of nucleus regions (first and third regions) indicated by the nucleus region annotation image.

[0144] By using the F value, it is possible to calculate the determination accuracy that is not dependent on the ratio of the area occupied by the nuclear region in the nuclear region image to the area occupied by the background other than the nuclear region. The F-score for image PC1 was 0.932, and the F-score for image PC2 was 0.909.

[0145] Next, referring to Fig. 20, the F value for the entire Z-stack nuclear region image ZN1a is shown. Fig. 20 is a diagram showing an example of the F value for the entire Z-stack nuclear region image ZN1a according to this embodiment. The average F value for the entire Z-stack nuclear region image ZN1a is 0.854. In the uterine cancer determination device 1a, the nuclear region is extracted with high accuracy from both the image PC1 and the image PC2.

[0146] Next, the determination result Aa of the uterine cancer determination process of the uterine cancer determination device 1a will be described with reference to Fig. 21. Fig. 21 is a diagram showing an example of the determination result Aa according to this embodiment. The uterine cancer determination process of the uterine cancer determination device 1a is a uterine cancer determination process in the case where the Z-stack nuclear region image ZN1a is generated by the nuclear region determination process. When the nuclear region determination process is completed, the uterine cancer determination device 1a performs the uterine cancer determination process by performing the same process as the learning process of Fig. 10 and the determination process of Fig. 12 described above.

[0147] The average value of the determination accuracy for all Z-stack captured images ZP1, which was calculated by using a certain sample as a determination sample and the others as learning samples, was 90.7%. According to the uterine cancer determination device 1a, even when the Z-stack nuclear region image ZN1a is generated by the nuclear region determination process, it is possible to determine the possibility that the uterine tissue is cancerous tissue with a determination accuracy of about 90%.

[0148] (summary) As described above, the uterine cancer diagnosis device 1a according to the present embodiment includes the nuclear region image generating unit 10a. The nuclear region image generating unit 10a generates a Z-stack nuclear region image ZN1a, which is a plurality of images in which a region showing a cell nucleus of a uterine tissue is determined from the Z-stack captured image ZP1 based on machine learning. With this configuration, the uterine cancer determination device 1a according to this embodiment can generate a Z-stack nuclear region image ZN1a based on machine learning, thereby reducing the effort required to determine a region indicating the cell nuclei of the uterine tissue from the Z-stack captured image ZP1.

[0149] (Third embodiment) The third embodiment will be described in detail below with reference to the drawings. In the above first and second embodiments, the case where the uterine cancer diagnosis device determines the possibility that the uterine tissue of the subject is cancerous tissue based on the THG image is described. In the present embodiment, the case where the uterine cancer diagnosis device determines the stage of cancer based on the second harmonic image for the uterine tissue determined to be highly likely to be cancerous tissue based on the THG image is described. The uterine cancer diagnosis system according to this embodiment is referred to as a uterine cancer diagnosis system STb. The uterine cancer diagnosis device according to this embodiment is referred to as a uterine cancer diagnosis device 1b, and the multiphoton microscope is referred to as a multiphoton microscope 2b.

[0150] As the tumor progresses, fibrosis occurs in the surrounding tissue. The fibers generated by fibrosis are composed of various collagen-containing molecules. Some fibrous collagen generates SHG. In addition to the THG image, the multiphoton microscope 2b of this embodiment images the uterine tissue using SHG to generate a second harmonic image. The second harmonic image is an image of a cross section of the uterine tissue generated based on the excitation light irradiated from the irradiation unit of the multiphoton microscope 2b and the light generated by second harmonic generation due to the interaction with the uterine tissue.

[0151] In this embodiment, the captured images included in the Z-stack captured image include a THG image and a second harmonic image, which is an image of uterine tissue captured using SHG. Hereinafter, the second harmonic image is referred to as an SHG image. Also, the Z-stack captured image of the SHG image is referred to as a Z-stack SHG image ZS1. In other words, the Z-stack SHG image ZS1 is a plurality of SHG images of uterine tissue captured using SHG while changing the distance from the uterine tissue in the Z-axis direction.

[0152] Furthermore, each of the multiple images included in the Z-stack SHG image ZS1 is referred to as an SHG image PSi (i=1, 2, . . . , N: N is the number of images included in the Z-stack SHG image ZS1), etc. The multiphoton microscope 2b captures a THG image and an SHG image for each depth of the uterine tissue. The SHG image PSi included in the Z-stack SHG image ZS1 and the THG image PTi included in the Z-stack THG image ZT1 are images captured at a common depth of the uterine tissue and correspond to each other.

[0153] Here, referring to FIG. 22, an SHG image captured by the multiphoton microscope 2b will be described. FIG. 22 is a diagram showing an example of an SHG image PS0 according to this embodiment. In FIG. 22, a fiber-like structure image F1 is shown together with the SHG image PS0 for comparison. The fiber-like structure image F1 is an image extracted and detected from the SHG image PS0 by machine learning. It can be seen that the SHG image PS0 captured by the multiphoton microscope 2b contains fiber-like structures to the same extent as those captured in the fiber-like structure image F1.

[0154] As the tumor progresses, fibrosis spreads from deep to superficial uterine tissue.

[0155] The spread of fibrosis from deep to shallow parts of uterine tissue will now be described with reference to Fig. 23. Fig. 23 is a diagram showing an example of cross sections of uterine tissue for each stage of tumor progression according to this embodiment. In Figure 23A, fibrous structures are present in the deep part of the uterine tissue, while in Figure 23B, a cross-section of the uterine tissue after tumor progression shows fibrosis in the superficial part.

[0156] In this embodiment, the Z axis is also set in a direction from the epithelium of the uterine tissue toward the inside. As the tumor progresses, fibrosis spreads from areas with high Z coordinate values ​​to areas with low Z coordinate values ​​in the uterine tissue. In the shallow part of the uterine tissue, the depth at which fibrosis occurs is unknown in advance because it varies depending on the progression of the tumor. In the uterine cancer determination device 1b, by using the Z-stack SHG image ZS1, fibrous structures can be extracted regardless of the depth at which fibrosis occurs in the shallow part of the tissue.

[0157] (Configuration of the uterine cancer diagnosis device) FIG. 24 is a diagram showing an example of a uterine cancer diagnosis device 1b according to this embodiment. Comparing the uterine cancer diagnosis device 1b according to this embodiment (FIG. 24) with the uterine cancer diagnosis device 1a according to the second embodiment (FIG. 14), the SHG image acquisition unit 18b and the cancer progression stage determination unit 19b are different. Here, the functions of the other components (nucleus region image generation unit 10a, feature amount processing unit 11, data division unit 12, classification model generation unit 13, cancer tissue determination unit 14, and output unit 15) are the same as those of the second embodiment. The description of the same functions as those of the second embodiment will be omitted, and the third embodiment will be mainly described with respect to the parts different from the second embodiment.

[0158] The uterine cancer assessment device 1b classifies the stage of a tumor by extracting fibrous structural features from a Z-stack SHG image ZS1 of the shallow part of the uterine tissue using a multiphoton microscope 2b and quantifying it as the amount of fibrosis. Here, the stage of a tumor is classified into non-invasive and invasive, for example. Non-invasive corresponds to pre-cancer, and invasive corresponds to invasive cancer.

[0159] The uterine cancer diagnosis device 1b includes a nuclear region image generating unit 10a, a feature amount processing unit 11, a data dividing unit 12, a classification model generating unit 13, a cancer tissue determining unit 14, an output unit 15, a storage unit 17, an SHG image acquiring unit 18b, and a cancer progression determining unit 19b. Note that the uterine cancer diagnosis device 1b may include a nuclear region image generating unit 10 instead of the nuclear region image generating unit 10a, similarly to the uterine cancer diagnosis device 1 according to the first embodiment (FIG. 4).

[0160] The SHG image acquisition unit 18b acquires a Z-stack SHG image ZS1 captured by the multiphoton microscope 2b. When the cancer tissue determination unit 14 determines that the subject's uterine tissue is highly likely to be cancer tissue, the cancer progression determination unit 19b determines the progression of cancer based on the state of the fibrous structure in the SHG image PSi contained in the Z-stack SHG image ZS1. The cancer progression determination section 19b includes a fibrous structure determination section 190b, a fibrous proliferation amount calculation section 191b, and a precancer / invasive cancer determination section 192b.

[0161] The fibrous structure determination unit 190b performs a fibrous structure determination for the Z-stack SHG image ZS1. Here, in the present embodiment, the fibrous structure determination unit 190b performs a fibrous structure determination based on machine learning, as an example. The fibrous structure determination unit 190b generates a Z-stack fibrous structure image ZF1 from the Z-stack SHG image ZS1 based on the determination result. The Z-stack fibrous structure image ZF1 is a Z-stack image in which a fibrous structure is shown. Each of the multiple images included in the Z-stack fiber-like structure image ZF1 is referred to as a fiber-like structure image PFi (i=1, 2, . . . , N: N is the number of images included in the Z-stack fiber-like structure image ZF1), etc.

[0162] The fibrous proliferation amount calculation unit 191b calculates the amount of fibrous proliferation in the shallow part of the uterine tissue captured as the Z-stack SHG image ZS1 based on the Z-stack fibrous structure image ZF1 generated by the fibrous structure determination unit 190b. The precancer / invasive cancer determination unit 192b determines whether the uterine tissue determined to be highly likely to be cancerous tissue is precancer or invasive cancer, based on the amount of fibrosis calculated by the fibrosis amount calculation unit 191b.

[0163] (Processing of uterine cancer diagnosis system) Next, a uterine cancer diagnosis process, which is a process of the uterine cancer diagnosis system STb, will be described. Fig. 25 is a diagram showing an example of the uterine cancer diagnosis process according to this embodiment. Note that the processes in steps S610, S620, S630, and S640 are similar to the processes in steps S10, S20, S30, and S40 in FIG. 7, and therefore descriptions thereof will be omitted.

[0164] Step S650: The uterine cancer determination device 1b performs a precancer / invasive cancer determination process, which is a process for determining whether the uterine tissue of the subject is a precancerous tissue or an invasive cancer tissue, based on the Z-stack SHG image ZS1 captured by the multiphoton microscope 2b. The details of the precancer / invasive cancer determination process will be described later with reference to FIG.

[0165] Step S660: The output unit 15 of the uterine cancer determination device 1b performs processing based on the determination result Ab of the cancer progress determination unit 19b. The determination result Ab indicates whether the subject's uterine tissue is precancerous or invasive cancer. If the subject's uterine tissue is determined to be precancerous (step S660; YES), the output unit 15 outputs a result indicating that the subject's uterine tissue is precancerous to the display device 3. Thereafter, the display device 3 performs processing of step S670. On the other hand, if it is determined that the subject's uterine tissue is not precancerous, i.e., is invasive cancer (step S660; NO), the output unit 15 outputs to the display device 3 a result indicating that the subject's uterine tissue is invasive cancer.

[0166] Step S670: The display device 3 displays the result indicating that the subject's uterine tissue is precancerous. Step S680: The display device 3 displays the result indicating that the subject's uterine tissue has invasive cancer. With the above, the uterine cancer diagnosis system STb ends the uterine cancer diagnosis process.

[0167] Next, the precancer / invasion cancer determination process will be described with reference to Fig. 26. Fig. 26 is a diagram showing an example of the precancer / invasion cancer determination process according to this embodiment. Step S700: The SHG image acquiring unit 18b acquires a Z-stack SHG image ZS1 captured by the multiphoton microscope 2b. That is, the SHG image acquiring unit 18b acquires a second harmonic image of the uterine tissue of the subject obtained by the multiphoton microscope 2b. The SHG image acquisition unit 18b supplies the acquired Z-stack SHG image ZS1 to the fibrous structure determination unit 190b.

[0168] Step S710: The fibrous structure determination unit 190b performs fibrous structure determination for the Z-stack SHG image ZS1. The fibrous structure determination unit 190b generates a Z-stack fibrous structure image ZF1 from the Z-stack SHG image ZS1 based on the determination result. The fibrous structure determination unit 190b supplies the generated Z-stack fibrous structure image ZF1 to the fibrous increase amount calculation unit 191b.

[0169] Here, the fibrous structure determination unit 190b performs the determination based on machine learning, for example. The fibrous structure determination unit 190b uses U-Net, for example, as the machine learning.

[0170] Here, a fiber-like structure image PF1 included in the Z-stack fiber-like structure image ZF1 generated by the fiber-like structure determination unit 190b will be described with reference to Fig. 27. Fig. 27 is a diagram showing an example of the fiber-like structure image PF1 according to the present embodiment. The fibrous structure image PF1 is an image in which the fibrous structure is determined based on the SHG image PS1 included in the Z-stack SHG image ZS1. As an example, the SHG image PS1 is an SHG image obtained by capturing tissue at a depth of 15 μm from the epithelial tissue of the uterine tissue toward the inside.

[0171] In the present embodiment, an example of the case where the fibrous structure determination unit 190b determines the fibrous structure of the Z-stack SHG image ZS1 using machine learning has been described, but the present invention is not limited to this. The determination of the fibrous structure may be performed by a user of the uterine cancer determination device 1b. When the determination of the fibrous structure is performed by a user of the uterine cancer determination device 1b, the cancer progress determination unit 19b receives, for example, an operation to determine the fibrous structure in the Z-stack SHG image ZS1 via the operation input unit 16. The cancer progress determination unit 19b generates a Z-stack fibrous structure image ZF1 based on the received operation to determine the fibrous structure.

[0172] Returning to FIG. 26, the precancer / invasive cancer determination process will be continued. Step S720: The fibrosis amount calculation unit 191b calculates the amount of fibrosis in the shallow part of the uterine tissue captured as the Z-stack SHG image ZS1 based on the Z-stack fibrous structure image ZF1 generated by the fibrous structure determination unit 190b. The fibrosis amount calculation unit 191b supplies the calculated amount of fibrosis to the precancer / invasive cancer determination unit 192b.

[0173] Here, the fibrosis amount calculation unit 191b calculates the ratio of the area of ​​the portion in which the fibrous structure is imaged to the total area of ​​the Z-stack fibrous structure image ZFi as the amount of fibrosis. The fibrosis amount calculation unit 191b calculates the area based on the number of pixels. That is, the fibrosis amount calculation unit 191b calculates the ratio of the number of pixels of the portion in which the fibrous structure is imaged to the total number of pixels of the Z-stack fibrous structure image ZFi as the amount of fibrosis. Hereinafter, this ratio will be referred to as a detected pixel ratio.

[0174] Here, the detection pixel ratio calculated by the fibrosis increase amount calculation unit 191b will be described with reference to Fig. 28 and Fig. 29. Fig. 28 is a diagram showing an example of the detection pixel ratio according to this embodiment. In Fig. 28, the detection pixel ratios of 16 specimens of invasive cancerous uterine tissue, 5 specimens of precancerous uterine tissue, and 15 specimens of normal uterine tissue are shown with respect to the Z axis of the Z stack image. As described above, the precancer / invasive cancer determination process is performed after the uterine tissue is determined to be cancerous tissue. In Figure 28, however, in addition to the cancerous tissue (invasive cancer and precancer), the detection pixel ratio calculated using Z-stack SHG images of normal tissue is also shown for comparison.

[0175] As the Z-axis coordinate value increases, the depth of the imaged uterine tissue from the epithelium increases. The Z-coordinate value of the epithelium is 0. The Z-coordinate value of the boundary between the shallow and deep uterine tissue is 50.

[0176] FIG. 29 is a diagram showing an example of the average value of the detection pixel ratio for the shallow part of the uterine tissue according to this embodiment. The average value for the shallow part of the uterine tissue is the average for the Z-axis value of the Z-stack image. The three graphs shown in FIG. 29 show the average value of the detection pixel ratio for the shallow part of the uterine tissue against the number indicating each specimen. Graph G1 is the average value of the detection pixel ratio for invasive cancer uterine tissue. Graph G2 is the average value of the detection pixel ratio for precancerous uterine tissue. Graph G3 is the average value of the detection pixel ratio for normal uterine tissue.

[0177] From graphs G1, G2, and G3, it can be seen that the detection pixel ratio is higher in cancer tissue than in normal tissue. Also, from graphs G1 and G2, it can be seen that the detection pixel ratio is higher in invasive cancer than in precancer. As the stage of cancer increases, the detection pixel ratio increases. Here, the detection pixel ratio corresponds to the amount of fibrosis in the superficial part of the uterine tissue. Therefore, as the stage of cancer increases, the amount of fibrosis in the superficial part of the uterine tissue increases.

[0178] Returning to FIG. 26, the precancer / invasive cancer determination process will be continued. Step S730: The precancer / invasive cancer determination unit 192b determines whether the uterine tissue determined to be highly likely to be cancerous tissue is precancer or invasive cancer based on the detection pixel ratio calculated by the fibrosis amount calculation unit 191b. The precancer / invasive cancer determination unit 192b supplies the determination result to the output unit 15 as a determination result Ab.

[0179] Here, the precancer / invasive cancer determination unit 192b determines that the specimen is invasive cancer when the average value of the detected pixel ratio is higher than a predetermined determination threshold. On the other hand, the precancer / invasive cancer determination unit 192b determines that the specimen is precancer when the average value of the detected pixel ratio is lower than a predetermined determination threshold. With the above, the cancer progression determination unit 19b ends the precancer / invasive cancer determination process. In this embodiment, the average value of the detection pixel ratio for the shallow part of the uterine tissue has been used for the explanation, but this is not limited to this, and for example, the maximum value of the detection pixel ratio in the shallow part may be replaced with the average value for the judgment.

[0180] Here, the determination rate of the cancer progress determination unit 19b will be described using a receiver operating characteristic (ROC) curve with reference to Fig. 30 and Fig. 31. Fig. 30 is a diagram showing an example of an ROC curve showing the determination rate of the cancer progress determination unit 19b according to this embodiment. Fig. 31 is a diagram showing an example of the relationship between the determination threshold and the determination rate of the cancer progress determination unit 19b according to this embodiment.

[0181] The ROC curve in Figure 30 shows the detection rate of invasive cancer as a measure of sensitivity, relative to the detection rate of precancer as a measure of specificity. In Figure 30, the evaluation is performed by corresponding invasive cancer to a positive value and precancer to a negative value. In other words, the detection rate of precancer corresponds to the true negative rate, and the detection rate of invasive cancer corresponds to the true positive rate. In the ROC curve in Figure 30, the AUC (Area under the ROC curve) value is 0.88.

[0182] As shown in Fig. 31, when the judgment threshold used by the precancer / invasive cancer judgment unit 192b was changed, a judgment rate of 81.3 percent sensitivity and 80.0 percent specificity was obtained when the judgment threshold was 0.0025. The AUC value of the ROC curve in Fig. 30 is the value when the judgment threshold is 0.0025.

[0183] In this embodiment, the cancer progress determination unit 19b determines whether the cancer progress level is precancer or invasive cancer, but the present invention is not limited to this. The cancer progress determination unit 19b may determine whether the cancer progress level is any of the following: (a) Precancerous; (b) Invasive cancer; (c) Mild dysplasia (CIN1), moderate dysplasia (CIN2), or severe dysplasia / carcinoma in situ (CIN3); (ii) Microinvasive squamous cell carcinoma or squamous cell carcinoma; or (e) Cancer requiring treatment or Cancer not requiring treatment.

[0184] (summary) As described above, the uterine cancer diagnosing device 1b according to this embodiment includes a second harmonic image acquiring section (in this example, an SHG image acquiring section 18b) and a cancer progression determining section 19b. The second harmonic image acquisition section (in this example, the SHG image acquisition section 18b) acquires a second harmonic image (in this example, a Z-stack SHG image ZS1) of the subject's uterine tissue obtained by the multiphoton microscope 2b. When the cancer tissue determination unit 14 determines that the subject's uterine tissue is highly likely to be cancer tissue, the cancer progression determination unit 19b determines the progression of the cancer (in this example, classified into precancer and invasive cancer) based on the state of the fibrous structure in the second harmonic image (in this example, Z-stack SHG image ZS1).

[0185] With this configuration, the uterine cancer diagnosis device 1b according to this embodiment can determine the stage of cancer based on the state of the fibrous structure in the second harmonic image of the shallow part of the uterine tissue of the subject, and can therefore determine the stage of cancer in the uterine tissue without staining the uterine tissue. In other words, the uterine cancer diagnosis device 1b can determine the stage of cancer in the uterine tissue without examining the deep part of the uterine tissue and without staining the uterine tissue.

[0186] Moreover, in the uterine cancer determination device 1b according to this embodiment, the cancer progress determination section 19b determines whether the progress of cancer falls under any of the following conditions. (a) Precancerous; (b) Invasive cancer; (c) Mild dysplasia (CIN1), moderate dysplasia (CIN2), or severe dysplasia / carcinoma in situ (CIN3); (ii) Microinvasive squamous cell carcinoma or squamous cell carcinoma; or (e) Cancer requiring treatment or Cancer not requiring treatment.

[0187] With this configuration, the uterine cancer diagnosis device 1b of this embodiment can determine whether the cancer stage satisfies the above-mentioned items (i) to (e) based on the state of the fibrous structure in the second harmonic image of the shallow part of the subject's uterine tissue, and can therefore determine whether the cancer stage satisfies the above-mentioned items (i) to (e) without staining the uterine tissue.

[0188] (Fourth embodiment) The fourth embodiment will be described in detail below with reference to the drawings. There are two types of uterine cancer: cervical cancer, which is cancer of the tissues of the cervix, and uterine body cancer, which is cancer of the tissues of the uterine body. Cervical cancer progresses from cervical intraepithelial neoplasia, in which cancer develops in the epithelial tissue, to invasive cancer, depending on the degree of progression of the cancer. It is known that fibrosis occurs in the surrounding tissues as the cancer progresses. On the other hand, a technique for identifying cancer tissue by analyzing an image of a biological tissue is known. When identifying cancer tissue by image analysis, an image of the biological tissue that has been stained and captured is used.

[0189] For example, a cancer screening device is known that captures an image of a group of biological cells coated with a dye that selectively stains cancer-related gene products in the biological cells in chromatic colors, and includes a judgment unit that judges the malignancy level of the cancer of the group of biological cells based on the stained state of the group of biological cells in the obtained image (Patent Document 1). Previously, it was difficult to analyze the state of unstained tissue at the cellular level.

[0190] 32 is a diagram showing an example of a uterine cancer progress determination system STc according to the present embodiment. The uterine cancer progress determination system STc includes a cancer progress determination device 1c, a multiphoton microscope 2c, and a display device 3c. In the uterine cancer progress determination system STc, the cancer progress determination device 1c analyzes a Z-stack captured image ZP1c of the subject's uterine tissue captured by the multiphoton microscope 2c to determine the progress of the subject's uterine cancer. In this embodiment, the uterine tissue is, by way of example, tissue of the cervix and tissue of the uterine body.

[0191] The Z-stack captured image ZP1c is a Z-stack image. A Z-stack image is a set of multiple images captured at different distances from the uterine tissue in the Z-axis direction. The Z-stack captured image ZP1c is made up of multiple images of the uterine tissue at each Z coordinate. Here, capturing an image of the uterine tissue at each Z coordinate means capturing an image of the uterine tissue at various distances between the objective lens and the uterine tissue. The Z-stack captured image ZP1c is a set of multiple images captured by the multiphoton microscope 2c at different distances between the uterine tissue and the lens.

[0192] In this embodiment, the Z axis is selected in the direction from the epithelium of the uterine tissue toward the inside. In other words, the closer the imaging plane is to the uterine epithelium, the smaller the Z coordinate value is, and the deeper the imaging plane is into the uterine tissue, the larger the Z coordinate value is. The origin of the Z axis is selected at a shallow position close to the uterine epithelium. In this way, the Z-stack captured image ZP1c is a plurality of cross-sectional images perpendicular to the depth direction of the subject's uterine epithelial tissue, and is also simply called a cross-sectional image. Note that the depth direction is the direction from the surface layer to the basal layer, and the plurality of cross-sectional images do not need to be completely perpendicular to the depth direction of the subject's uterine epithelial tissue, and may be inclined by about ±5 degrees.

[0193] The multiphoton microscope 2c observes and images the subject's uterine tissue in an unstained state. The multiphoton microscope 2c images the uterine tissue using nonlinear optical phenomena. The nonlinear optical phenomena that the multiphoton microscope 2c uses for imaging include second harmonic generation (SHG). SHG is a phenomenon in which light with twice the frequency of the excitation light is generated by interaction with nonlinear optical crystals such as collagen fibers.

[0194] As the tumor progresses, fibrosis occurs in the surrounding tissue. The fibers generated by fibrosis are composed of various collagen-containing molecules. Some fibrous collagen generates SHG. In this embodiment, the multiphoton microscope 2c uses SHG to image the uterine tissue and generate a second harmonic image. The second harmonic image is an image of a cross section of the uterine tissue generated based on the excitation light irradiated from the irradiation unit of the multiphoton microscope 2c and the light generated by second harmonic generation due to the interaction with the uterine tissue. Hereinafter, the second harmonic image is referred to as an SHG image.

[0195] In this embodiment, since the Z-stack captured image ZP1c is a Z-stack captured image of an SHG image, the Z-stack captured image ZP1c will be referred to as a Z-stack SHG image ZS1c below. That is, the Z-stack SHG image ZS1c is a plurality of SHG images of the uterine tissue captured using SHG with the distance from the uterine tissue changed in the Z-axis direction. That is, the Z-stack SHG image ZS1c is a plurality of cross-sectional images captured at a plurality of depths of the uterine epithelial tissue. Here, the cross-sectional images are second harmonic generation images obtained by the multiphoton microscope 2c.

[0196] Furthermore, each of the multiple images included in the Z-stack SHG image ZS1c is referred to as an SHG image PSic (i=1, 2, . . . , N: N is the number of images included in the Z-stack SHG image ZS1c), etc. The multiphoton microscope 2c captures SHG images at each depth of the uterine tissue.

[0197] Here, an SHG image captured by the multiphoton microscope 2c will be described with reference to Fig. 33. Fig. 33 is a diagram showing an example of an SHG image PS0c according to this embodiment. For comparison, Fig. 33 shows a fiber-like structure image F1c together with the SHG image PS0c. It can be seen that the fiber-like structure is captured in the SHG image PS0c captured by the multiphoton microscope 2c to the same extent as that captured in the fiber-like structure image F1c.

[0198] As the tumor progresses, fibrosis spreads from deep to superficial uterine tissue.

[0199] Here, the spread of fibrosis from deep to shallow parts of uterine tissue will be described with reference to Fig. 34. Fig. 34 is a diagram showing an example of cross-sections of uterine tissue for each stage of tumor progression according to this embodiment. In Figure 34A, fibrous structures are present in the deep part of the uterine tissue, while in Figure 34B, a cross-section of the uterine tissue after tumor progression shows fibrosis in the superficial part.

[0200] In this embodiment, the Z axis is also set in a direction from the epithelium of the uterine tissue toward the inside. As the tumor progresses, fibrosis spreads from areas with high Z coordinate values ​​to areas with low Z coordinate values ​​in the uterine tissue. In the shallow part of the uterine tissue, the depth at which fibrosis occurs is unknown in advance because it varies depending on the stage of the tumor. In the cancer progression assessment device 1c, by using the Z-stack SHG image ZS1c, fibrous structures can be extracted regardless of the depth at which fibrosis occurs in the shallow part of the tissue.

[0201] (Configuration of the cancer progression assessment device) Next, the configuration of the cancer progress determination device 1c will be described with reference to Fig. 35. Fig. 35 is a diagram showing an example of the cancer progress determination device 1c according to the present embodiment. The cancer progress determination device 1c is, for example, a computer.

[0202] The cancer progression assessment device 1c includes a cross-sectional image acquisition unit 10c, a fiber pixel extraction unit 11c, a fibrosis amount calculation unit 12c, a cancer progression assessment unit 13c, an output unit 14c, and a storage unit 15c. The cross-sectional image acquisition unit 10c, the fiber pixel extraction unit 11c, the fibrosis amount calculation unit 12c, the cancer progression assessment unit 13c, and the output unit 14c are each a module realized by a CPU (Central Processing Unit) reading a program from a ROM (Read Only Memory) and executing processing.

[0203] The cross-sectional image acquisition unit 10c acquires a Z-stack SHG image ZS1c captured by the multiphoton microscope 2.

[0204] The fiber pixel extracting unit 11c extracts fiber pixels, which are pixels in which a fiber-like structure is captured, from among the pixels of each SHG image PSic included in the Z-stack SHG image ZS1c. Here, in the present embodiment, the fiber pixel extracting unit 11c extracts fiber pixels based on machine learning, as an example. The fiber pixel extracting unit 11c generates a Z-stack fiber-like structure image ZF1c from the Z-stack SHG image ZS1c based on the extracted fiber pixels. The Z-stack fiber-like structure image ZF1c is a Z-stack image in which a fiber-like structure is shown. Each of the multiple images included in the Z-stack fiber-like structure image ZF1c is referred to as a fiber-like structure image PFic (i=1, 2, . . ., N: N is the number of images included in the Z-stack fiber-like structure image ZF1c), etc.

[0205] The fibrous proliferation amount calculation unit 12c calculates the amount of fibrous structure captured in the Z-stack fibrous structure image ZF1c generated by the fiber pixel extraction unit 11c. Hereinafter, the amount of fibrous structure captured in the Z-stack SHG image ZS1c may be referred to as the fibrous proliferation amount. The cancer progression determination unit 13c determines the progression of uterine cancer in the subject based on the amount of fibrosis calculated by the fibrosis amount calculation unit 12c. In this embodiment, the progression of uterine cancer is classified in order of increasing degree of progression, for example, into non-cancer, pre-cancer, or invasive cancer.

[0206] The output unit 14c outputs the determination result Ac of the cancer progress determination unit 13c to the display device 3c. The determination result Ac indicates the progress of uterine cancer in the subject. The display device 3c displays the determination result Ac determined by the cancer progress stage determination device 1c. The display device 3c is, for example, a display.

[0207] The storage unit 15c stores various information. The various information includes, for example, threshold information THc calculated from a case image of a subject. The threshold information THc is information indicating a threshold for the amount of fibrosis, and includes a first threshold TH1c and a second threshold TH2c. The first threshold TH1c indicates the boundary between the amount of precancerous fibrosis and the amount of non-cancerous fibrosis. The second threshold TH2c indicates the boundary between the amount of precancerous fibrosis and the amount of invasive cancer fibrosis. The second threshold TH2c is larger than the first threshold TH1c.

[0208] (Cancer progression assessment process) Next, a cancer stage determination process in which the cancer stage determination device 1c determines the stage of uterine cancer in a subject will be described. Fig. 36 is a diagram showing an example of the cancer stage determination process according to this embodiment.

[0209] Step S810: The cross-sectional image acquiring unit 10c acquires a Z-stack image ZP1c captured by the multiphoton microscope 2c. That is, the cross-sectional image acquiring unit 10c acquires a cross-sectional image of the uterine epithelial tissue of the subject.

[0210] Step S820: The fiber pixel extracting unit 11c extracts fiber pixels, which are pixels in which a fiber-like structure is captured, from the pixels of each SHG image PSic included in the Z-stack SHG image ZS1c. The fiber pixel extracting unit 11c generates a Z-stack fiber-like structure image ZF1c from the Z-stack SHG image ZS1c based on the extraction result. The fiber pixel extracting unit 11c supplies the generated Z-stack fiber-like structure image ZF1c to the fiber augmentation amount calculating unit 12c.

[0211] Here, the fiber pixel extraction unit 11c judges whether or not a fiber-like structure is imaged in each pixel based on a predetermined criterion, and extracts fiber pixels based on the judgment result. As an example, the fiber pixel extraction unit 11c performs the judgment based on machine learning. That is, the fiber pixel extraction unit 11c uses the criterion generated by machine learning as the predetermined criterion for judging a fiber-like structure.

[0212] The fiber pixel extraction unit 11c uses, for example, Conditional GAN ​​(GAN: Generative Adversarial Networks) as machine learning. Conditional GAN ​​is composed of two networks: a network called a generation unit and a network called a discrimination unit. The generation unit learns to generate fake images that cannot be detected by the discrimination unit. On the other hand, the discrimination unit learns a discriminator that can detect fake images generated by the generation unit. In this embodiment, the criterion generated by machine learning is that the identification unit of the Conditional GAN ​​determines that a fiber-like structure is imaged in a pixel. Note that the machine learning used by the fiber pixel extraction unit 11c may be other than Conditional GAN.

[0213] Here, a fiber-like structure image PF1c included in the Z-stack fiber-like structure image ZF1c generated by the fiber pixel extraction unit 11c will be described with reference to Fig. 37. Fig. 37 is a diagram showing an example of the fiber-like structure image PF1c according to the present embodiment. The fiber-like structure image PF1c is an image in which the fiber-like structure is determined based on the SHG image PS1c included in the Z-stack SHG image ZS1c. The SHG image PS1c is, for example, an SHG image obtained by capturing tissue at a depth of 15 μm from the epithelial tissue of the uterine tissue toward the inside.

[0214] As described above, in this embodiment, the fiber pixel extraction unit 11c extracts fiber pixels based on the Z-stack captured image ZP1c acquired by the cross-sectional image acquisition unit 10c and a criterion generated by machine learning.

[0215] In the present embodiment, an example has been described in which the fiber pixel extracting unit 11c uses machine learning to determine whether a fiber-like structure is present in the Z-stack SHG image ZS1c, but the present invention is not limited to this. The fiber pixel extracting unit 11c may determine whether a fiber-like structure is present in a pixel based on a predetermined criterion, for example, whether the pixel is included in a region of continuous pixels having a shape that is equal to or smaller than a predetermined width and equal to or larger than a predetermined length. The Z-stack SHG image ZS1c may be configured without including an image classified as non-cancerous.

[0216] In other words, the fiber pixel extraction unit 11c extracts fiber pixels, which are pixels in which fiber-like structures are imaged, from the pixels of the Z stack image ZP1c acquired by the cross-sectional image acquisition unit 10c based on a predetermined criterion and the Z stack image ZP1c acquired by the cross-sectional image acquisition unit 10c.

[0217] In addition, in the present embodiment, an example of a case where fiber pixels are determined by the fiber pixel extracting unit 11c has been described, but the present invention is not limited to this. The fiber pixels may be determined by a user of the cancer progress assessment device 1c. When the fiber pixels are determined by a user of the cancer progress assessment device 1c, the cancer progress assessment device 1c includes an operation input unit, and the fiber pixel extracting unit 11c receives, for example, an operation for determining a fiber-like structure in the Z-stack SHG image ZS1c via the operation input unit. The fiber pixel extracting unit 11c generates a Z-stack fiber-like structure image ZF1c based on the received operation for determining a fiber-like structure.

[0218] Returning to FIG. 36, the cancer progression determination process will be continued. Step S830: The fibrous increase amount calculation unit 12c calculates the amount of fibrous increase in the shallow part of the uterine tissue captured as the Z-stack SHG image ZS1c based on the Z-stack fibrous structure image ZF1c generated by the fibrous pixel extraction unit 11c. Here, the Z-stack fibrous structure image ZF1c generated by the fibrous pixel extraction unit 11c is a Z-stack image in which a fibrous structure is shown in the Z-stack SHG image ZS1c acquired by the cross-sectional image acquisition unit 10c. That is, the fibrous increase amount calculation unit 12c calculates the amount of fibrous structure captured in the Z-stack SHG image ZS1c acquired by the cross-sectional image acquisition unit 10c. The fibrous increase amount calculation unit 12c supplies the calculated amount of fibrous increase to the cancer progression determination unit 13c.

[0219] Here, the fibrous augmentation amount calculation unit 12c calculates the amount of fibrous structure (fibrous augmentation amount) captured in the Z-stack SHG image ZS1c based on the fiber pixels extracted by the fiber pixel extraction unit 11c. The fibrous augmentation amount calculation unit 12c calculates the ratio of the area of ​​the portion in which the fibrous structure is captured to the entire area of ​​the Z-stack fibrous structure image ZFic as the amount of fibrous augmentation. The fibrous augmentation amount calculation unit 12c calculates these areas based on the number of pixels. In other words, the fibrous augmentation amount calculation unit 12c calculates the ratio of the number of pixels of the portion in which the fibrous structure is captured to the entire number of pixels of the Z-stack fibrous structure image ZFic as the amount of fibrous augmentation. Hereinafter, this ratio will be referred to as a detection pixel ratio.

[0220] In this embodiment, an example has been described in which the fibrosis amount calculation unit 12c calculates the amount of fibrosis as the ratio of the area of ​​the portion in which the fibrous structure is imaged to the total area of ​​the Z-stack fibrous structure image ZFic, but this is not limited to this. The fibrosis amount calculation unit 12c may calculate the area of ​​the fibrous structure captured in the Z-stack fibrous structure image ZFic as the amount of fibrosis, or may calculate the length or number of the fibrous structure as the amount of fibrosis. In addition, the fibrosis amount calculation unit 12c may apply a predetermined pre-processing to the Z-stack fibrous structure image ZFic in advance in order to calculate the amount of fibrosis. An example of the predetermined pre-processing is a binarization process. This makes it possible to more reliably calculate the amount of the fibrous structure captured in the Z-stack SHG image ZS1c. In addition, the pre-processing is not limited to the binarization process, and may include a smoothing process and a morphology process.

[0221] Furthermore, the fibrosis amount calculation unit 12c may calculate, as the amount of fibrosis, a total amount or an average amount of fibrous structures captured in a plurality of Z-stack fibrous structure images ZFic included in the Z-stack SHG image ZS1c, or a ratio of the number of pixels of the fibrous structures to the total number of pixels of the cross-sectional image. When the total amount or average amount of fibrous structures captured in a plurality of Z-stack fibrous structure images ZFic, or a ratio of the number of pixels of the fibrous structures to the total number of pixels of the cross-sectional image, is calculated, it is possible to prevent a decrease in accuracy of determining the stage of uterine cancer when there is variation in the amount of fibrosis between the SHG images ZSic included in the Z-stack SHG image ZS1c.

[0222] Here, the detection pixel ratio calculated by the fibrosis increase amount calculation unit 12c will be described with reference to Fig. 38 and Fig. 39. Fig. 38 is a diagram showing an example of the detection pixel ratio according to this embodiment. In Fig. 38, the detection pixel ratio is shown with respect to the Z axis of the Z stack image for 16 specimens of invasive cancerous uterine tissue, 3 specimens of precancerous uterine tissue, and 13 specimens of normal uterine tissue. The larger the Z-axis coordinate value, the deeper the imaged uterine tissue is from the epithelium. The Z-coordinate value of the surface of the epithelium is 0. The Z-coordinate value of the boundary between the shallow and deep uterine tissue is 50.

[0223] FIG. 39 is a diagram showing an example of the average value of the detection pixel ratio for the shallow part of the uterine tissue according to this embodiment. The average value for the shallow part of the uterine tissue is the average for the Z-axis value of the Z-stack image. The three graphs shown in FIG. 39 show the average value of the detection pixel ratio for the shallow part of the uterine tissue for the number indicating each specimen. Graph G1c is the average value of the detection pixel ratio for invasive cancer uterine tissue. Graph G2c is the average value of the detection pixel ratio for precancerous uterine tissue. Graph G3c is the average value of the detection pixel ratio for normal uterine tissue.

[0224] From graphs G1c, G2c, and G3c, it can be seen that the detection pixel ratio is higher in cancer tissue than in normal tissue. Also, from graphs G1c and G2c, it can be seen that the detection pixel ratio is higher in invasive cancer than in precancer. As the stage of cancer increases, the detection pixel ratio increases. Here, the detection pixel ratio corresponds to the amount of fibrosis in the superficial part of the uterine tissue. Therefore, as the stage of cancer increases, the amount of fibrosis in the superficial part of the uterine tissue increases.

[0225] Returning to FIG. 36, the cancer progression determination process will be continued. Step S840: The cancer progress determination unit 13c determines the progress of uterine cancer of the subject based on the detection pixel ratio calculated by the fibrosis amount calculation unit 12c and the first threshold TH1c and the second threshold TH2c indicated by the threshold information THc. Here, the cancer progress determination unit 13c reads out the threshold information THc from the storage unit 15c. The cancer progress determination unit 13c supplies the result of the determination as a determination result Ac to the output unit 14c.

[0226] Here, when the amount of fibrous structure is equal to or less than the first threshold TH1c, the cancer progress determination unit 13c determines that the uterine epithelial tissue is non-cancerous tissue, whereas when the amount of fibrous structure is greater than the first threshold TH1c, the cancer progress determination unit 13c determines that the subject has uterine cancer or is highly likely to have uterine cancer. When the amount of fibrous structures exceeds the first threshold TH1c and is equal to or smaller than the second threshold TH2c that is greater than the first threshold TH1c, the cancer progress determination unit 13c determines that the subject's uterine cancer is in a precancerous state.When the amount of fibrous structures exceeds the second threshold TH2c, the cancer progress determination unit 13c determines that the subject's uterine cancer is invasive cancer.

[0227] Step S850: The output unit 14c outputs the determination result Ac to the display device 3c. With the above, the cancer progress determination device 1c ends the cancer progress determination process.

[0228] Here, the determination rate of the cancer progress determination unit 13c will be described using a receiver operating characteristic (ROC) curve with reference to Fig. 40 and Fig. 41. Fig. 40 is a diagram showing an example of an ROC curve showing the determination rate of the cancer progress determination unit 13c according to this embodiment. Fig. 41 is a diagram showing an example of the relationship between the determination threshold and the determination rate of the cancer progress determination unit 13c according to this embodiment.

[0229] The ROC curve in Figure 40 shows the detection rate of invasive cancer as a measure of sensitivity, relative to the detection rate of precancer as a measure of specificity. In Figure 40, the evaluation is performed by corresponding invasive cancer to a positive value and precancer to a negative value. In other words, the detection rate of precancer corresponds to the true negative rate, and the detection rate of invasive cancer corresponds to the true positive rate. In the ROC curve in Figure 40, the AUC (Area under an ROC curve) value is 0.88.

[0230] As shown in Fig. 41, when the judgment threshold used by the cancer progression judgment unit 13c was changed, a judgment rate of 81.3 percent sensitivity and 80.0 percent specificity was obtained when the judgment threshold was 0.0025. The AUC value of the ROC curve in Fig. 40 is the value when the judgment threshold is 0.0025.

[0231] In this embodiment, the cancer progression determination unit 13c determines whether the cancer progression is precancer or invasive cancer, but the present invention is not limited to this. The cancer progression determination unit 13c may determine whether the cancer progression is any of the following: (a) Precancerous; (b) Invasive cancer; (c) Mild dysplasia (CIN1), moderate dysplasia (CIN2), or severe dysplasia / carcinoma in situ (CIN3); (ii) Microinvasive squamous cell carcinoma or squamous cell carcinoma; or (e) Cancer requiring treatment or Cancer not requiring treatment.

[0232] As described above, the cancer progression assessment device 1c according to this embodiment includes the cross-sectional image acquisition unit 10c, a calculation unit (in this example, the fibrosis amount calculation unit 12c), and a cancer progression assessment unit 13c. The cross-sectional image acquisition unit 10c acquires a cross-sectional image (in this example, an SHG image PSic) of the uterine epithelial tissue of the subject. The calculation unit (in this example, the fibrosis amount calculation unit 12c) calculates the amount of fibrous structures captured in the cross-sectional image (in this example, the SHG image PSic) acquired by the cross-sectional image acquisition unit 10c. The cancer progression determination unit 13c determines the progression of the subject's uterine cancer (in this example, non-cancer, pre-cancer, or invasive cancer) based on the amount of fibrous structures calculated by the calculation unit (in this example, the fibrosis amount calculation unit 12c).

[0233] With this configuration, the cancer progression determination device 1c of this embodiment can determine the progression of a subject's uterine cancer (in this example, non-cancer, pre-cancer, or invasive cancer) based on the amount of fibrous structures captured in a cross-sectional image of the subject's uterine epithelial tissue (in this example, SHG image PSic), and can therefore determine the progression of uterine cancer (in this example, non-cancer, pre-cancer, or invasive cancer) without staining the uterine tissue.

[0234] Moreover, the cancer progression determination device 1c according to this embodiment includes an extraction unit (in this example, a fiber pixel extraction unit 11c). The extraction unit (in this example, fiber pixel extraction unit 11c) extracts fiber pixels, which are pixels in which fiber-like structures are imaged, from the pixels of the cross-sectional image (in this example, SHG image PSic) acquired by the cross-sectional image acquisition unit 10c, based on a predetermined criterion (in this example, criterion generated by machine learning). In addition, the calculation unit (in this example, the fibrous proliferation amount calculation unit 12c) calculates the amount of fibrous structure imaged in the cross-sectional image (in this example, the SHG image PSic) based on the fiber pixels extracted by the extraction unit (in this example, the fiber pixel extraction unit 11c).

[0235] With this configuration, the cancer progression assessment device 1c of this embodiment can extract fiber pixels, which are pixels in which fiber-like structures are captured, from the pixels of a cross-sectional image (in this example, SHG image PSic), thereby reducing the effort required to extract fiber pixels from the pixels of a cross-sectional image (in this example, SHG image PSic).

[0236] In addition, in the cancer progression determination device 1c of this embodiment, a calculation unit (in this example, a fibrosis amount calculation unit 12c) calculates the area of ​​the fibrous structure captured in the cross-sectional image (in this example, an SHG image PSic) as the amount of the fibrous structure. With this configuration, the cancer progression assessment device 1c of this embodiment can calculate the area of ​​the fibrous structure captured in the cross-sectional image (in this example, the SHG image PSic) as the amount of fibrous structure, and can therefore assess the progression of uterine cancer (in this example, non-cancerous, pre-cancerous, or invasive cancer) based on the area of ​​the fibrous structure captured in the cross-sectional image (in this example, the SHG image PSic) without staining the uterine tissue.

[0237] Moreover, in the cancer progress assessment device 1c according to this embodiment, the cross-sectional image (in this example, the SHG image PSic) is a second harmonic generation image obtained by the multiphoton microscope 2c. With this configuration, the cancer progression assessment device 1c of this embodiment can use a cross-sectional image (in this example, an SHG image PSic) of fibrous structures that arise as a tumor progresses, captured using second harmonic generation, and can therefore assess the progression of uterine cancer (in this example, non-cancerous, pre-cancerous, or invasive cancer) using the fibrous structures captured based on second harmonic generation without staining the uterine tissue. This can be determined without staining uterine tissue.

[0238] Furthermore, in the cancer progression assessment device 1c according to this embodiment, the cross-sectional image (in this example, the SHG image PSic) is a plurality of cross-sectional images (in this example, the Z-stack SHG image ZS1c) captured at a plurality of depths of the subject's uterine epithelial tissue, and the calculation unit (in this example, the fibrosis amount calculation unit 12c) calculates the total amount, average amount, or ratio of the fibrous structures captured in the plurality of cross-sectional images (in this example, the Z-stack SHG image ZS1c) to the number of pixels of the cross-sectional image.

[0239] With this configuration, the cancer progression assessment device 1c of this embodiment can assess the progression of uterine cancer (in this example, non-cancer, pre-cancer, or invasive cancer) based on the total amount, average amount, or ratio of fibrous structures to the number of pixels in the cross-sectional images (in this example, Z-stack SHG image ZS1c) of fibrous structures captured in multiple cross-sectional images, thereby preventing a decrease in the accuracy of assessing the progression of uterine cancer (in this example, non-cancer, pre-cancer, or invasive cancer) when there is variation in the amount of fibrous structures between multiple cross-sectional images (in this example, Z-stack SHG image ZS1c).

[0240] Moreover, in the cancer progress determination device 1c according to this embodiment, the cancer progress determination section 13c determines whether the progress of uterine cancer falls under any of the following conditions. (a) Precancerous; (b) Invasive cancer; (c) Mild dysplasia (CIN1), moderate dysplasia (CIN2), or severe dysplasia / carcinoma in situ (CIN3); (ii) Microinvasive squamous cell carcinoma or squamous cell carcinoma; or (e) Cancer requiring treatment or Cancer not requiring treatment.

[0241] With this configuration, the cancer progression determination device 1c of this embodiment can determine whether the progression of uterine cancer satisfies the above-mentioned items (i) to (e) based on the amount of fibrous structures in a cross-sectional image (in this example, an SHG image PSic), and can therefore determine whether the progression of uterine cancer satisfies the above-mentioned items (i) to (e) without staining the uterine tissue.

[0242] In addition, in the cancer progression determination device 1c of this embodiment, the cancer progression determination unit 13c determines that the uterine epithelial tissue is non-cancerous tissue if the amount of fibrous structures is equal to or less than the first threshold value TH1c, and determines that the subject is suffering from uterine cancer or is likely to be suffering from uterine cancer if the amount of fibrous structures exceeds the first threshold value TH1c. With this configuration, the cancer progression determination device 1c of this embodiment can determine the progression of uterine cancer as non-cancerous tissue, uterine cancer, or high possibility of uterine cancer, based on the amount of fibrous structures in a cross-sectional image (in this example, an SHG image PSic).

[0243] Furthermore, in the cancer progression assessment device 1c of this embodiment, the cross-sectional image is an image of precancer or invasive cancer, and as the progression of uterine cancer, the cancer progression assessment unit 13c assesses that the subject has precancer if the amount of fibrous structure exceeds a first threshold value TH1c and is equal to or smaller than a second threshold value TH2c that is greater than the first threshold value TH1c, and assesses that the subject has invasive cancer if the amount of fibrous structure exceeds the second threshold value TH2c. With this configuration, the cancer progression determination device 1c of this embodiment can determine the progression of uterine cancer, whether the subject has precancer or invasive cancer, based on the amount of fibrous structures in a cross-sectional image (in this example, an SHG image PSic). In addition, in the cancer progression assessment device 1c of this embodiment, the cross-sectional image is a cross-sectional image of precancer or invasive cancer, and the cancer progression assessment unit 13c determines that the stage of the subject's uterine cancer is precancer if the amount of fibrous structures is smaller than the first threshold value TH1c when the stage of the subject's uterine cancer is precancer or invasive cancer, and determines that the stage of the subject's uterine cancer is invasive cancer if the amount of fibrous structures exceeds the first threshold value TH1c. With this configuration, in the cancer progression determination device of this embodiment, the cross-sectional image (Z-stack SHG image ZS1c) is a cross-sectional image of precancer or invasive cancer, and by determining from the image that the subject's uterine epithelial tissue is precancer or invasive cancer, precancer can be accurately determined.

[0244] In addition, a part of the uterine cancer determination device 1, uterine cancer determination device 1a, uterine cancer determination device 1b, and cancer stage determination device 1c in the above-mentioned embodiment, for example, the nuclear region image generation unit 10, the feature amount processing unit 11, the data division unit 12, the classification model generation unit 13, the cancer tissue determination unit 14, the output unit 15, the operation input unit 16, the nuclear region image generation unit 10a, the SHG image acquisition unit 18b, the cancer stage determination unit 19b, the cross-sectional image acquisition unit 10c, the fiber pixel extraction unit 11c, the fibrosis amount calculation unit 12c, the cancer stage determination unit 13c, and the output unit 14c may be realized by a computer. In this case, a program for realizing this control function may be recorded in a computer-readable recording medium, and the program recorded in the recording medium may be read into and executed by a computer system. In addition, the "computer system" referred to here is a computer system built into the uterine cancer determination device 1, uterine cancer determination device 1a, uterine cancer determination device 1b, and cancer stage determination device 1c, and includes hardware such as an OS and peripheral devices. In addition, "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, and storage devices such as hard disks built into computer systems. Furthermore, "computer-readable recording medium" may also include devices that dynamically hold a program for a short period of time, such as a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line, and devices that hold a program for a certain period of time, such as volatile memory inside a computer system that serves as a server or client in such cases. Furthermore, the above program may be one that realizes part of the above-mentioned functions, or may be one that can realize the above-mentioned functions in combination with a program already recorded in the computer system. In addition, the uterine cancer diagnosis device 1, the uterine cancer diagnosis device 1a, the uterine cancer diagnosis device 1b, and the cancer progression assessment device 1c in the above-mentioned embodiments may be realized as an integrated circuit such as LSI (Large Scale Integration). Each functional block of the uterine cancer diagnosis device 1, the uterine cancer diagnosis device 1a, the uterine cancer diagnosis device 1b, and the cancer progression assessment device 1c may be individually processed, or may be integrated into a processor in part or in whole. The integrated circuit method is not limited to LSI, and may be realized by a dedicated circuit or a general-purpose processor. In addition, when an integrated circuit technology that replaces LSI appears due to the progress of semiconductor technology, an integrated circuit using this technology may be used.

[0245] Although one embodiment of the present invention has been described in detail above with reference to the drawings, the specific configuration is not limited to the above, and various design changes, etc. are possible within the scope that does not deviate from the gist of the present invention. [Explanation of symbols]

[0246] 1, 1a, 1b...uterine cancer diagnosis device, 100...THG image acquisition unit, 14...cancer tissue diagnosis unit, ZT1...Z-stack THG image, 1c...cancer progression diagnosis device, 10c...cross-sectional image acquisition unit, 11c...fiber pixel extraction unit, 12c...fibrous proliferation amount calculation unit, 13c...cancer progression diagnosis unit, PSic...SHG image, ZS1c...Z-stack SHG image

Claims

1. an irradiation unit that irradiates excitation light onto unstained uterine tissue; a third harmonic image acquisition unit that acquires a third harmonic image of the uterine tissue based on light generated by third harmonic generation caused by an interaction between the uterine tissue and the excitation light; a cancer tissue determination unit that determines a possibility that the uterine tissue is cancer tissue based on a state of the cell nucleus in the third harmonic image; a second harmonic image acquisition unit that acquires a second harmonic image based on light generated by second harmonic generation caused by an interaction between the uterine tissue and the excitation light; a cancer progression determination unit that determines a progression level of cancer based on a state of a fibrous structure in the second harmonic image when the cancer tissue determination unit determines that the uterine tissue is highly likely to be cancer tissue; Equipped with the second harmonic image and the third harmonic image are a plurality of cross-sectional images captured at different depths in a direction from an epithelial tissue of the uterine tissue toward the inside, the depth is common to the second harmonic image and the third harmonic image; the cancer tissue determination unit performs a first determination of whether the uterine tissue is cancer tissue or normal tissue based on a feature amount of the cell nucleus corresponding to the depth from the third harmonic image; the cancer progression assessment unit performs a second assessment on the uterine tissue determined to be cancerous tissue in the first assessment, in which the uterine tissue is determined to be precancer or invasive cancer based on the amount of the fibrous structure corresponding to the depth from the second harmonic image. Image processing device.

2. The imaging device of claim 1 , wherein the tissue is pre-exposed to acetic acid.

3. The image processing device according to claim 1 , wherein the state of the cell nucleus is at least one selected from the group consisting of an area of ​​the cell nucleus, a density of the cell nucleus, and a shape of the cell nucleus.

4. The image processing device of claim 1, wherein the cancer tissue determination unit determines that the tissue is highly likely to be cancer tissue if at least one of the following is true: (i) the average area of ​​cell nuclei is increased compared to normal tissue; (ii) the variation in the area of ​​cell nuclei is increased compared to normal tissue; (iii) the density of cell nuclei is increased compared to normal tissue; (iv) the variation in the density of cell nuclei is increased compared to normal tissue; (v) the irregularity of the shape of the cell nuclei is increased compared to normal tissue; and (vi) the variation in the shape of the cell nuclei is increased compared to normal tissue.

5. 4. The image processing device according to claim 1, wherein the cancer tissue determination unit refers to a classification model trained using a learning third harmonic image of normal tissue and / or a learning third harmonic image of cancer tissue, and determines the possibility that the tissue is cancer tissue based on the state of cell nuclei in the third harmonic image.

6. The image processing apparatus according to claim 5 , wherein the learning third harmonic images are a plurality of cross-sectional images of epithelial tissue.

7. The image processing apparatus according to claim 5 , wherein the third harmonic images are a plurality of cross-sectional images of epithelial tissue.

8. 8. The image processing device according to claim 7, wherein the cancer tissue determination unit determines a possibility that each of the plurality of third harmonic images is cancer tissue, and calculates a possibility that the tissue is cancer tissue based on a ratio of the third harmonic images determined to have a high possibility of being cancer tissue to all of the third harmonic images.

9. The image processing device according to claim 1 , wherein the cancer progression determination unit determines whether the cancer progression level falls under any of the following: (a) It is precancerous; (b) Invasive cancer; (c) mild dysplasia (CIN1), moderate dysplasia (CIN2), or severe dysplasia / carcinoma in situ (CIN3); (ii) Microinvasive squamous cell carcinoma or squamous cell carcinoma; or (e) Cancer requiring treatment or Cancer not requiring treatment.

10. A calculation unit that calculates the amount of a fibrous structure imaged in the second harmonic image acquired by the second harmonic image acquisition unit, The state of the fibrous structure is determined based on the amount of the fibrous structure calculated by the calculation unit. The image processing device according to claim 1 .

11. An extraction unit is further provided that extracts fiber pixels, which are pixels in which a fiber-like structure is imaged, from the pixels of the second harmonic image based on the second harmonic image and a predetermined criterion, The calculation unit calculates the amount based on the fiber pixels extracted by the extraction unit. The image processing device according to claim 10.

12. The calculation unit calculates an area of ​​the fibrous structure imaged in the second harmonic image as the amount. The image processing device according to claim 10 or 11.

13. The calculation unit calculates a total amount, an average amount, or a ratio of the number of pixels of the fiber-like structures to the total number of pixels of the cross-sectional images. The image processing device according to any one of claims 10 to 12.

14. The cancer progression assessment unit determines the stage of cancer as follows: When the amount of the fibrous structure exceeds a first threshold and is equal to or less than a second threshold that is greater than the first threshold, the tumor is determined to be precancerous; If the second threshold is exceeded, the cancer is determined to be invasive cancer. The image processing device according to any one of claims 9 to 13.

15. The second harmonic image is a cross-sectional image of a pre-cancer or invasive cancer, The cancer progression assessment unit determines the stage of cancer as follows: If the amount of the fibrous structure is equal to or less than a first threshold, the tumor is determined to be precancerous; The image processing device according to claim 10 , wherein, when the amount of the fibrous structure exceeds a first threshold, the cancer is determined to be invasive cancer.

16. An irradiation unit, A third harmonic image acquisition unit; A cancer tissue determination unit; A second harmonic image acquisition unit; A cancer progression assessment unit; A method for operating an image processing device comprising: an irradiation step in which the irradiation unit irradiates excitation light onto unstained uterine tissue; a third harmonic image acquiring step in which the third harmonic image acquiring unit acquires a third harmonic image of the uterine tissue based on light generated by third harmonic generation caused by an interaction between the uterine tissue and the excitation light; a cancer tissue determination step in which the cancer tissue determination unit determines a possibility that the uterine tissue is cancer tissue based on a state of a cell nucleus in the third harmonic image; a second harmonic image acquiring step in which the second harmonic image acquiring unit acquires a second harmonic image based on light generated by second harmonic generation caused by an interaction between the uterine tissue and the excitation light; a cancer progression determination step in which, when the cancer tissue determination step determines that the uterine tissue is highly likely to be cancer tissue, the cancer progression determination unit determines a stage of cancer based on a state of a fibrous structure in the second harmonic image; having the second harmonic image and the third harmonic image are a plurality of cross-sectional images captured at different depths in a direction from an epithelial tissue of the uterine tissue toward the inside, the depth is common to the second harmonic image and the third harmonic image; In the cancer tissue determination step, the cancer tissue determination unit performs a first determination of whether the uterine tissue is cancer tissue or normal tissue based on a feature amount of the cell nucleus corresponding to the depth from the third harmonic image, In the cancer progression determination process, the cancer progression determination unit performs a second determination on the uterine tissue determined to be cancer tissue in the first determination, determining whether the uterine tissue is a precancer or an invasive cancer based on the amount of the fibrous structure corresponding to the depth from the second harmonic image. A method for operating an image processing device.

17. On the computer, an irradiation step of irradiating an unstained uterine tissue with excitation light; a third harmonic image acquisition step of acquiring a third harmonic image of the uterine tissue based on light generated by third harmonic generation caused by an interaction between the uterine tissue and the excitation light; a cancer tissue determination step of determining a possibility that the uterine tissue is cancer tissue based on a state of the cell nucleus in the third harmonic image; a second harmonic image acquisition step of acquiring a second harmonic image based on light generated by second harmonic generation caused by an interaction between the uterine tissue and the excitation light; a cancer progression determination step of determining a stage of cancer based on a state of a fibrous structure in the second harmonic image when the cancer tissue determination step determines that the uterine tissue is highly likely to be cancer tissue; A program for executing the second harmonic image and the third harmonic image are a plurality of cross-sectional images captured at different depths in a direction from an epithelial tissue of the uterine tissue toward the inside, the depth is common to the second harmonic image and the third harmonic image; the cancer tissue determination step performs a first determination of whether the uterine tissue is cancer tissue or normal tissue based on a feature amount of the cell nucleus corresponding to the depth from the third harmonic image; The cancer progression assessment step includes performing a second assessment on the uterine tissue determined to be cancer tissue in the first assessment, in which the uterine tissue is assessed to be either precancer or invasive cancer based on the amount of the fibrous structure according to the depth from the second harmonic image. program.

Images