Learning model generation device, learning model generation method, and program

Abstract

To provide a technique capable of generating a highly accurate CD map on the basis of a two-dimensional fundus image.SOLUTION: A generation method of a learning model includes a step of machine-learning a first learning model 510 using teacher data. The first learning model 510 is the learning model that uses a two-dimensional fundus image 210 as an input and a CD map 220 indicating the planar distribution state about a cup C (optic papilla) and a disk D (optic papilla recess) as an output. The first learning model 510 is learned by using the teacher data applied with a CD map 260 based on an analysis result by a fundus three-dimensional image analysis device as a correct answer label.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to an image processing device for a fundus image and the related technology. [Background technology]

[0002] An indicator for diagnosing glaucoma, an eye disease, is the ratio of the optic disc cup to the optic disc (disc) in a two-dimensional fundus image (also simply referred to as a fundus image), known as the cup-to-dicellar diameter ratio (hereinafter also referred to as the C / D ratio). In a fundus image, the cup portion (optic disc cup) with reduced thickness appears as a bright area. If the ratio of such an area (cup C) to the disc D (C / D ratio (particularly the vertical C / D ratio)) is greater than a predetermined value (for example, 0.6), there is a high possibility of glaucoma.

[0003] This C / D ratio can be measured with high accuracy using a three-dimensional fundus image analyzer (such as an OCT (Optical Coherence Tomography) device). A three-dimensional fundus image analyzer is capable of measuring the retinal nerve fiber layer in three dimensions, including the thickness of the retinal nerve fiber layer. This three-dimensional measurement method makes it possible to accurately identify the optic disc (disk D) and the optic disc cup (cup C). As a result, it is possible to measure the C / D ratio more accurately. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] K. Park et al., Automatic optic nerve head localization and cup-to-disc ratio detection using state-of-the-art deep-learning architectures, Scientific Reports volume 10, Article number: 5025 (2020) Summary of the Invention [Problem to be solved by the invention]

[0005] Incidentally, there is a need among doctors to view a map (hereinafter also referred to as a CD map) showing the distribution of cups C and disks D (planar distribution) in a fundus image. To meet this need, it is desirable to be able to generate a CD map relatively easily.

[0006] Among the above-mentioned three-dimensional fundus image analyzers, there are some that display CD maps.

[0007] However, 3D fundus image analysis devices are very expensive, making it practically impossible to introduce them into all hospitals.

[0008] In response to this, the inventors of the present application have devised a technique for generating a CD map using a relatively inexpensive two-dimensional fundus image. More specifically, the inventors have devised a technique for machine learning a learning model based on training data in which a person manually identifies a cup C and a disk D in a two-dimensional fundus image and the CD map based on each identified region is used as the correct label.

[0009] However, two-dimensional fundus images basically only have planar information (do not have accurate thickness information). It is not easy for a person to accurately identify the disk D and the cup C from such a planar two-dimensional fundus image alone. More specifically, it is not easy for a person to accurately identify the range of the disk D and the cup C (especially the cup C) based on subtle differences in brightness, etc., and the accuracy of identifying the cup C and the disk D (especially the cup C) is not necessarily sufficient. Furthermore, when a person performs the task of identifying such a disk D and the cup C, individual differences (variations between individuals) occur. As a result, there is a problem that the accuracy of the correct label itself, and ultimately the accuracy of the learning model (trained model), is reduced.

[0010] Therefore, an object of the present invention is to provide a technique that can generate a highly accurate CD map based on a two-dimensional fundus image.

[0011] Non-Patent Document 1 discloses a machine learning technology for machine learning a learning model that uses a two-dimensional fundus image as an input and a C / D ratio as an output. More specifically, the technology discloses a machine learning technology for machine learning the learning model using training data in which the C / D ratio from an OCT image is used as a correct label. However, the technology in Non-Patent Document 1 does not require a CD map. [Means for solving the problem]

[0012] In order to solve the above problem, the method for generating a learning model according to the present invention includes the steps of: (a) machine learning a first learning model using a two-dimensional fundus image as an input and a CD map showing a planar distribution state of cups and discs as an output. and b) machine learning a second learning model that uses both the two-dimensional fundus image and a CD map related to the two-dimensional fundus image output from the first learning model as inputs and outputs a C / D ratio, In the step a), the first learning model is machine-trained using training data to which a CD map based on the analysis result by the fundus three-dimensional image analyzer is attached as a correct answer label; In the step b), the second learning model is machine-learned using training data to which the C / D ratio, which is the analysis result of the fundus three-dimensional image analysis device, is attached as a correct answer label. In step b), an integrated learning model connecting the first learning model and the second learning model may be machine-learned using training data to which the C / D ratio as the analysis result by the fundus 3D image analysis device is attached as a correct answer label, thereby machine-learning not only the second learning model but also the first learning model.

[0013] In order to solve the above problem, the learning model generation device of the present invention comprises a control unit that machine-learns a first learning model that takes a two-dimensional fundus image as input and outputs a CD map showing the planar distribution of cups and discs, and also machine-learns a second learning model that takes both the two-dimensional fundus image and a CD map related to the two-dimensional fundus image that is output from the first learning model as input and outputs a C / D ratio, and the control unit machine-learns the first learning model using training data to which a CD map based on the analysis results by a three-dimensional fundus image analysis device is attached as a correct answer label, and machine-learns the second learning model using training data to which the C / D ratio as the analysis result by the three-dimensional fundus image analysis device is attached as a correct answer label. The control unit may machine-learn not only the second learning model but also the first learning model by using training data to which the C / D ratio as the analysis result by the fundus 3D image analysis device is attached as a correct answer label to machine-learn an integrated learning model that connects the first learning model and the second learning model.

[0014] In order to solve the above problem, the method for generating a learning model according to the present invention comprises: a) a step of machine learning an integrated learning model comprising a first learning model and a second learning model based on a plurality of training data that uses analysis information from a plurality of three-dimensional fundus image analysis devices as a correct label, wherein the first learning model is a learning model that receives a two-dimensional fundus image as an input and outputs a CD map showing a planar distribution state of cups and discs, and the second learning model is a learning model that receives both the two-dimensional fundus image and a CD map related to the two-dimensional fundus image as an input and outputs a C / D ratio, and the step a) comprises: a-1) when it is possible to obtain both the CD map based on the analysis result of a first device out of the plurality of three-dimensional fundus image analysis devices and the C / D ratio based on the analysis result of the first device, performing supervised learning of the first learning model using first training data that sets the CD map based on the analysis result of the first device as a correct label, and setting the C / D ratio based on the analysis result of the first device as a correct label; a-2) if a CD map based on the analysis results of a second device among the plurality of fundus three-dimensional image analysis devices can be obtained but a C / D ratio based on the analysis results cannot be obtained, a substep of not performing supervised learning of the second learning model and performing supervised learning of the first learning model using first teacher data in which the CD map based on the analysis results of the second device is used as a correct label; and a-3) if a CD map based on the analysis results of a third device among the plurality of fundus three-dimensional image analysis devices cannot be obtained but a C / D ratio based on the analysis results can be obtained, a substep of causing the first learning model to function as a feature extractor and using the CD map output from the first learning model as an input to the second learning model, while performing supervised learning of the second learning model based on second teacher data in which the C / D ratio based on the analysis results of the third device is used as a correct label.

[0015] In order to solve the above problem, the learning model generation device of the present invention includes a control unit that performs machine learning to generate an integrated learning model comprising a first learning model and a second learning model based on a plurality of training data that uses analysis information from a plurality of three-dimensional fundus image analysis devices as a correct label, wherein the first learning model is a learning model that receives a two-dimensional fundus image as an input and outputs a CD map that shows a planar distribution state of cups and discs, and the second learning model is a learning model that receives both the two-dimensional fundus image and a CD map related to the two-dimensional fundus image as an input and outputs a C / D ratio, and the control unit is configured to: a) when it is possible to obtain both the CD map based on the analysis result of a first device among the plurality of three-dimensional fundus image analysis devices and the C / D ratio based on the analysis result, perform supervised learning of the first learning model using first training data that sets the CD map based on the analysis result of the first device as a correct label, and set the C / D ratio based on the analysis result of the first device as a correct label a) when a CD map based on the analysis results of a second device among the plurality of fundus three-dimensional image analyzers can be obtained but a C / D ratio based on the analysis results cannot be obtained, not performing supervised learning of the second learning model, and performing supervised learning of the first learning model using first supervised data in which the CD map based on the analysis results of the second device is used as a correct label; and c) when a CD map based on the analysis results of a third device among the plurality of fundus three-dimensional image analyzers cannot be obtained but a C / D ratio based on the analysis results can be obtained, performing supervised learning of the second learning model based on second supervised data in which the C / D ratio based on the analysis results of the third device is used as a correct label while making the first learning model function as a feature extractor and using the CD map output from the first learning model as an input to the second learning model.

[0016] In order to solve the above problem, the program according to the present invention is characterized in that it is a program that causes a computer to execute any one of the above learning model generation methods. [Effects of the Invention]

[0026] According to the present invention, it is possible to generate a highly accurate CD map based on a two-dimensional fundus image. [Brief explanation of the drawings]

[0027] [Figure 1] FIG. 1 is a schematic diagram illustrating an image processing system. [Figure 2] FIG. 1 is a conceptual diagram illustrating an overview of processing in an image processing device. [Figure 3] 10 is a flowchart showing processing in the image processing device. [Figure 4] FIG. 1 is a conceptual diagram showing a learning model (during learning). [Figure 5] FIG. 1 is a conceptual diagram showing a learning model (at the time of inference). [Figure 6] FIG. 10 is a diagram illustrating how an input image is generated. [Figure 7] 10A to 10C are diagrams showing segmentation results and the like for a two-dimensional fundus image. [Figure 8] FIG. 1 shows an estimated CD map and a correct CD map. [Figure 9] 10 is a flowchart showing a process of generating teacher data. [Figure 10] 10 is a flowchart showing a learning model generation process. [Figure 11] FIG. 10 is a diagram showing an example of a result display image obtained by the fundus three-dimensional image analyzer. [Figure 12] FIG. 10 is a diagram showing training data of an integrated learning model. [Figure 13] FIG. 10 is a conceptual diagram showing how first teacher data is generated. [Figure 14] FIG. 10 is a conceptual diagram showing how second teacher data is generated. [Figure 15] 10 is a flowchart showing an inference process using a learning model. [Figure 16]FIG. 10 is a diagram showing an aspect of the second learning model in which supervised learning is not performed. [Figure 17] FIG. 10 is a diagram showing an aspect of the first learning model in which supervised learning is not performed. [Figure 18] 10 is a flowchart showing a learning process according to the second embodiment. [Figure 19] FIG. 10 is a conceptual diagram showing how both learning models are trained using the second correct label. DETAILED DESCRIPTION OF THE INVENTION

[0028] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0029] 1. First Embodiment <1-1. System Overview> As described above, the inventors of the present invention have devised a method for generating a CD map using a relatively inexpensive two-dimensional fundus image.

[0030] The inventors of the present application initially devised a technology for manually identifying the cup C and disk D regions in a two-dimensional fundus image, and for machine learning a learning model based on training data in which a CD map based on each identified region is used as the correct label.

[0031] However, when a person identifies cup C and disk D from only a planar 2D fundus image, the accuracy of identifying cup C and disk D (especially cup C) is not necessarily sufficient, and there are also individual differences (variations between people), etc. As a result, there is a problem that the accuracy of the correct label itself and, ultimately, the accuracy of the learning model (trained model) decreases.

[0032] Therefore, the inventor of the present application further devised a technique that does not use, as a correct label, a CD map based on (each region of) the cup C and the disk D manually identified only from a two-dimensional fundus image. Specifically, the inventor of the present application further devised a technique that uses, as a correct label, a CD map based on (each region of) the cup C and the disk D obtained as an analysis result (high-precision analysis result) by a three-dimensional fundus image analyzer 80.

[0033] In detail, training data is generated using information acquired by a three-dimensional fundus image analyzer (such as an OCT (Optical Coherence Tomography) device) installed in some hospitals (such as flagship hospitals), and a learning model that estimates a CD map or the like from a two-dimensional fundus image taken by a relatively inexpensive photographing device 20 is machine-learned based on the training data. In small-scale hospitals, a relatively inexpensive photographing device 20 is installed (without the need to install a three-dimensional fundus image analyzer). Then, a CD map is estimated using the trained model based on the two-dimensional fundus image taken by the photographing device 20.

[0034] Such techniques are described in more detail below.

[0035] FIG. 1 is a schematic diagram showing an image processing system 1. As shown in FIG. 1, the image processing system 1 includes a fundus image capturing device (also simply referred to as a capturing device) 20 that captures a two-dimensional fundus image (also simply referred to as a fundus image), and an image processing device 30 that processes the fundus image, etc. The image processing device 30 performs processing such as determining (estimating) the C / D ratio and CD map, which are indicators for diagnosing glaucoma, based on the fundus image captured by the capturing device 20. Such a capturing device 20 and image processing device 30 are installed, for example, in a small-scale hospital, etc. In such a small-scale hospital, etc., it is not necessary to install a three-dimensional fundus image analyzing device 80.

[0036] As described above, the C / D ratio is the ratio of the optic disc cup to the optic disc (disc) (cup diameter ratio). Here, the vertical C / D ratio is used as the C / D ratio. The CD map is a map that shows the distribution (planar distribution) of the cup C (optic disc cup) and the disc D (optic disc) in a fundus image.

[0037] The fundus image captured by the photographing device 20 is input to the image processing device 30 via a communication network (such as a LAN and / or the Internet) or a wired cable conforming to various standards. The image processing device 30 then estimates (infers) a CD map and a C / D ratio for the input image (fundus image) using a machine-learned learning model 400 (see FIGS. 1 and 4, etc.). The image processing device 30 also executes a learning process for the learning model 400 (a process for generating a trained model). The learning model 400 is machine-learned by a predetermined learning method (such as backpropagation) using training data in which the CD map based on the analysis results of the fundus three-dimensional image analyzer 80 and the C / D ratio based on the analysis results are used as correct answer labels. The image processing device 30 also generates training data to be used in the learning process for the learning model 400. These processes (see Figure 2, etc.), specifically the process of generating training data (Phase PH1), the process of generating a learning model (learning process) (Phase PH2), and the inference process using the learning model 400 (Phase PH3), will be described in detail later.

[0038] <1-2. Image processing device 30> As shown in FIG. 1, the image processing device 30 includes a controller 31 (also referred to as a control unit), a storage unit 32, a communication unit 34, and an operation unit 35.

[0039] The controller 31 is a control device that is built into the image processing device 30 and controls the operation of the image processing device 30 .

[0040] The controller 31 is configured as a computer system including one or more hardware processors (for example, a central processing unit (CPU) and a graphics processing unit (GPU)). The controller 31 performs various processes by executing, in the CPU or the like, a predetermined software program (hereinafter also simply referred to as a program) stored in a storage unit (a non-volatile storage unit such as a ROM and / or a hard disk) 32. The program (more specifically, a group of program modules) may be recorded on a portable recording medium such as a USB memory, read from the recording medium, and installed in the image processing device 30. Alternatively, the program may be downloaded via a communication network or the like and installed in the image processing device 30.

[0041] Specifically, the controller 31 executes processing in the generation stage of the teacher data (phase PH1 (see Figure 2, etc.)), processing in the generation stage of the learning model (learning stage processing) (phase PH2), and processing in the inference stage using the learning model 400 (phase PH3) (see also Figure 3).

[0042] The storage unit 32 is configured with a storage device such as a hard disk drive (HDD) and / or a solid state drive (SSD), etc. The storage unit 32 stores the learning model 400 (including learning parameters and programs related to the learning model) (and thus the trained model 420), etc.

[0043] The communication unit 34 is capable of performing network communication via a network. This network communication utilizes various protocols such as TCP / IP (Transmission Control Protocol / Internet Protocol). By utilizing this network communication, the image processing device 30 can exchange various data (photographed image data, correct answer data, etc.) with desired counterparts (e.g., the photographing device 20 and the fundus three-dimensional image analyzing device 80, etc.).

[0044] The operation unit 35 includes an operation input unit 35a that accepts operation inputs to the image processing device 30, and a display unit 35b that displays and outputs various information. A mouse, a keyboard, or the like is used as the operation input unit 35a, and a display (such as a liquid crystal display) is used as the display unit 35b. A touch panel that functions as both a part of the operation input unit 35a and a part of the display unit 35b may also be provided.

[0045] For example, the operation unit 35 accepts operation inputs for two-dimensional fundus images (for learning and inference). The operation unit 35 also accepts operation inputs for analysis results of a three-dimensional fundus image analyzer for a target person in a two-dimensional fundus image for learning.

[0046] The image processing device 30 is also referred to as a training data generating device because it has a function of generating training data for machine learning. The image processing device 30 is also referred to as a training model generating device because it has a function of machine learning the training model 400 using the training data (a function of generating a training model (trained model)). The image processing device 30 is also referred to as an inference device because it is a device that executes inference regarding the CD map and the C / D ratio using the trained training model 400.

[0047] Furthermore, although various processes (functions) are realized by one image processing device 30 here, this is not limiting. For example, various processes may be shared and realized by multiple devices. For example, the teacher data generation process in the first phase PH1 (see the upper part of FIG. 2), the learning process (learning model generation process) in the second phase PH2 (see the middle part of FIG. 2), and the inference process in the third phase PH3 (see the lower part of FIG. 2) may each be executed by separate devices.

[0048] <1-3. Learning Model 400> Fig. 4 is a conceptual diagram showing a learning model 400. Fig. 4 shows the learning model 400, inputs and outputs related to the learning model 400, and correct labels for machine learning related to the learning model 400.

[0049] 4, the learning model 400 includes two learning models, specifically, a first learning model 510 and a second learning model 520. In other words, the learning model 400 is a multi-stage model including two-stage models.

[0050] A neural network model consisting of multiple layers, specifically a convolutional neural network model, is used for each of the learning models 510 and 520. Machine learning adjusts parameters (learning parameters) of various image filters (image filters of convolutional layers) for feature extraction in multiple layers (particularly multiple intermediate layers) in the convolutional neural network model.

[0051] <First Learning Model 510> The first learning model 510 is a learning model that estimates a CD map based on a two-dimensional fundus image. Specifically, the first learning model 510 is a learning model that receives a two-dimensional fundus image (more specifically, a portion 210 of the entire two-dimensional fundus image 110) as input and outputs a CD map 220. More specifically, the first learning model 510 is a learning model that generates a segmentation map (a map indicating to which of a plurality of divided (classified) objects each pixel corresponds) for the fundus image 210. Here, a map (CD map) indicating the distribution (planar distribution) of the disk D (optic disc) and the cup C (optic disc cup) in the fundus image 210 is generated as the segmentation map (classification map).

[0052] The first learning model 510 is configured, for example, as a segmentation network comprising an encoder network and a decoder network.

[0053] The encoder network is a network that extracts high-level image features (here, cups C and disks D) from the pixel values ​​of an input image (high-resolution image). The encoder network performs convolution processing (feature extraction processing) and pooling processing (resolution reduction processing) on ​​the high-resolution image, and generates a low-resolution feature map that captures the image features.

[0054] The decoder network is a network that performs mapping processes (such as upsampling and deconvolution) to individual pixels in the input image (high-resolution image) based on the low-resolution feature map generated by the encoder network.

[0055] Here, as the first learning model 510, U 2 Net (U-squared net) is used as the first learning model 510. However, without being limited to this, various other segmentation networks such as SegNet may also be used as the first learning model 510.

[0056] Such a first learning model 510 (segmentation network) generates a CD map 220 as an output for the input image 210. Because the CD map 220 is a map estimated by the first learning model 510 (a map estimated based on the input image 210), it is also referred to as an estimated CD map.

[0057] Here, the CD map 220 is configured as a segmentation map (image) having two layer images (channels) 220C and 220D (see the left half of FIG. 8). Specifically, the CD map 220 is configured with a layer (channel) 220C indicating whether each pixel in the input image 210 is a pixel of a cup C, and a layer (channel) 220D indicating whether each pixel in the input image 210 is a pixel of a disc D. For example, the layer 220C is an image in which, at each pixel position of the input image 210, pixels of the cup C (relatively light (close to white) gray parts in FIG. 8) are assigned a "1" and other pixels (black parts in FIG. 8) are assigned a "0." The layer 220D is an image in which, at each pixel position of the input image 210, pixels of the disc D (relatively dark gray parts in FIG. 8) are assigned a "1" and other pixels (black parts in FIG. 8) are assigned a "0." In Figure 8 and other figures, for convenience of illustration, the pixels of cup C and the pixels of disk D are shown with different densities, but in reality, the same "1" (or 255, etc.) is assigned to both pixels.

[0058] <First learning model input image 210> An input image (two-dimensional fundus image) 210 input to the first learning model 510 is generated as a partial image (partial image) of the original two-dimensional fundus image (whole image) 110.

[0059] 6 is a diagram showing how an input image 210 is generated by cutting out from an original two-dimensional fundus image (whole image) 110. Specifically, as shown in FIG. 6, a partial area (area 117 surrounded by a dashed line) including (the entire) disk D is extracted as the input image 210.

[0060] Specifically, as shown in Fig. 7, a partial region including the disk D is extracted using a learning model 610 that has been machine-learned in advance. Fig. 7 is a diagram showing a segmentation result (output result by the learning model 610) for a two-dimensional fundus image.

[0061] The learning model 610 is constructed as a segmentation network that identifies (a region of) a disk D in the two-dimensional fundus image 110. The learning model 610 is a learning model (learner) that receives the two-dimensional fundus image (whole image) 110 as input and outputs a disk map 113. The disk map 113 is configured with a single layer (single channel) that indicates whether each pixel in the two-dimensional fundus image 110 is a pixel of a disk D.

[0062] Note that this learning model 610 (unlike the above-mentioned first learning model 510) may be machine-learned in advance based on training data in which the range of the disk D identified by a human (the range of the disk D within the entire image 110 of the two-dimensional fundus image) is used as the correct data. Here, taking into consideration that the range of the disk D can be identified relatively accurately by a human (compared to the cup C), a human annotation technique (a technique in which the correct area of ​​the disk D is identified by a human) is used.

[0063] A partial image extracted from the entire image 110 of the two-dimensional fundus image based on the position and size of the disk D extracted in this manner is specified as the input image 210. Specifically, the center position of the area of ​​the disk D is determined, and a square 115 having a size twice the size (height) of the disk D in the vertical direction is determined. The position of the square 115 is determined so that the center of the square 115 coincides with the center of the area of ​​the disk D. Then, a range (corresponding position range) 117 (see FIG. 6 ) corresponding to the position of the square 115 (in the disk map 113) in the entire image 110 of the two-dimensional fundus image is extracted as the input image 210.

[0064] Here, a square 115 having twice the size (height) of the disk D in the vertical direction is obtained, but this is not limiting. For example, a rectangle having twice the size (height and width) of the disk D in both the vertical and horizontal directions may be obtained. Then, the corresponding range of the rectangle may be extracted as the input image 210.

[0065] In this way, a portion of the entire two-dimensional fundus image 110 is cut out and extracted as an input image 210. The input image 210 is generated as an image having W1×H1 (e.g., 512×512) pixels (a predetermined number of pixels) by resizing or the like. The input image 210 is also a color image having three RGB channels (R image, G image, and B image). Therefore, the input image 210 is an image having a size of W1 (pixels)×H1 (pixels)×3 (channels) (see FIG. 4).

[0066] <Second Learning Model 520> The second learning model 520 is configured as a network that performs regression processing. Here, VoVNet is used as the second learning model 520. However, without being limited to this, various other networks for regression processing (ResNet, VGG, etc.) may also be used as the second learning model 520.

[0067] The second learning model 520 is a learning model that estimates the C / D ratio based on a two-dimensional fundus image. Specifically, the second learning model 520 is a learning model that receives the input image 210 as input and the C / D ratio 240 as output. More specifically, the second learning model 520 is a learning model that receives not only the input image 210 but also the CD map 220 (output from the first learning model 510) as input and outputs the C / D ratio 240. In other words, the second learning model 520 is a learning model that receives the composite image 230 (described below) as input. The second learning model 520 can also be expressed as a learning model that executes a regression process to determine the relationship between the input image 210 and the C / D ratio 240, or a regression process to determine the relationship between the composite image 230 and the C / D ratio 240. The C / D ratio 240 is also referred to as an estimated C / D ratio because it is a C / D ratio estimated by the second learning model 520 (a C / D ratio estimated based on the input image 210).

[0068] The composite image 230 is a five-channel image created by combining the fundus image 210 (input image (three-channel image) to the first learning model 510) and the CD map 220 (two-channel image) output from the first learning model 510 when the fundus image 210 is input (see Figure 14).

[0069] Furthermore, it is preferable that the composite image 230 be input to the second learning model 520 after its image size has been compressed. In other words, it is preferable that the composite image 230 is an image converted from an image having a size of W1 (pixels) × H1 (pixels) × 5 (channels) to an image having a size of W2 (pixels) × H2 (pixels) × 5 (channels). For example, an image having a size of 256 × 256 × 5 (resizing composite image 230) can be generated by performing a convolution process with a kernel size of 3 × 3 and a stride of 2 on an image (W1 = H1 = 512) before resizing that has a size of 512 × 512 × 5. This can reduce the amount of calculation and avoid overlearning.

[0070] In this way, the first learning model 510 and the second learning model 520 are partially linked and configured as an integrated whole (see FIG. 4). Specifically, the output of the first learning model 510 constitutes part of the input of the second learning model 520. In short, the learning model 400 is constructed as an integrated learning model that integrates the first learning model 510 and the second learning model 520. This learning model 400 can also be expressed as a learning device that receives the input image 210 as input and outputs the CD map 220 and the C / D ratio 240 (see the bottom of FIG. 2 and FIG. 4, etc.). The first learning model 510 and the second learning model 520 can also be expressed as sub-learning models (sub-models) of the learning model 400.

[0071] This integrated learning model 400 is trained based on training data including an input image 210, a first correct label 260 (described later), and a second correct label 270. In particular, a first learning model 510 is trained using first training data 310 (see FIG. 13) having the first correct label 260. A second learning model 520 is trained using second training data 320 (see FIG. 14) having the second correct label 270. The first learning model 510 and the second learning model 520 are trained simultaneously using training data (a set of data) including the input image 210, the first correct label 260 (described later), and the second correct label 270. The first correct label 260 and the second correct label 270, as well as the process of generating them, will be described in detail later.

[0072] The learning model 400 (510, 520) after being trained by machine learning is also called a trained model. The learning parameters of the learning model 400 (learner) are adjusted using a predetermined machine learning method to generate the trained learning model 400 (trained model) (see FIG. 2).

[0073] In this application, generating a learning model (trained model) means manufacturing (producing) a learning model (trained model), and a "method for generating a learning model (trained model)" means a "method for producing a learning model (trained model)."

[0074] <1-4. Training data generation process> Fig. 3 is a flowchart showing the processing in the image processing device 30. As shown in Fig. 3, the image processing device 30 executes a process of generating training data (step S1), a process of generating a learning model (learning process) (step S2), and an inference process using the learning model 400 (step S3).

[0075] First, the teacher data generation process (step S1: phase PH1) will be described below.

[0076] Fig. 9 is a flowchart showing the detailed processing in step S1. The processing in step S1 will be described below with reference to Fig. 9. Note that the processing in step S1 (and part of step S2 (steps S21 to S23)) is repeated for multiple target persons, thereby generating multiple pieces of training data.

[0077] First, in step S11, the image processing device 30 acquires an entire 2D fundus image 110 (see the left side of FIG. 7, etc.) as learning data in response to input operations by the operating user. Also in step S11, the image processing device 30 acquires an analysis result (specifically, a result display image 120 (see FIG. 11) showing the result of the 3D fundus image analysis) obtained by the 3D fundus image analyzer 80 for the target person (the same person) of the 2D fundus image. The result display image 120 is an image displayed (output) by the 3D fundus image analyzer 80. The analysis result (such as the result display image 120) by the 3D fundus image analyzer 80 may be input to the image processing device 30 from the 3D fundus image analyzer 80 via a communication network (or via a storage device such as an HDD) in response to input operations by the operating user. The result display image 120 may also be referred to as (an image of) an OCT analysis screen.

[0078] 11 is a diagram showing an example of a result display image 120. The result display image 120 is an image showing the analysis result by the fundus three-dimensional image analysis device 80. The result display image 120 is displayed by the fundus three-dimensional image analysis device 80. The result display image 120 can also be displayed by the image processing device 30, for example, by the image processing device 30 acquiring display data for the result display image 120.

[0079] 11, the result display image 120 has a color image display area 123, a thickness map (planar) display area 124, a thickness map (tomographic) display area 125, a CD map information display area 121, and a numerical information display area 122. Each of the areas 121 to 125 is disposed at a predetermined position within the result display image 120.

[0080] The color image display area 123 is an area for displaying a color image similar to the entire two-dimensional fundus image 110.

[0081] The thickness map (planar) display area 124 is an area for displaying a thickness map (planar) of the retinal nerve fiber layer. In the thickness map (planar), the planar distribution of the thickness of the retinal nerve fiber layer is displayed in different colors. In detail, the thickness is divided into predetermined stages, each of which is assigned a different color, and the thickness at each planar position is displayed in different colors.

[0082] The thickness map (tomography) display area 125 is an area for displaying a thickness map (tomography) of the retinal nerve fiber layer. In the thickness map (plane), the distribution of the thickness of the retinal nerve fiber layer is displayed as a difference in the height direction.

[0083] The CD map information display area 121 is an area for displaying CD map information 131 generated as an analysis result by the fundus three-dimensional image analyzer 80 .

[0084] The numerical information display area 122 is an area that displays numerical information related to the analysis results by the fundus three-dimensional image analyzer 80. The numerical information display area 122 includes a C / D ratio display area 123 and the like.

[0085] The C / D ratio display area 123 is an area that displays the (vertical) C / D ratio 132 calculated as the analysis result by the fundus three-dimensional image analyzer 80. For example, in FIG. 11, the value "0.65" is displayed as the C / D ratio 132.

[0086] Of these regions, each region other than the region 123 (specifically, regions 124, 125, 121, 122, etc.) displays the analysis results based on the three-dimensional information around the disk D obtained by the three-dimensional fundus image analyzer 80. That is, the result display image 120 includes the analysis results obtained by the three-dimensional fundus image analyzer 80 (particularly, the CD map information 131 and the C / D ratio 132).

[0087] Here, the CD map does not necessarily have to be composed of an image layer indicating the region of cup C and an image layer indicating the region of disc D. The same applies to the CD map information 131. For example, the CD map information 131 in the fundus three-dimensional image analyzer 80 may be image information (map information) having a closed curve 131C indicating the edge of the region of cup C (the boundary between cup C and its outer region) and a closed curve 131D indicating the edge of the region of disc D (the boundary between disc D and its outer region). In the CD map information 131 (see FIG. 11 ), the outer (larger) closed curve (approximately elliptical curve) 131D is the edge of disc D, and the inner (smaller) closed curve 131C is the edge of cup C.

[0088] Next, in step S12, the image processing device 30 generates an input image (a partial image of a two-dimensional fundus image) 210 as input data for learning based on the entire two-dimensional fundus image 110. The generation process of the input image 210 may be performed as described above (see FIGS. 6 and 7, etc.).

[0089] Here, a part of the two-dimensional fundus image of the entire image 110 is used as the input image 210, but this is not limiting, and the entire image 110 may be used as the input image 210. However, for the purpose of improving the efficiency of the learning process, it is preferable to use a part of the two-dimensional fundus image of the entire image 110 (a partial image (enlarged image) near the disk D) as the input image 210. Furthermore, the input image 210 may be generated based on a color image in the color image display area 123 (a fundus image acquired by the fundus three-dimensional image analysis device 80) instead of the two-dimensional fundus image of the entire image 110.

[0090] Furthermore, in step S13, the image processing device 30 extracts the CD map information 131 and the C / D ratio 132 from the result display image 120 (see the right part in FIG. 11). Specifically, the CD map information 131 (image data) displayed in the CD map information display area 121 is extracted, and the C / D ratio 132 (image data) displayed in the C / D ratio display area 123 is extracted (see also FIG. 12). The CD map information 131 and the C / D ratio 132 may be extracted from predetermined positions in the result display image 120. Note that FIG. 12 is a diagram showing training data (the input image 210, the first correct answer label 260, and the second correct answer label 270) related to the learning model 400, etc.

[0091] Then, in step S14, the image processing device 30 generates a first correct label 260 based on the CD map information 131, and generates a second correct label 270 based on the C / D ratio 132.

[0092] Specifically, the C / D ratio 132 (image data) has a state in which a numerical value indicating the C / D ratio is displayed as an image. By performing OCR (Optical Character Recognition) processing on this image (C / D ratio 132), a numerical value (e.g., "0.65") (numerical data) indicating the C / D ratio is obtained. Then, the numerical value (C / D ratio) is obtained as the second correct label 270 (also referred to as the correct C / D ratio) (see FIG. 12). For example, this OCR processing may be performed by performing recognition processing using a neural network for numerical recognition or the like.

[0093] Also, a first correct label 260 is generated based on the CD map information 131 (image data) (see FIGS. 12 and 13). FIG. 13 is a conceptual diagram showing how the first correct label 260 and the like are generated based on the CD map information 131 (image data) (in the analysis result display image).

[0094] As shown in FIG. 13 , first, adjusted CD map information 135 (a CD map image that has been resized and aligned) is obtained based on the CD map information 131. Specifically, the position of the image of the CD map information 131 is adjusted (aligned) so that the center of the disc D in the CD map information 131 is positioned at the same position as the center of the disc D in the input image 210. Furthermore, the image of the CD map information 131 is scaled (enlarged or reduced) (resized) so that the size of the disc D in the CD map information 131 is equal to the size of the disc D in the input image 210. In this resizing and alignment process, the size and position of the square 115 (see FIG. 7 ) based on the segmentation result by the learning model 610 or the like may be used. By such resizing and alignment process, the CD map information 135 (more specifically, an adjusted CD map image) is generated.

[0095] Furthermore, a first correct label 260 is generated by performing a channel separation process (a process of separating into two channels) on the adjusted CD map information 135. The first correct label 260 is a CD map (also referred to as a correct CD map) as a correct label.

[0096] The first correct answer label 260 is generated as a segmentation map (CD segmentation map (image)) having two layers (channels), similar to the CD map 220 that is output from the first learning model 510. The first correct answer label 260 is configured with a layer (channel) 260C that indicates whether each pixel is a pixel of a cup C, and a layer (channel) 260D that indicates whether each pixel is a pixel of a disc D (see the right half of FIG. 8).

[0097] Specifically, the inner region of a relatively small closed curve 135C (a closed curve indicating the edge of the region of cup C) in the CD map information 135 (see FIG. 13) is determined to be the internal region of cup C. Also, the inner region of a relatively large closed curve 135D (a closed curve indicating the edge of the region of disc D) is determined to be the internal region of disc D. Then, layer 260C (see FIG. 8) is generated as a layer image in which "1" is assigned to pixels in the internal region of cup C and "0" is assigned to other pixels (black portions). Also, layer 260D is generated as a layer image in which "1" is assigned to pixels in the internal region of disc D and "0" is assigned to other pixels (black portions).

[0098] In this way, the first correct label 260 is acquired (generated) as a CD map based on the analysis by the fundus three-dimensional image analyzer 80.

[0099] Furthermore, in step S15, the image processing device 30 generates first training data 310 based on the input image 210 and the first correct label 260 (see FIG. 13). In other words, data (labeled data) in which the CD map 260 is assigned as a correct label to the input image 210 is generated as the first training data 310.

[0100] Furthermore, in step S16, the image processing device 30 generates second training data 320 based on the input image 210 and the second correct label 270 (see FIG. 14). FIG. 14 is a conceptual diagram showing how the second training data 320 is generated.

[0101] At the time of completion of the processing of step S16, data (labeled data) in which the C / D ratio 270 is assigned as a correct label to the input image 210 has been generated as the second training data 320.

[0102] However, the second training data 320 changes in steps S22 and S23 (FIG. 10) described below (after the composite image 230 is generated). Specifically, data (labeled data) in which the C / D ratio 270 is assigned as a correct label to the composite image 230 is generated as the second training data 320. In other words, the second training data 320 is generated using as input a multi-channel image (multi-layer image) 230 that includes not only the input image 210 but also the output (CD map 220) from the first learning model 510.

[0103] In this way, in step S13, CD map information 131 is automatically extracted from the result display image 120 (the analysis result display image of the fundus three-dimensional image analyzer 80) acquired in step S11, and a CD map (first correct label) 260 is acquired (generated) based on the CD map information 131. Then, first teacher data 310 is generated with the CD map 260 based on the analysis result of the fundus three-dimensional image analyzer 80 as the label (first correct label). Furthermore, the C / D ratio 270 (numerical data) automatically extracted in steps S13 and S14 from the result display image 120 acquired in step S11 is acquired (as the second correct label). Then, second teacher data 320 is generated with the C / D ratio 270 as the label (second correct label).

[0104] Furthermore, this process (FIG. 9 (and part of FIG. 10)) is repeated for multiple target persons. As a result, multiple two-dimensional fundus images 210 for the multiple target persons are obtained in association with multiple CD maps 260 (and multiple C / D ratios 270) based on the analysis by the fundus three-dimensional image analyzer.

[0105] Note that, here, the image processing device 30 extracts the CD map information 131 from the result display image 120 to generate the first correct label 260, but this is not limiting. For example, in a case where the fundus three-dimensional image analysis device 80 has the CD map information 131 separately (independently) from the result display image 120, the CD map information 131 may be output directly from the fundus three-dimensional image analysis device 80 to the image processing device 30. The same applies to the C / D ratio 132.

[0106] <1-5. Learning process of learning model (process of generating trained model)> Next, the learning model generation process (learning process) (step S2: phase PH2) (see FIGS. 3 and 10) will be described.

[0107] Fig. 10 is a flowchart showing detailed processing in the process of generating a learning model (step S2). The process of step S2 will be described below with reference to Fig. 10. The process of step S2 is repeated for multiple pieces of training data, thereby machine learning the learning model 400. Note that some of the processing in step S2 (steps S21 to S23) is also (part of) the process of generating the second training data 320.

[0108] In step S21, the image processing device 30 inputs the input image 210 to the first learning model 510, and obtains the CD map 220 as the output of the first learning model 510 (see also FIG. 4).

[0109] In step S22, the image processing device 30 combines the output from the first learning model 510 (i.e., the CD map 220) with the input image 210 to generate a combined image 230 (see also FIG. 14). As described above, the combined image 230 is a multi-channel (multi-layer) image having a total of five channels (five layers). In detail, the combined image 230 is a combined image of the input image (color image) 210 having three channels, R, G, and B, and the CD map 220 (see FIG. 8) having two channels, C and D.

[0110] In step S23, the image processing device 30 generates second training data 320 based on the composite image 230 and the second correct label (C / D ratio) 270 (see also FIG. 14). In other words, data (labeled data) in which the C / D ratio 270 is assigned as the correct label to the composite image 230 is generated as the second training data 320. That is, the second training data 320 is generated using not only the input image 210 but also the CD map 220 as input to the second learning model 520.

[0111] Then, in step S24, the image processing device 30 uses the first teacher data 310 to train the first learning model 510. In other words, the CD map 260 based on the analysis result by the fundus three-dimensional image analysis device 80 is used as a correct label, and supervised learning is performed. Specifically, machine learning is performed to minimize (optimize) a loss function (evaluation function) related to the difference between the CD map 220 output from the first learning model 510 and the CD map 260 (which is the first correct label).

[0112] In step S25, the image processing device 30 uses the second teacher data 320 to train the second learning model 520. In other words, the C / D ratio 270 based on the analysis result by the fundus three-dimensional image analysis device 80 is used as the correct label, and supervised learning is performed. Specifically, machine learning is performed to minimize (optimize) a loss function (evaluation function) related to the difference between the C / D ratio 240 output from the second learning model 520 and the C / D ratio 270 (which is the second correct label). In addition to the input image 210, the CD map 220 output from the first learning model 510 is also used as an input to the second learning model 520.

[0113] The above process is repeated for multiple training data, whereby the learning model 400 is machine-learned, and a learned model 400 (420) is generated. The first learning model 510 (first learned model) in the learned model 420 is generated as an inference device that infers a CD map (more specifically, a CD segmentation map) based on the input image 210. The second learning model 520 (second learned model) in the learned model 420 is generated as an inference device that infers a C / D ratio based on the input image 210. The second learning model 520 is configured to be connected (integrated) with the first learning model 510.

[0114] In such machine learning processing, the CD map 260 based on the analysis results by the fundus three-dimensional image analyzer 80 is used as the correct label. Therefore, it is possible to generate a highly accurate first learning model 510. Furthermore, even when estimating (generating) a CD map based on a two-dimensional fundus image, it is possible to estimate (generate) a highly accurate CD map by using the first learning model 510.

[0115] Similarly, since the C / D ratio 270 based on the analysis result by the fundus three-dimensional image analyzer 80 is used as the correct label, it is possible to generate a highly accurate second learning model 520. Furthermore, by using the second learning model 520, it is possible to estimate the C / D ratio with high accuracy based on the two-dimensional fundus image.

[0116] The processing order of steps S21 to S25 is not limited to the above. For example, the generation of the second teacher data 320 (particularly the input image (synthetic image 230) to the second learning model 520) used in learning the second learning model 520 may be performed after learning the first learning model 510 (step S24). In particular, steps S21 to S23 may be performed after step S24, and then step S25 may be performed.

[0117] <1-6. Inference processing using learning models> Next, the inference process using the learning model (step S3: phase PH3) (see FIGS. 3 and 15) will be described.

[0118] 15 is a flowchart showing detailed processing in the inference processing (step S3) using the learning model. The processing in step S3 will be described below with reference to FIG.

[0119] First, in step S31, the image processing device 30 accepts an operation input of the entire image 110 of a two-dimensional fundus image related to a target person (target person to be estimated) of the inference process (estimation process). Then, in step S32, the image processing device 30 generates an input image 210 (212) (see FIGS. 6 and 7) based on the entire image 110 of the two-dimensional fundus image (similar to step S12 (FIG. 9)). In this way, a two-dimensional fundus image (110, 210) as a target image of the inference process (estimation process) is acquired.

[0120] Next, in step S33, the image processing device 30 inputs the input image 212 to the first learning model 510 and acquires the output (CD map 220 (222)) from the first learning model 510 (see FIG. 5). In other words, the CD map 222 for the target image 212 is estimated using the first learning model 510 that has been machine-trained based on training data that utilizes the analysis results of the fundus three-dimensional image analysis device 80.

[0121] In addition, in step S34, the image processing device 30 generates a composite image 230 (232) based on the CD map 220 (222) and the input image 210 (212).

[0122] Then, in step S35, the image processing device 30 inputs the composite image 232 to the second learning model 520 and acquires the C / D ratio 240 (242) output from the second learning model 520 in response to the input of the composite image 232 (see FIG. 5). In other words, the C / D ratio 242 for the target image 212 is estimated using the second learning model 520 (and the first learning model 510) that has been machine-trained based on training data that utilizes the analysis results of the fundus three-dimensional image analysis device 80.

[0123] In step S36, the image processing device 30 outputs the inference result. Specifically, the inference result (the CD map 222 and the C / D ratio 242) is displayed on the display unit 35b. The inference result may be output as a data file or the like.

[0124] The C / D ratio 242 is an estimate of the C / D ratio based on the input image 210 (such as 0.63).

[0125] The CD map 222 is an estimated CD segmentation map based on the input image 210. The CD map 222 is displayed, for example, with its two channels (C and D channels) superimposed (with cup C and disc D color-coded). Alternatively, the two channels of the CD map 222 may be displayed separately (by channel (by layer)). The CD map 222 may also be displayed superimposed on the input image 212. Furthermore, these multiple display states may be switched in response to an input operation by a user (such as a doctor). This allows doctors and others to easily view the CD map (e.g., make it easier to see).

[0126] Furthermore, by visually checking the CD map 222, a doctor or the like can determine the validity of the inference result (the CD map 222) by the first learning model 510. Ultimately, it is also possible to determine the validity of the inference result (the C / D ratio 242) by the second learning model 520. For example, if the CD map 222 is invalid, it can be determined that the inference result (the C / D ratio 242) by the second learning model 520 is also invalid.

[0127] In the above processing, a CD map 222 for the target image is estimated using a learning model 400 (particularly the first learning model 510) that has been machine-trained based on training data using the analysis results of the fundus 3D image analysis device 80.

[0128] In particular, in the machine learning of the learning model 400, a CD map 260 based on the analysis results by the fundus three-dimensional image analysis device 80 is used as a correct label. Therefore, it is possible to generate a highly accurate CD map 222 based on a two-dimensional fundus image. That is, it is possible to obtain a highly accurate CD map 222 based on a two-dimensional fundus image captured by the imaging device 20. Furthermore, in order to obtain a highly accurate CD map 222, it is not necessary for all hospitals to introduce the fundus three-dimensional image analysis device 80, and it is sufficient for small-scale hospitals, in particular, to introduce a relatively inexpensive imaging device 20 (an imaging device 20 that captures two-dimensional fundus images).

[0129] Furthermore, the C / D ratio 270 based on the analysis results by the fundus three-dimensional image analyzer 80 is used as the correct label. Therefore, it is possible to estimate the C / D ratio with high accuracy based on the two-dimensional fundus image.

[0130] Furthermore, in the learning model 400 (see FIGS. 4 and 5, etc.), the C / D ratio is estimated based on the input image 210. In other words, the learning model 400 including the second learning model 520 deals with a regression problem of determining the relationship between the input image 210 and the C / D ratio. However, in detail, both the input image 210 and the CD map 220 output from the first learning model 510 (i.e., the composite image 230) are input to the second learning model 520 that estimates the C / D ratio, and the relationship between the composite image 230 and the C / D ratio is learned.

[0131] Generally, machine learning of a learning model for regression processing requires a relatively large amount of training data. It is difficult to ensure sufficient accuracy with machine learning based on a relatively small amount of training data. This tendency is particularly pronounced in regression processing that seeks the relationship between input information (input information having a large amount of information (multidimensional information)) such as input image 210 and a small amount of output information (low-dimensional (e.g., one-dimensional) information).

[0132] In contrast, in the above embodiment, the CD map 220 is also used to learn the C / D ratio based on the input image 210. Although the CD map 220 contains a small amount of information, it is dense information that is more directly related to the C / D ratio than the input image 210. Therefore, even with machine learning based on a relatively small amount of training data, it is possible to achieve highly accurate regression processing (compared to a case in which the CD map 220 is not used). Furthermore, the CD map 220 is estimated by the first learning model 510 trained using the correct CD map (first correct label) 260 based on the analysis results of the fundus three-dimensional image analyzer 80. Therefore, it is possible to improve the estimation accuracy of the CD map 220, and ultimately, it is possible to improve the estimation accuracy of the C / D ratio based on the CD map 220.

[0133] In this way, the integrated learning model 400 is constructed by linking and integrating the first learning model 510 and the second learning model 520, and in particular, the CD map 220 output from the first learning model 510 constitutes (part of) the input of the second learning model 520. This makes it possible to obtain an estimated value of the C / D ratio with high accuracy.

[0134] In the above embodiment, the C / D ratio 242 is output as an estimated value of the C / D ratio (step S36), but this is not limited to this. For example, in addition to the C / D ratio 242, a C / D ratio 250 (252) (not shown) based on the output from the first learning model 510 (CD map 222 (FIG. 5)) may be calculated. The C / D ratio 252 may be calculated as the ratio between the length of the cup C and the length of the disc D (in the vertical direction) in the CD map 222. The average value of the C / D ratio 252 and the C / D ratio 242 may then be output as the estimated value of the C / D ratio by the learning model 400.

[0135] This method (a method of finding the average value as the final estimated value of the C / D ratio) is particularly useful when the number of pieces of training data (teacher data) is small and it is determined that the accuracy of the C / D ratio 240 output from the second learning model 520 for regression processing is insufficient. In such cases, by also using the C / D ratio 252 based on the CD map 220 (other related estimated values) from the first learning model 510, it is possible to improve the accuracy of the C / D ratio estimation.

[0136] In the above embodiment, both the input image 210 and the CD map 220 are input to the second learning model 520 to calculate the C / D ratio 240. However, this is not limiting. For example, the (estimated) C / D ratio 240 may be output by inputting only the CD map 220 to the second learning model 520 (without inputting the input image 210). However, in this case, the inaccuracy of the CD map (estimated CD map) 220 (such as inaccuracy in the alignment of the CD map 220 with the input image 210 (and thus inaccuracy as to whether each pixel of the CD map 220 is a cup C)) significantly affects the estimation of the C / D ratio. However, by using not only the CD map but also the input image 210 (using the composite image 230) as in the above embodiment, such an effect can be mitigated.

[0137] 2. Second Embodiment The second embodiment is a modification of the first embodiment.

[0138] In the learning stage of the first embodiment, supervised learning is always performed on both the first learning model 510 and the second learning model 520 based on each training data (see FIG. 4), but the present invention is not limited to this. For example, when performing machine learning on the integrated learning model 400 comprising the first learning model 510 and the second learning model 520, supervised learning does not necessarily have to always be performed on both of the two learning models (the first learning model and the second learning model).

[0139] Here, in the first embodiment, the fundus three-dimensional image analyzer 80 that outputs both the CD map information 131 and the C / D ratio 132 as analysis results is exemplified.

[0140] However, among the multiple fundus three-dimensional image analyzers 80, there are some that output only one of the CD map information 131 and the C / D ratio 132 as the analysis result. For training the learning model 400, it is preferable to collect as much training data as possible by utilizing various types of fundus three-dimensional image analyzers 80. Therefore, it is preferable to train the learning model 400 by effectively utilizing the analysis results of the fundus three-dimensional image analyzers 80 that output only one of the CD map information 131 and the C / D ratio 132.

[0141] In the second embodiment, such a technique will be described.

[0142] Here, the analysis information of a plurality of three-dimensional fundus image analyzers 80 (however, a plurality of types of three-dimensional fundus image analyzers 80) is used as the correct answer label. More specifically, an integrated learning model 400 (see FIG. 4, etc.) configured by linking a first learning model 510 and a second learning model 520 is machine-learned using a plurality of training data to which the analysis information of a plurality of three-dimensional fundus image analyzers is assigned as the correct answer label.

[0143] The plurality of fundus three-dimensional image analyzers 80 are roughly classified into the following three types (three kinds).

[0144] The first type three-dimensional fundus image analyzer 80 (also referred to as 80A) outputs both the CD map information 131 and the C / D ratio 132, similarly to the first embodiment.

[0145] The second type three-dimensional fundus image analyzer 80 (also referred to as 80B) outputs CD map information 131, but does not output the C / D ratio 132 (unlike the first embodiment).

[0146] The third type three-dimensional fundus image analyzer 80 (also referred to as 80C) outputs the C / D ratio 132 but does not output the CD map information 131 (unlike the first embodiment).

[0147] In the second embodiment, when the image processing device 30 uses the analysis data of the fundus three-dimensional image analysis device 80B, it generates a first correct label (CD map) 260 based on the analysis data (especially the CD map information 131), but does not generate a second correct label (C / D ratio) 270. Then, supervised learning is performed using the first correct label 260 (CD map) (supervised learning based on the second teacher data 320 is not performed) (see FIG. 16). Note that FIG. 16 is a diagram showing an overview of the learning process etc. when the analysis data of the fundus three-dimensional image analysis device 80B is used.

[0148] Furthermore, when the image processing device 30 uses the analysis data of the fundus three-dimensional image analysis device 80C, it generates a second correct label (C / D ratio) 270 based on the analysis data (especially the C / D ratio 132), but does not generate a first correct label 260. Then, supervised learning is performed using the second correct label 270 (C / D ratio) (supervised learning based on the first teacher data 320 is not performed) (see FIG. 17). Note that FIG. 17 is a diagram showing an overview of the learning process etc. when the analysis data of the fundus three-dimensional image analysis device 80C is used.

[0149] Fig. 18 is a flowchart showing the learning process of step S2 (also referred to as S2B) according to the second embodiment. Fig. 18 shows the learning process using each piece of learning data (teaching data of the learning model 400). In practice, the learning process is performed using a plurality of pieces of learning data (teaching data), and the process of step S2B (Fig. 18) is repeated a plurality of times. The learning process according to the second embodiment will be described below with reference to Fig. 18.

[0150] In step S51, branching processing is performed according to the type of the target fundus three-dimensional image analyzing device 80 (the device 80 from which the target data is obtained). Specifically, the processing branches to steps S52, S53, and S54 depending on which of the three types of devices 80A, 80B, and 80C the analysis data of which device is to be used as learning data (correct answer labels, etc.).

[0151] When the target device 80 is the first type device 80A, the image processing device 30 can acquire both a CD map 260 (first correct label) based on the analysis result of the fundus three-dimensional image analysis device 80A and a C / D ratio 270 (second correct label) based on the analysis result. Specifically, the CD map 260 is acquired based on the CD map information 131, and the C / D ratio 270 is acquired based on the C / D ratio 132.

[0152] In this case, the process proceeds to step S52. In step S52, a learning process similar to that in the first embodiment is executed (see FIG. 4, etc.). Specifically, supervised learning of the first learning model 510 is executed using first training data 310 (FIG. 13) in which the CD map 260 based on the analysis results of the fundus three-dimensional image analysis device 80A is used as the correct answer label. Furthermore, supervised learning of the second learning model 520 is executed based on second training data 320 (FIG. 14) in which the C / D ratio 270 based on the analysis results of the fundus three-dimensional image analysis device 80A is used as the correct answer label.

[0153] When the target device 80 is the second type device 80B, the image processing device 30 can acquire a CD map 260 (first correct label) based on the analysis result of the fundus three-dimensional image analysis device 80B. However, the image processing device 30 cannot acquire a C / D ratio 270 (second correct label) based on the analysis result. Specifically, only the CD map 260 based on the CD map information 131 is acquired as the correct label.

[0154] In this case, the process proceeds to step S53, where the processing shown in Fig. 16 is executed. Specifically, in step S53, supervised learning of the first learning model 510 is executed using the first training data 310, which has the CD map 260 based on the analysis result of the fundus three-dimensional image analysis device 80B as the correct answer label. On the other hand, supervised learning of the second learning model 520 is not executed (the learning parameters of the second learning model 520 do not change). In detail, only steps S21 and S24 in Fig. 10 are executed, and steps S22, S23, and S25 are not executed.

[0155] When the target device 80 is a third type device 80C, the image processing device 30 can obtain a C / D ratio 270 (first correct label) based on the analysis result of the fundus three-dimensional image analysis device 80C. However, the image processing device 30 cannot obtain a CD map 260 (first correct label) based on the analysis result. Specifically, only the C / D ratio 270 based on the C / D ratio 132 is obtained as the correct label.

[0156] In this case, the process proceeds to step S54, where the process shown in FIG. 17 is executed. In step S54, supervised learning of the first learning model 510 is not executed (the learning parameters of the first learning model 510 are not changed). However, the first learning model 510 is made to function as a feature extractor, and a CD map 220 output from the first learning model 510 in response to the input of the input image 210 is acquired. In other words, unsupervised learning is executed for the first learning model 510. The CD map 220 is then used as an input (more specifically, a part of the input) of the second learning model 520. Specifically, a composite image 230 of the CD map 220 and the input image 210 is input to the second learning model 520. A learning process is executed to minimize a loss function (evaluation function) related to the difference between the C / D ratio 240 output from the second learning model 520 in response to such an input (composite image 230) and the C / D ratio 270 as the correct label. That is, supervised learning of the second learning model 520 is performed based on the second training data 320. In detail, only steps S21, S22, S23, and S25 in Fig. 10 are performed, and step S24 is not performed.

[0157] In the second embodiment, a plurality of analysis results from at least two of the above-mentioned three types of devices 80A, 80B, and 80C are used. A learning model 400 is trained using a plurality of training data (learning data) based on the analysis results from a plurality of fundus three-dimensional image analyzers 80 including the at least two types of devices. In other words, when the process of step S2B (FIG. 18) is repeated multiple times in the learning process (step) using a plurality of training data, at least two (sub)steps of (sub)steps S52, S53, and S54 are executed.

[0158] When two or more types of device 80 out of the above three types are used, it is possible to secure one or more pieces of training data for supervised learning of first learning model 510 and one or more pieces of training data for supervised learning of second learning model 520. In other words, when at least two (sub)steps of steps S52, S53, and S54 are executed, it is possible to perform supervised learning of first learning model 510 based on one or more pieces of training data and supervised learning of second learning model 520 based on one or more pieces of training data.

[0159] According to the above-described process, even if the plurality of three-dimensional fundus image analyzing devices 80 includes a device that can provide only one of the two correct labels (the first correct label 260 and the second correct label 270), it is possible to effectively utilize the analysis results of that device to train the learning model 400. In short, it is not necessary for all of the plurality of three-dimensional fundus image analyzing devices 80 to be devices that can provide both of the two correct labels (the first correct label 260 and the second correct label 270). Therefore, even if the amount of training data provided by the three-dimensional fundus image analyzing device 80 is small, it is possible to effectively utilize the training data (small amount of data).

[0160] Specifically, an integrated learning model 400 constructed by linking a first learning model 510 and a second learning model 520 is machine-learned. More specifically, the CD map 220 output from the first learning model 510 constitutes (a part of) the input of the second learning model 520. This makes it possible to effectively utilize less data than in a case where the second learning model 520 is trained in a state completely separated from the first learning model 510. In particular, it is possible to effectively utilize a larger amount of data to estimate the C / D ratio with higher accuracy.

[0161] For example, consider a situation in which there is analysis data for 100 people obtained by device 80A, analysis data for 100 people obtained by device 80B, and analysis data for 100 people obtained by device 80C. In this situation, consider a case in which a system is constructed that performs machine learning on only a learning model (referred to as 820) that estimates a C / D ratio 240 based on an input image 210. In this case, the analysis data for a total of 200 people obtained by devices 80A and 80C, which can output a C / D ratio 132, can be used for machine learning, etc., of the learning model 820 using the second correct answer label 270, and contributes to the estimation of the C / D ratio. However, the analysis data for 100 people obtained by device 80B, which cannot output a C / D ratio 132, cannot be used for machine learning, etc., of the learning model 820, and does not contribute at all to the estimation of the C / D ratio. Furthermore, the analysis data for 100 people obtained by device 80B is not used at all to estimate the CD map.

[0162] On the other hand, according to the second embodiment, it is possible to utilize not only the analysis data for a total of 200 people obtained by devices 80A and 80C, but also the analysis data for 100 people obtained by device 80B. Specifically, first, the analysis data for a total of 200 people obtained by devices 80A and 80C can be used for machine learning of the learning model 520, directly contributing to the estimation of the C / D ratio. Furthermore, the analysis data for 100 people obtained by device 80B can be used for machine learning of the CD map 220 by the first learning model 510, contributing to the estimation process (and learning process) of the CD map 220. Furthermore, the analysis data for 100 people obtained by device 80B indirectly contributes to the estimation process (and learning process) of the C / D ratio 240 (particularly to improving the estimation accuracy) by using the CD map 220 as input to the second learning model 520. In this way, it is possible to effectively utilize a large amount of data for 300 people.

[0163] Furthermore, by inputting the CD map 220 into the second learning model 520, as described above, it is also possible to achieve highly accurate regression processing (highly accurate estimation of the C / D ratio) based on a relatively small amount of training data.

[0164] <3. Modifications, etc.> Although the embodiment of the present invention has been described above, the present invention is not limited to the above-described contents.

[0165] <Learning of the first learning model 510 using the second correct label 270> For example, in each of the above embodiments, in step S25 (FIG. 10), only the second learning model 520 is trained (supervised) using the second correct answer label 270 (FIG. 4). However, this is not limited to this, and in step S25, not only the second learning model 520 but also the first learning model 510 may be trained (supervised) using the second correct answer label 270 (see FIG. 19). Note that FIG. 19 is a conceptual diagram showing how both the second learning model 520 and the first learning model 510 are trained using the second correct answer label 270.

[0166] Specifically, the first learning model 510 and the second learning model 520 are considered to be connected via the CD map 220 to construct a single learning model (specifically, a CNN). The CD map 220 is both the output layer of the first learning model 510 and (part of) the input layer of the second learning model 520. The CD map 220 is an intermediate output in such a single learning model (a connected model of the first learning model 510 and the second learning model 520), and can also be considered to be one of multiple intermediate layers in the single learning model. The entire single learning model (the learning models connected via the CD map 220) may then be trained using the second correct label 270. In particular, both learning models 510 and 520 (the entire single learning model) may be trained to minimize the difference between the C / D ratio 240 and the second correct label 270 (see, in particular, the thick arrows in the middle of Figure 19). More specifically, the backpropagation method updates the weighting parameters, etc. between multiple layers (including an input layer, multiple intermediate layers, and an output layer) in the second learning model 520, and also updates the weighting parameters, etc. between multiple layers (including an input layer, multiple intermediate layers, and an output layer) in the first learning model 510 (upstream via the CD map 220).

[0167] According to this, not only the second learning model 520 but also the first learning model 510 are trained (in a supervised manner) using training data including the second correct label 270 (such as training data in which the second correct label 270 is assigned to the input image 210). That is, the second correct label 270 is also used to train the first learning model 510. Therefore, the accuracy of the first learning model 510 is improved, and ultimately the accuracy of the estimated CD map 220 is improved. In short, the feedback effect of the second correct label 270 (with respect to the estimated C / D ratio 240) also extends to improving the estimation accuracy of the CD map 220 by the first learning model 510 (contributing to the estimation of the CD map 220). Furthermore, by further training the first learning model 510 (in a supervised manner) using the first correct label 260, the accuracy of the first learning model 510 is further improved.

[0168] The above modifications may be made to either the first embodiment or the second embodiment.

[0169] For example, similar modifications may be made in steps S52 and S54 (see FIG. 18) of the second embodiment. Specifically, the entire learning model (an integrated learning model in which the first learning model 510 and the second learning model 520 are connected via the CD map 220) (i.e., both learning models 510 and 520 (see FIGS. 4 and 17, etc.)) may be trained by the backpropagation algorithm using the second correct answer label 270.

[0170] In particular, in the modified step S54, supervised learning (of the first learning model 510) using the first ground truth label 260 is not performed, but supervised learning using the second ground truth label 270 is performed on the first learning model 510 (particularly during backward propagation). More specifically, during forward propagation (when generating the CD map 220 and C / D ratio 240), as in FIG. 17 , the first learning model 510 functions as a feature extractor to obtain the CD map 220 output from the first learning model 510 in response to the input of the input image 210. The CD map 220 is used as part of the input of the second learning model 520 (part of the composite image 230). For such input, the second learning model 520 outputs an estimated C / D ratio 240. On the other hand, during backward propagation (parameter update), supervised learning is performed (by backpropagation) on both the second learning model 520 and the first learning model 510 so as to minimize a loss function related to the difference between the estimated C / D ratio 240 and the correct C / D ratio 270 (see the thick arrows in the middle of Figure 19). That is, supervised learning using the second correct label 270 is performed on not only the second learning model 520 but also the first learning model 510.

[0171] Such modifications may be made.

[0172] <Order of update process using first and second correct labels> Furthermore, the model optimization using the first correct label 260 (see step S24) and the model optimization using the second correct label 270 (see step S25) may be performed in the reverse order (to that of FIG. 10).

[0173] For example, both a process of updating both the parameter set (weight parameter set, etc.) of the second learning model 520 and the parameter set of the first learning model 510 using the second correct label 270 and a process of updating the parameter set of the first learning model 510 using the first correct label 260 may be executed in this order. That is, the model optimization process using the second correct label 270 may be executed first (before the model optimization process using the first correct label 260).

[0174] <Repeated unit in optimization process> In the above-described embodiment, the optimization process using the first correct label 260 (see step S24) and the optimization process using the second correct label 270 (see step S25) are performed separately and sequentially (see FIG. 10). For example, after the process of step S24 (the process of optimizing the parameter group of the first learning model 510 using the first correct label 260) is completed, the process of step S25 (the process of optimizing the parameter group of the first learning model 510 (and the second learning model 520) using the second correct label 270) is performed. However, this is not limited thereto, and the optimization process using the first correct label 260 and the optimization process using the second correct label 270 may be performed without being completely separated (integrated). In short, it is sufficient that the integrated learning model 400 is trained (while remaining integrated) using the backpropagation algorithm or the like.

[0175] for example, A process of updating the parameter sets of the second learning model 520 and the parameter sets of the first learning model 510 using the second correct answer label 270 (see the thick arrow lines in the middle and bottom rows of Figure 19), A process of updating the parameter set of the first learning model 510 using the first correct label 260 (see the thick arrow line in the upper part of Figure 19), Both of these processes may be executed as one unit process. Then, the one unit process may be repeatedly executed (as a process during backward propagation) to gradually advance the optimization process of the integrated learning model 400. The optimization process may be carried out by the error backpropagation method or the like.

[0176] In the one unit process, both a learning process using the first correct label 260 and a learning process using the second correct label 270 are executed. Then, by repeating the one unit process, the optimization process using the first correct label 260 and the optimization process using the second correct label 270 are executed in parallel (and gradually).

[0177] More specifically, first, as a forward process, A process in which an input image 210 is input to a first learning model 510 and a CD map 220 is generated (see the upper part of Figure 19); A process in which the CD map 220 (and the input image 210) is input into the second learning model 520 to generate the C / D ratio 240 (see the bottom part of Figure 19); Both processes are executed.

[0178] Next, as the reverse process, A process of updating (by a small amount) the parameter sets of the second learning model 520 and the first learning model 510 so as to reduce the difference between the C / D ratio 240 generated in the forward process and the second correct label 270 (see the thick arrow lines in the middle and bottom rows of Figure 19); A process of updating (a small amount) the parameter set of the first learning model 510 so as to reduce the difference between the CD map 220 generated in the forward process and the first correct label 260 (see the thick arrow line in the upper part of Figure 19), Both processes are executed.

[0179] Then, a total of four (four types) of processes, including both forward processes and both reverse processes, form one unit process, and the first learning model 510 and the second learning model 520 are learned in parallel by repeatedly executing this one unit process (until a specified termination condition is met).

[0180] Such a learning process may be performed.

[0181] Moreover, such a learning process (particularly the reverse process within one unit process) may be performed, for example, as follows.

[0182] First, the second correct label 270 is used to update each parameter of the second learning model 520 (so as to reduce the difference between the C / D ratio 240 and the second correct label 270) (see the arrows in the lower part of Figure 19). Next, the second correct label 270 is used to calculate an update value (first type of update value) for each parameter of the first learning model 510 (so as to reduce the difference between the C / D ratio 240 and the second correct label 270) (see the arrows in the middle part of Figure 19). Furthermore, the first correct label 260 is used to calculate an update value (second type of update value) for each parameter of the first learning model 510 (so as to reduce the difference between the CD map 220 and the first correct label 260) (see the arrows in the upper part of Figure 19). Then, the first type of update value and the second type of update value are summed for each parameter to determine the update value for each parameter of the first learning model 510. This allows the update process for the first learning model 510 to be performed collectively.

[0183] <Other> In addition, in the above-described embodiments, the CD map 220 (estimated CD map) is used as an input (part of the input, more specifically) of the second learning model 520 in both the learning process (step S2) and the inference process (step S3). However, this is not limited to this. The estimated CD map 220 may be used as an input to the second learning model 520 in the inference process, and the correct CD map 260 may be used as an input to the second learning model 520 in the learning process. In other words, in the learning process of the second learning model 520, the CD map 220 output from the first learning model 510 may be considered to match the first correct label 260, and the first correct label 260 may be used as an input to the second learning model 520. This can improve the accuracy of the second learning model 520.

[0184] More specifically, for example, in the learning process of the second learning model 520 in the first embodiment, the composite image 230 (also referred to as 230B) may be generated (as input data to the second learning model 520) by combining the input image 210 and the CD map 260 (first correct label). Furthermore, in the second embodiment, when analysis data from the device 80A is used in the learning process of the second learning model 520, the composite image 230 (230B) may be generated by combining the input image 210 and the CD map 260 (first correct label). In this way, the "CD map related to the two-dimensional fundus image" input to the second learning model 520 may be the CD map 220 (the CD map output from the first learning model 510 when the input image 210 is input) or the first correct label 260. The second learning model 520 may be trained using second training data 320 including the composite image 230B as input. When the analysis data of the device 80C is used, the composite image 230 may be generated by combining the input image 210 and the CD map 220, as in the above-described embodiments.

[0185] In addition, in the above-described embodiments, the composite image 230 is used as an input to the second learning model 520. In other words, both the fundus image 210 and the CD map 220 (CD map related to a two-dimensional fundus image) output from the first learning model 510 when the fundus image 210 is input are used as input to the second learning model 520. However, the present invention is not limited to this. For example, only the fundus image 210 may be used as an input to the second learning model 520. Alternatively, only the CD map 220 (CD map related to a two-dimensional fundus image) output from the first learning model 510 when the fundus image 210 is input may be used as an input to the second learning model 520. However, as described above, it is preferable that the composite image 230 be used as an input to the second learning model 520.

[0186] Note that the CD map 220 is estimated based on the input image 210 (see FIG. 4). Therefore, even when only the CD map 220 is input to the second learning model 520 (when the input image 210 itself is not input), the estimated C / D ratio 240 output from the second learning model 520 is a C / D ratio estimated (indirectly) based on the input image (two-dimensional fundus image) 210. In other words, even when only the CD map 220 is used as input to the second learning model 520, the second learning model 520 is a model that estimates the C / D ratio based on the two-dimensional fundus image.

[0187] In addition, in the above-described embodiments, both the CD map 220 and the C / D ratio 240 are estimated by the learning model 400, but this is not limiting. For example, only the CD map 220 may be estimated, without estimating the C / D ratio 240. However, a large amount of information is preferable for diagnosis by a doctor, and it is therefore preferable that both the CD map 220 and the C / D ratio 240 be estimated. [Explanation of symbols]

[0188] 1. Image processing system 20 Imaging equipment 30 Image processing device 80,80A,80B,80C Fundus 3D image analysis device 110 2D fundus image 120 (Result display image related to the analysis result of the fundus three-dimensional image analyzer 80) 131 CD Map Information 132 (Correct) C / D ratio 210 Input image for integrated learning model (2D fundus image) 220 Output from the first learning model (estimated CD map) 230 Input to the second learning model (synthetic image) 240 Output of the second learning model (estimated C / D ratio) 260 First correct label (correct CD map) 270 Second correct label (correct C / D ratio) 400 Integrated Learning Model 510 First Learning Model 520 Second Learning Model C cup (optic disc) D disc (optic disc cup)

Claims

1. a) a step of machine learning a first learning model using a two-dimensional fundus image as an input and a CD map showing a planar distribution state of cups and discs as an output; b) a step of machine learning a second learning model that uses both the two-dimensional fundus image and a CD map related to the two-dimensional fundus image output from the first learning model as inputs and outputs a C / D ratio; Equipped with In the step a), the first learning model is machine-learned using training data to which a CD map based on an analysis result by a fundus three-dimensional image analyzer is attached as a correct answer label; A method for generating a learning model, characterized in that in step b), the second learning model is machine-learned using training data to which the C / D ratio as the analysis result by a fundus 3D image analysis device is attached as a correct answer label.

2. The method for generating a learning model described in claim 1, characterized in that in step b), an integrated learning model connecting the first learning model and the second learning model is machine-learned using training data to which the C / D ratio as the analysis result by a fundus 3D image analysis device is attached as a correct answer label, thereby machine-learning not only the second learning model but also the first learning model.

3. a control unit that performs machine learning to generate a first learning model that receives a two-dimensional fundus image as an input and outputs a CD map showing a planar distribution state of cups and discs, and also performs machine learning to generate a second learning model that receives both the two-dimensional fundus image and a CD map related to the two-dimensional fundus image output from the first learning model as input and outputs a C / D ratio; Equipped with The control unit machine learning the first learning model using training data to which a CD map based on the analysis result by the fundus three-dimensional image analyzer is attached as a correct answer label; A learning model generation device characterized by machine learning the second learning model using training data to which the C / D ratio, which is the analysis result of a three-dimensional fundus image analysis device, is attached as a correct answer label.

4. The control unit of the learning model generation device described in claim 3 is characterized in that it machine-learns not only the second learning model but also the first learning model by using training data to which the C / D ratio obtained as an analysis result by a fundus three-dimensional image analysis device is attached as a correct answer label to machine-learn an integrated learning model that connects the first learning model and the second learning model.

5. a) a step of machine learning an integrated learning model including a first learning model and a second learning model based on a plurality of training data sets that use analysis information from a plurality of three-dimensional fundus image analyzers as correct labels; A method for generating a learning model, comprising: the first learning model is a learning model that receives a two-dimensional fundus image as an input and outputs a CD map that indicates a planar distribution state of cups and discs, the second learning model is a learning model that receives both the two-dimensional fundus image and a CD map related to the two-dimensional fundus image as inputs and outputs a C / D ratio; The step a) a-1) when both a CD map based on the analysis result of a first device among the plurality of fundus three-dimensional image analyzers and a C / D ratio based on the analysis result can be obtained, a sub-step of performing supervised learning of the first learning model using first teacher data in which the CD map based on the analysis result of the first device is used as a correct answer label, and performing supervised learning of the second learning model using second teacher data in which the C / D ratio based on the analysis result of the first device is used as a correct answer label; a-2) when a CD map based on the analysis result of a second device among the plurality of fundus three-dimensional image analyzers can be acquired but a C / D ratio based on the analysis result cannot be acquired, a sub-step of not performing supervised learning of the second learning model, and performing supervised learning of the first learning model using first teacher data in which the CD map based on the analysis result of the second device is used as a correct answer label; a-3) when a CD map based on the analysis result of a third device among the plurality of fundus three-dimensional image analyzers cannot be obtained but a C / D ratio based on the analysis result can be obtained, a substep of causing the first learning model to function as a feature extractor and using the CD map output from the first learning model as an input of the second learning model, while executing supervised learning of the second learning model based on second teacher data in which the C / D ratio based on the analysis result of the third device is used as a correct answer label; A method for generating a learning model, comprising at least two substeps:

6. A learning model generation device, a control unit that performs machine learning to generate an integrated learning model including a first learning model and a second learning model based on a plurality of training data that uses analysis information from a plurality of fundus three-dimensional image analyzers as correct labels; Equipped with the first learning model is a learning model that receives a two-dimensional fundus image as an input and outputs a CD map that indicates a planar distribution state of cups and discs, the second learning model is a learning model that receives both the two-dimensional fundus image and a CD map related to the two-dimensional fundus image as inputs and outputs a C / D ratio; The control unit a) when both a CD map based on the analysis result of a first device among the plurality of fundus three-dimensional image analyzers and a C / D ratio based on the analysis result can be obtained, a process of performing supervised learning of the first learning model using first teacher data in which the CD map based on the analysis result of the first device is used as a correct answer label, and performing supervised learning of the second learning model using second teacher data in which the C / D ratio based on the analysis result of the first device is used as a correct answer label; b) when a CD map based on the analysis result of a second device among the plurality of fundus three-dimensional image analyzers can be obtained but a C / D ratio based on the analysis result cannot be obtained, a process of not performing supervised learning of the second learning model, and performing supervised learning of the first learning model using first teacher data in which the CD map based on the analysis result of the second device is used as a correct answer label; c) if a CD map based on the analysis results of a third device among the plurality of fundus three-dimensional image analyzers cannot be obtained but a C / D ratio based on the analysis results can be obtained, a process of causing the first learning model to function as a feature extractor and using the CD map output from the first learning model as an input to the second learning model, while performing supervised learning of the second learning model based on second teacher data in which the C / D ratio based on the analysis results of the third device is used as a correct answer label; A learning model generation device characterized by executing at least two of the three processes above.

7. A program that causes a computer to execute the learning model generation method according to any one of claims 1, 2, and 5.

Images