Medical image processing device, medical image processing system, medical image processing method and program

The medical image processing device uses machine learning to accurately detect retinal layer boundaries in diseased eyes, addressing false detections in conventional methods and improving diagnostic precision.

JP7579372B2Active Publication Date: 2024-11-07CANON KK
View PDF 7 Cites 0 Cited by

Patent Information

Application Number
JP2023024953
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2018-12-10
Filing Date
2023-02-21
Publication Date
2024-11-07
Estimated Expiration
2039-08-09

AI Technical Summary

Technical Problem

Conventional medical image processing methods for retinal layers in diseased eyes, such as those with irregular retinal shapes due to diseases like glaucoma, suffer from false detections when automatically identifying layer boundaries due to irregularities and changes in the retina.

Method used

A medical image processing device and method that utilizes a trained model to detect retinal layers in tomographic images, incorporating machine learning algorithms to accurately identify layer boundaries regardless of disease or irregularities, generating frontal images based on positional information and depth ranges.

Benefits of technology

Enables precise detection of retinal layer boundaries in diseased eyes, improving accuracy and robustness against irregular retinal shapes and lesions, enhancing diagnostic capabilities.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007579372000001
    Figure 0007579372000001
  • Figure 0007579372000002
    Figure 0007579372000002
  • Figure 0007579372000003
    Figure 0007579372000003
Patent Text Reader

Abstract

An image processing device is provided that can detect the boundary of retinal layers regardless of disease, location, etc. [Solution] An image processing device comprising: an acquisition unit that acquires a tomographic image of a test eye; and a first processing unit that executes a first detection process to detect at least one of multiple retinal layers in the acquired tomographic image using a trained model obtained by learning data showing at least one of multiple retinal layers in the tomographic image of the test eye.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to a medical image processing device, Medical Image Processing System The present invention relates to a medical image processing method and a program. [Background technology]

[0002] Tomographic imaging devices for the eye, such as devices using optical coherence tomography (OCT devices), are capable of observing the state of the inside of the retina in three dimensions. These tomographic imaging devices have been attracting attention in recent years because they are useful for more accurate diagnosis of diseases.

[0003] One form of OCT is TD-OCT (Time domain OCT), which combines a broadband light source with a Michelson interferometer. This is configured to obtain depth-resolved information by measuring the interference light with the backscattered light of the signal arm by scanning the delay of the reference arm. However, it is difficult to obtain images at high speed with this type of TD-OCT.

[0004] Therefore, as a method for acquiring images at higher speeds, Spectral domain OCT (SD-OCT) is known, which uses a broadband light source and acquires an interferogram with a spectrometer. Also, Swept Source OCT (SS-OCT) is known, which uses a high-speed wavelength swept light source as a light source and measures the spectral interference with a single channel photodetector.

[0005] When a tomographic image is acquired by OCT, if the thickness of the nerve fiber layer can be measured, the progression of a disease such as glaucoma and the degree of recovery after treatment can be quantitatively diagnosed. In order to quantitatively measure the thickness of these layers, a technology is disclosed in Patent Document 1 in which the boundaries of each retinal layer are detected from the tomographic image using a computer and the thickness of each layer is measured. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2008-73099 A Summary of the Invention [Problem to be solved by the invention]

[0007] However, the conventional technology has the following problems. In diseased eyes, the shape of the retina becomes irregular due to loss of layers, bleeding, and the occurrence of white spots and neovascularization. Therefore, the conventional image processing method that judges the result of image feature extraction using the regularity of the retinal shape and detects the boundary of the retinal layer has a limitation in that erroneous detection occurs when automatically detecting the boundary of the retinal layer.

[0008] Therefore, the present invention provides a medical image processing apparatus capable of detecting the boundaries of retinal layers regardless of disease, location, etc. Medical Image Processing System One of the objects of the present invention is to provide a medical image processing method and program. [Means for solving the problem]

[0009] A medical image processing apparatus according to an embodiment of the present invention includes an acquisition unit for acquiring a tomographic image of a subject's eye, and a tomographic image of the subject's eye and a tomographic image of at least one layer of the tomographic image. position information Images containing A trained model obtained by training using training data including the above, wherein the acquired tomographic image is used as input data of the trained model, and the trained model is outputted as Outputting an image including position information relating to at least one layer in the acquired tomographic image, Regarding at least one layer in the acquired tomographic image position a processing unit for acquiring information; and a processing unit for acquiring information relating to at least one layer in at least a part of a depth range in a three-dimensional tomographic image of the subject's eye, position and a generating unit that generates a front image corresponding to the determined depth range using the information.

[0010] A medical image processing method according to another embodiment of the present invention includes a step of acquiring a tomographic image of a subject's eye, and a step of detecting a tomographic image of the subject's eye and at least one layer in the tomographic image. position information Images containing A trained model obtained by training using training data including the above, wherein the acquired tomographic image is used as input data of the trained model, and the trained model is outputted as Outputting an image including position information relating to at least one layer in the acquired tomographic image, Regarding at least one layer in the acquired tomographic image position and acquiring information about at least one layer in at least a portion of a depth range in a three-dimensional tomographic image of the subject's eye, the at least one layer being acquired. position and using the information to generate a frontal image corresponding to the determined depth range. Effect of the Invention

[0011] According to one embodiment of the present invention, it is possible to detect the boundaries of retinal layers regardless of the disease, location, and the like.

[0012] Further features of the invention will become apparent from the following description of exemplary embodiments with reference to the accompanying drawings. [Brief description of the drawings]

[0013] [Figure 1] 1 illustrates an example of a schematic configuration of an image processing system according to a first embodiment. [Diagram 2] 1A and 1B are diagrams for explaining the structure of an eye, a tomographic image, and a fundus image. [Diagram 3] 2 is a flowchart of a series of processes according to the first embodiment. [Figure 4] FIG. 13 is a diagram for explaining an example of a learning image. [Diagram 5] FIG. 11 is a diagram for explaining an example of the size of a learning image. [Figure 6] FIG. 2 is a diagram for explaining an example of a machine learning model according to the first embodiment; [Figure 7] 1 shows an example of a display screen. [Figure 8]13 illustrates an example of a schematic configuration of an image processing system according to a second embodiment. [Figure 9] 11 is a flowchart of a series of processes according to the second embodiment. [Figure 10] FIG. 13 is a diagram for explaining detection of a retinal region. [Figure 11] FIG. 11 is a diagram for explaining an example of the size of a learning image. [Figure 12] FIG. 11 is a diagram for explaining an example of a machine learning model according to a second embodiment. [Figure 13] FIG. 11 is a diagram for explaining retinal layer detection according to the second embodiment. [Figure 14] FIG. 13 is a diagram for explaining examples of input and output images in a trained model. [Figure 15] 13 illustrates an example of a schematic configuration of an image processing system according to a fourth embodiment. [Figure 16] 13 is a flowchart of a series of processes according to the fourth embodiment. [Figure 17] 13 illustrates an example of a schematic configuration of an image processing system according to a fifth embodiment. [Figure 18] 13 is a flowchart of a series of processes according to a fifth embodiment. [Figure 19] FIG. 11 is a diagram for explaining a correction process for a retina area. [Figure 20] FIG. 23 is a diagram for explaining an example of a learning image according to the sixth embodiment. [Figure 21] 13 shows an example of multiple OCTA En-Face images and brightness tomographic images. [Figure 22] 13 shows an example of a user interface according to the seventh embodiment. [Diagram 23] 13 shows an example of a user interface according to the seventh embodiment. [Figure 24] 13 shows an example of a region label image relating to a term explanation. [Diagram 25] 1 shows an example of a neural network configuration relating to the explanation of terms. [Figure 26] 1 shows an example of a neural network configuration relating to the explanation of terms. [Figure 27]13 shows an example of a region label image relating to a term explanation. [Figure 28] 13 illustrates an example of the configuration of an image processing device according to an eighth embodiment. [Figure 29] 13 is a flowchart showing an example of a process flow of an image processing device according to an eighth embodiment. [Diagram 30] 13 is a flowchart showing an example of a process flow of an image processing device according to an eighth embodiment. [Diagram 31] FIG. 23 is a diagram showing an example of a user interface included in the photographing device according to the eighth embodiment. [Diagram 32] FIG. 23 is a diagram showing an example of a user interface included in the photographing device according to the eighth embodiment. [Diagram 33] 13 is a flowchart showing an example of a process flow of an image processing device according to a ninth embodiment. [Diagram 34] 13 shows image processing according to an eleventh embodiment. [Diagram 35] 23 is a flowchart showing an example of a process flow of the image processing device according to the eleventh embodiment. [Diagram 36] 12 shows image processing according to a twelfth embodiment. [Figure 37] 23 is a flowchart showing an example of a process flow of an image processing device according to a thirteenth embodiment. [Figure 38] 13 shows image processing according to a thirteenth embodiment. [Figure 39] 23 is a flowchart showing an example of a process flow of an image processing device according to a thirteenth embodiment. [Diagram 40] 13 shows image processing according to a thirteenth embodiment. [Diagram 41] 23 is a flowchart showing an example of a process flow of an image processing device according to a fourteenth embodiment. [Diagram 42] FIG. 23 is a diagram showing an example of a user interface included in the photographing device according to the fifteenth embodiment. [Diagram 43] 23 illustrates an example of the configuration of an image processing device according to an eighteenth embodiment. [Diagram 44] 23 illustrates an example of the configuration of an image processing device according to a nineteenth embodiment. [Diagram 45]23 is a flowchart showing an example of a process flow of an image processing device according to a nineteenth embodiment. [Figure 46] 13 shows an example of the configuration of a neural network used as a machine learning model according to Modification 9. [Figure 47] 13 shows an example of the configuration of a neural network used as a machine learning model according to Modification 9. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Exemplary embodiments for carrying out the present invention will now be described in detail with reference to the accompanying drawings.

[0015] However, the dimensions, materials, shapes, and relative positions of components described in the following embodiments are arbitrary and can be changed according to the configuration of the device to which the present invention is applied or various conditions. In the drawings, the same reference numerals are used between the drawings to indicate elements that are the same or functionally similar.

[0016] Example 1 Hereinafter, an image processing system including an image processing device using a tomographic image of an eye according to a first embodiment of the present invention will be described with reference to FIGS. 1 to 7. In this embodiment, all target retinal layers are detected using a trained model related to a machine learning model. In the following, the machine learning model refers to a trained model using a machine learning algorithm such as deep learning. In addition, the trained model refers to a model that has been trained (learned) in advance using appropriate teacher data for a machine learning model using an arbitrary machine learning algorithm. However, the trained model is not limited to a model that does not undergo further learning, but can also undergo additional learning. In the following, the teacher data refers to training data, and is composed of a pair of input data and output data. In addition, the correct answer data refers to output data of the training data (teacher data).

[0017] 1 shows an example of a schematic configuration of an image processing system 1 including an image processing device 20 (medical image processing device) according to this embodiment. As shown in Fig. 1, the image processing system 1 includes an OCT device 10, which is an example of a tomographic image capturing device, the image processing device 20, a fundus image capturing device 30, an external storage device 40, a display unit 50, and an input unit 60.

[0018] The OCT device 10 is an example of a tomographic imaging device that is a device for capturing a tomographic image of a subject's eye. As the OCT device, any type of OCT device can be used, for example, SD-OCT or SS-OCT.

[0019] The image processing device 20 is connected to the OCT device 10, the fundus image capturing device 30, the external storage device 40, the display unit 50, and the input unit 60 via an interface, and can control these. The image processing device 20 can generate various images such as a tomographic image and an En-Face image (front image) of the subject's eye based on various signals acquired from the OCT device 10, the fundus image capturing device 30, and the external storage device 40. The image processing device 20 can also perform image processing on these images. The image processing device 20 may be configured by a general-purpose computer, or may be configured by a computer dedicated to the image processing system 1.

[0020] The fundus image capturing device 30 is a device for capturing a fundus image of the subject's eye, and as the device, for example, a fundus camera, a scanning laser ophothalmoscope (SLO), etc. may be used. The OCT device 10 and the fundus image capturing device 30 may be configured as an integrated device or as separate devices.

[0021] The external storage device 40 stores information about the subject's eye (such as the patient's name, age, and sex), various captured image data, shooting parameters, image analysis parameters, and parameters set by the operator, each associated with the other. The external storage device 40 may be configured by any storage device, and may be configured by a storage medium such as an optical disk or a memory.

[0022] The display unit 50 is configured by any display, and can display information about the subject's eye and various images under the control of the image processing device 20.

[0023] The input unit 60 is, for example, a mouse, a keyboard, or a touch operation screen, and the operator can input instructions to the image processing device 20, the OCT device 10, and the fundus image capturing device 30 to the image processing device 20 via the input unit 60. When the input unit 60 is a touch operation screen, the input unit 60 can be configured integrally with the display unit 50.

[0024] Although these components are shown as separate entities in FIG. 1, some or all of these components may be configured as an integrated unit.

[0025] Next, the OCT device 10 will be described. The OCT device 10 is provided with a light source 11, a galvanometer mirror 12, a focus lens stage 13, a coherence gate stage 14, a detector 15, and an internal fixation lamp 16. Note that since the OCT device 10 is a known device, a detailed description thereof will be omitted, and the following describes the capturing of a tomographic image performed in response to an instruction from an image processing device 20.

[0026] When an instruction to capture an image is transmitted from the image processing device 20, the light source 11 emits light. The light from the light source 11 is split into measurement light and reference light by a splitting unit (not shown). The OCT device 10 irradiates the subject (subject's eye) with the measurement light and detects interference light between the return light from the subject and the reference light, thereby generating an interference signal including tomographic information of the subject.

[0027] The galvanometer mirror 12 is used to scan the fundus of the subject's eye with the measurement light, and the imaging range of the fundus by OCT imaging can be determined by the scanning range of the measurement light by the galvanometer mirror 12. The image processing device 20 can determine the imaging range in the planar direction of the fundus and the number of scanning lines (scanning speed in the planar direction) by controlling the driving range and speed of the galvanometer mirror 12. In FIG. 1, the galvanometer mirror 12 is shown as one unit for the sake of simplicity of explanation, but the galvanometer mirror 12 is actually composed of two mirrors, one for X-scan and one for Y-scan, and can scan a desired range on the fundus with the measurement light. Note that the configuration of the scanning unit for scanning the measurement light is not limited to the galvanometer mirror, and any other deflection mirror can be used. In addition, as the scanning unit, for example, a deflection mirror capable of scanning the measurement light in a two-dimensional direction with one mirror, such as a MEMS mirror, may be used.

[0028] The focus lens stage 13 is provided with a focus lens (not shown). By moving the focus lens stage 13, the focus lens can be moved along the optical axis of the measurement light. Therefore, the measurement light can be focused on the retinal layer of the fundus via the anterior segment of the subject's eye by the focus lens. The measurement light that has irradiated the fundus is reflected and scattered at each retinal layer, and travels back along the optical path as return light.

[0029] The coherence gate stage 14 is used to adjust the length of the optical path of the reference light or the measurement light to accommodate differences in axial length of the subject's eye. In this embodiment, the coherence gate stage 14 is configured by a stage provided with a mirror, and can correspond to the optical path length of the reference light to the optical path length of the measurement light by moving in the optical axis direction in the optical path of the reference light. Here, the coherence gate represents a position where the optical distances of the measurement light and the reference light in OCT are equal. The coherence gate stage 14 can be controlled by the image processing device 20. The image processing device 20 can control the imaging range in the depth direction of the subject's eye by controlling the position of the coherence gate by the coherence gate stage 14, and can control imaging of the retinal layer side or imaging of a deeper side than the retinal layer.

[0030] The detector 15 detects interference light between the return light of the measurement light from the subject's eye and the reference light, which occurs in an interference unit (not shown), and generates an interference signal. The image processing device 20 acquires the interference signal from the detector 15 and performs Fourier transform or the like on the interference signal to generate a tomographic image of the subject's eye.

[0031] The internal fixation lamp 16 is provided with a display unit 161 and a lens 162. In this embodiment, a display unit 161 having a plurality of light-emitting diodes (LD) arranged in a matrix is ​​used as an example of the display unit 161. The lighting positions of the light-emitting diodes are changed according to the part to be photographed under the control of the image processing device 20. The light from the display unit 161 is guided to the subject's eye via the lens 162. The light emitted from the display unit 161 has a wavelength of, for example, 520 nm, and is displayed in a desired pattern under the control of the image processing device 20.

[0032] The OCT device 10 may be provided with a drive control unit for the OCT device 10 that controls the driving of each component based on the control by the image processing device 20.

[0033] Next, the structure of the eye and images acquired by the image processing system 1 will be described with reference to Figs. 2(a) to 2(c). Fig. 2(a) is a schematic diagram of an eyeball. Fig. 2(a) shows the cornea C, the crystalline lens CL, the vitreous body V, the macular region M (the center of the macula represents the fovea), and the optic disc D. In this embodiment, the case of mainly photographing the posterior pole of the retina including the vitreous body V, the macular region M, and the optic disc D will be described. Although not described below, the OCT device 10 is also capable of photographing the anterior segment of the eye such as the cornea and the crystalline lens.

[0034] FIG. 2(b) shows an example of a tomographic image obtained by photographing a retina using the OCT device 10. In FIG. 2(b), AS indicates an image unit obtained by one A-scan. Here, the A-scan refers to obtaining tomographic information in the depth direction at one point of the subject's eye by the above-mentioned series of operations of the OCT device 10. In addition, performing A-scans multiple times in any transverse direction (main scanning direction) to obtain two-dimensional tomographic information in the transverse direction and depth direction of the subject's eye is called a B-scan. A single B-scan image can be constructed by collecting multiple A-scan images obtained by A-scans. Hereinafter, this B-scan image will be called a tomographic image.

[0035] FIG. 2(b) shows blood vessels Ve, vitreous body V, macula M, and optic nerve head D. Boundary line L1 represents the boundary between the internal limiting membrane (ILM) and nerve fiber layer (NFL), boundary line L2 represents the boundary between the nerve fiber layer and ganglion cell layer (GCL), and boundary line L3 represents the photoreceptor inner-outer segment junction (ISOS). Boundary line L4 represents the retinal pigment epithelium layer (RPE), boundary line L5 represents Bruch's membrane (BM), and boundary line L6 represents the choroid. In the tomographic image, the horizontal axis (main scanning direction of OCT) is the x-axis, and the vertical axis (depth direction) is the z-axis.

[0036] Fig. 2(c) shows an example of a fundus image acquired by photographing the fundus of the examinee's eye using the fundus image photographing device 30. Fig. 2(c) shows the macular region M and optic disc D, and retinal blood vessels are shown by thick curves. In the fundus image, the horizontal axis (main scanning direction of OCT) is the x-axis, and the vertical axis (sub-scanning direction of OCT) is the y-axis.

[0037] Next, a description will be given of the image processing device 20. The image processing device 20 includes an acquisition unit 21, an image processing unit 22, a drive control unit 23, a storage unit 24, and a display control unit 25.

[0038] The acquiring unit 21 can acquire data of interference signals of the subject's eye from the OCT device 10. The data of interference signals acquired by the acquiring unit 21 may be analog signals or digital signals. When the acquiring unit 21 acquires analog signals, the image processing device 20 can convert the analog signals into digital signals. The acquiring unit 21 can acquire various images such as tomographic data, tomographic images, and En-Face images generated by the image processing unit 22. Here, the tomographic data refers to data including information regarding the tomography of the subject, including data based on interference signals by OCT, and data obtained by performing fast Fourier transform (FFT) or any signal processing on the data.

[0039] Furthermore, the acquisition unit 21 acquires a group of imaging conditions for the tomographic image to be image-processed (for example, imaging date and time, name of imaging part, imaging area, imaging angle of view, imaging method, image resolution and gradation, image pixel size, image filter, information on image data format, etc.). Note that the group of imaging conditions is not limited to those exemplified. Also, the group of imaging conditions does not need to include all of the exemplified ones, and may include some of them.

[0040] The acquisition unit 21 can also acquire data including fundus information acquired by the fundus image capturing device 30. Furthermore, the acquisition unit 21 can acquire information for identifying the subject's eye, such as a subject identification number, from the input unit 60, etc. The acquisition unit 21 can store the acquired various data and images in the storage unit 24.

[0041] The image processing unit 22 can generate a tomographic image, an En-Face image, etc. from the data acquired by the acquisition unit 21 or the data stored in the storage unit 24, and can perform image processing on the generated or acquired image. Therefore, the image processing unit 22 can function as an example of a generating unit that generates an En-Face image or a motion contrast front image described later. The image processing unit 22 is provided with a tomographic image generating unit 221 and a processing unit 222 (first processing unit).

[0042] The tomographic image generating unit 221 can generate tomographic data by performing processing such as Fourier transform on the interference signal acquired by the acquiring unit 21, and generate a tomographic image based on the tomographic data. Note that any known method may be adopted as a method for generating a tomographic image, and detailed description thereof will be omitted.

[0043] The processing unit 222 may include a trained model for a machine learning model based on a machine learning algorithm such as deep learning. Specific machine learning models will be described later. The processing unit 222 uses the trained model to execute a detection process for detecting retinal layers of the subject's eye in a tomographic image, and detects each retinal layer.

[0044] The drive control unit 23 can control the driving of each component of the OCT device 10 and the fundus image capturing device 30 connected to the image processing device 20. The storage unit 24 can store various images and data such as the tomographic data acquired by the acquisition unit 21 and the tomographic images generated and processed by the image processing unit 22. The storage unit 24 can also store programs and the like for performing the functions of each component of the image processing device 20 by being executed by the processor.

[0045] The display control unit 25 can control the display on the display unit 50 of various information acquired by the acquisition unit 21, tomographic images generated and processed by the image processing unit 22, and information input by the operator.

[0046] Each component of the image processing device 20 other than the storage unit 24 may be configured by a software module executed by a processor such as a CPU (Central Processing Unit) or an MPU (Micro Processing Unit). The processor may be, for example, a GPU (Graphical Processing Unit) or an FPGA (Field-Programmable Gate Array). Each component may be configured by a circuit that performs a specific function such as an ASIC. The storage unit 24 may be configured by any storage medium such as an optical disk or a memory.

[0047] Next, a series of processes according to this embodiment will be described with reference to Fig. 3. Fig. 3 is a flowchart of a series of processes according to this embodiment. When a series of processes according to this embodiment is started, the process proceeds to step S301.

[0048] In step S301, the acquisition unit 21 acquires a subject identification number, which is an example of information for identifying a subject's eye, from outside the image processing device 20, such as the input unit 60. The acquisition unit 21 acquires information about the subject's eye held in the external storage device 40 based on the subject identification number, and stores the information in the storage unit 24.

[0049] In step S302, the drive control unit 23 controls the OCT device 10 to scan the subject's eye to capture an image, and the acquisition unit 21 acquires an interference signal including tomographic information of the subject's eye from the OCT device 10. The drive control unit 23 controls the OCT device 10 to operate the light source 11, the galvanometer mirror 12, etc., in response to an instruction from an operator to start scanning, thereby scanning the subject's eye.

[0050] The galvanometer mirror 12 includes an X scanner for the horizontal direction and a Y scanner for the vertical direction. Therefore, the drive control unit 23 can scan the measurement light in each of the horizontal direction (X) and vertical direction (Y) in the device coordinate system by changing the orientation of each of these scanners. Note that the drive control unit 23 can also scan the measurement light in a direction that combines the horizontal direction and the vertical direction by changing the orientation of these scanners simultaneously. Therefore, the drive control unit 23 can scan the measurement light in any direction on the fundus plane.

[0051] The drive control unit 23 adjusts various shooting parameters when shooting. Specifically, the drive control unit 23 sets at least the position of the pattern displayed by the internal fixation light 16, the scan range and scan pattern by the galvanometer mirror 12, the coherence gate position, and the focus.

[0052] The drive control unit 23 controls the light emitting diodes of the display unit 161 to control the position of the pattern displayed by the internal fixation light 16 so as to capture the center of the macula and the optic disc of the subject's eye. The drive control unit 23 also sets a scan pattern such as a raster scan, radial scan, or cross scan for capturing a three-dimensional volume as the scan pattern of the galvanometer mirror 12. Regardless of which scan pattern is selected, multiple images (repetition count is two or more) are captured on one line repeatedly. In this embodiment, the scan pattern is a cross scan, and 150 images of the same location are repeatedly captured. After adjustment of these capture parameters is completed, the drive control unit 23 controls the OCT device 10 to capture the subject's eye in response to an instruction to start capturing images by the operator. The repetition count according to this embodiment is an example, and may be set to any number according to a desired configuration.

[0053] Although detailed description is omitted in this disclosure, the OCT device 10 can track the subject's eye in order to capture the same location for averaging. This allows the OCT device 10 to scan the subject's eye while reducing the influence of fixational eye movement.

[0054] In step S303, the tomographic image generating unit 221 generates a tomographic image based on the interference signals acquired by the acquiring unit 21. The tomographic image generating unit 221 can generate a tomographic image by performing a general reconstruction process on each interference signal.

[0055] First, the tomographic image generating unit 221 removes fixed pattern noise from the interference signal. The fixed pattern noise is removed by averaging the signals of multiple acquired A-scans to extract fixed pattern noise and subtracting it from the input interference signal. After that, the tomographic image generating unit 221 performs a desired window function process to optimize the depth resolution and dynamic range, which are in a trade-off relationship when the interference signal is Fourier transformed in a finite section. The tomographic image generating unit 221 generates tomographic data by performing a fast Fourier transform (FFT) process on the interference signal that has been subjected to the window function process.

[0056] The tomographic image generating unit 221 obtains each pixel value of the tomographic image based on the generated tomographic data, and generates the tomographic image. Note that the method of generating the tomographic image is not limited to this, and any known method may be used.

[0057] In step S304, the processing unit 222 of the image processing unit 22 performs a process of detecting the retinal layers. The process of the processing unit 222 will be described with reference to Figs.

[0058] The processing unit 222 detects the boundaries of retinal layers in a plurality of tomographic images acquired using the OCT device 10. The processing unit 222 detects each retinal layer using a trained model related to a machine learning model on which machine learning has been performed in advance.

[0059] Here, the machine learning algorithm according to the present embodiment will be described with reference to Figs. 4(a) to 6. The learning data (teacher data) of the machine learning model according to the present embodiment is composed of a pair group of one or more input data and output data. Specifically, the input data includes a tomographic image 401 acquired by OCT, and the output data includes a boundary image 402 in which the boundary of the retinal layer is specified for the tomographic image. In this embodiment, an image showing the boundary 403 between the ILM and the NFL, the boundary 404 between the NFL and the GCL, the ISOS 405, the RPE 406, and the BM 407 is used as the boundary image 402. Although not shown, images showing the boundary between the outer plexiform layer (OPL) and the outer nuclear layer (ONL), the boundary between the inner plexiform layer (IPL) and the inner nuclear layer (INL), the boundary between the INL and the OPL, the boundary between the GCL and the IPL, etc. may be used as other boundaries.

[0060] The boundary image 402 used as output data may be an image in which a doctor or the like has indicated a boundary in a tomographic image, or an image in which a boundary has been detected by a rule-based boundary detection process. However, if machine learning is performed using a boundary image in which boundary detection has not been performed appropriately as output data of training data, an image obtained using a trained model trained using the training data may also be a boundary image in which boundary detection has not been performed appropriately. Therefore, by removing pairs including such boundary images from the training data, it is possible to reduce the possibility that an inappropriate boundary image is generated using a trained model. Here, the rule-based process refers to a process that utilizes a known regularity, and the rule-based boundary detection refers to a boundary detection process that utilizes a known regularity, such as the regularity of the shape of the retina.

[0061] 4(a) and 4(b) show an example of one XZ cross section in the XY plane of the retina, but the cross section is not limited to this. Although not shown, tomographic images and boundary images for any number of XZ cross sections in the XY plane can be learned in advance to be able to handle cross sections captured with various different scan patterns such as raster scan and radial scan. For example, when using data such as a tomographic image of a retina captured three-dimensionally by raster scan, volume data in which adjacent tomographic images are aligned can be used as training data. In this case, it is possible to generate a group of pair images at any angle from one piece of volume data (three-dimensional tomographic image) and one piece of corresponding three-dimensional boundary data (three-dimensional boundary image). In addition, the machine learning model may be trained using images actually captured with various scan patterns as training data.

[0062] Next, images during learning will be described. A group of images constituting a pair group of a tomographic image 401 and a boundary image 402, which constitutes the teacher data of the machine learning model, is created by rectangular area images of a certain image size with corresponding positional relationships. The creation of the images will be described with reference to Figs. 5(a) to 5(c).

[0063] First, a case will be described where one of the pairs constituting the teacher data is a tomographic image 401 and a boundary image 402. In this case, as shown in Fig. 5(a), a pair is formed with a rectangular area image 501, which is the entire tomographic image 401, as input data, and a rectangular area image 502, which is the entire boundary image 402, as output data. Note that in the example shown in Fig. 5(a), a pair of input data and output data is formed by the entirety of each image, but the pairs are not limited to this.

[0064] 5(b), a pair may be formed by using a rectangular area image 511 in the tomographic image 401 as input data and a rectangular area image 513, which is a corresponding imaging area in the boundary image 402, as output data. The rectangular areas of the rectangular area images 511 and 513 are based on A-scan units. The A-scan unit may be a single A-scan unit or a number of A-scan units.

[0065] In Fig. 5(b), the A-scan unit is used as the basis, but instead of defining the entire depth direction of the image as the region, it is also possible to set parts outside the rectangular region above and below. In other words, the horizontal size of the rectangular region can be set to several A-scans, and the depth size of the rectangular region can be set to be smaller than the depth size of the image.

[0066] Alternatively, as shown in FIG. 5C, a pair may be formed with a rectangular area image 521 in the tomographic image 401 as input data and a rectangular area image 523, which is the corresponding imaging area in the boundary image 402, as output data.

[0067] During learning, the scan range (photographing angle of view) and scan density (number of A-scans) are normalized to make the image sizes uniform, so that the rectangular region sizes during learning can be made constant. The rectangular region images shown in Figures 5(a) to 5(c) are examples of rectangular region sizes when learning separately.

[0068] The number of rectangular regions can be set to one in the example shown in FIG. 5(a), and to multiple in the examples shown in FIG. 5(b) and (c). For example, in the example shown in FIG. 5(b), a pair can be formed with a rectangular region image 512 in the tomographic image 401 as input data and a rectangular region image 514, which is the corresponding photographed region in the boundary image 402, as output data. Also, for example, in the example shown in FIG. 5(c), a pair can be formed with a rectangular region image 522 in the tomographic image 401 as input data and a rectangular region image 524, which is the corresponding photographed region in the boundary image 402, as output data. In this way, pairs of mutually different rectangular region images can be created from pairs of one tomographic image and one boundary image. Note that the pair group constituting the teacher data can be enriched by creating a large number of pairs of rectangular region images while changing the positions of the regions in the original tomographic image and boundary image to different coordinates.

[0069] In the examples shown in Figs. 5(b) and (c), the rectangular regions are shown discretely, but in reality, the original tomographic image and boundary image can be divided into a group of rectangular region images of a certain image size that are continuous without gaps. The original tomographic image and boundary image may also be divided into a group of rectangular region images at random positions that correspond to each other. In this way, by selecting images of smaller regions as rectangular regions (or strip regions) as pairs of input data and output data, a lot of pair data can be generated from the tomographic image 401 and boundary image 402 that constitute the original pair. Therefore, the time required for training the machine learning model can be reduced. On the other hand, in a completed trained model of a machine learning model, the time required for image segmentation processing to be performed tends to be long. Here, image segmentation processing refers to processing for identifying and distinguishing regions and boundaries in an image.

[0070] Next, as an example of the machine learning model according to the present embodiment, a convolutional neural network (CNN) that performs image segmentation processing on an input tomographic image will be described with reference to FIG. 6. FIG. 6 shows an example of a configuration 601 of a machine learning model in the processing unit 222. Note that, as the machine learning model according to the present embodiment, for example, FCN (Fully Convolutional Network) or SegNet can also be used. In addition, a machine learning model that performs object recognition on a region-by-region basis may be used according to a desired configuration. As the machine learning model that performs object recognition, for example, RCNN (Region CNN), fastRCNN, or fasterRCNN can be used. Furthermore, as the machine learning model that performs object recognition on a region-by-region basis, YOLO (You Look Only Once) or SSD (Single Shot MultiBox Detector) can also be used.

[0071] The machine learning model shown in Fig. 6 is composed of a plurality of layers that process and output a group of input values. The types of layers included in the configuration 601 of the machine learning model include a convolution layer, a downsampling layer, an upsampling layer, and a merger layer.

[0072] The convolution layer is a layer that performs convolution processing on a group of input values ​​according to parameters such as the set filter kernel size, the number of filters, the stride value, the dilation value, etc. The number of dimensions of the filter kernel size may also be changed according to the number of dimensions of the input image.

[0073] The downsampling layer is a layer that performs processing to reduce the number of output value groups to be less than the number of input value groups by thinning out or combining input value groups. Specifically, such processing includes, for example, Max Pooling processing.

[0074] The upsampling layer is a layer that performs a process to make the number of output values ​​larger than the number of input values ​​by duplicating the input values ​​or adding values ​​interpolated from the input values, for example, a linear interpolation process.

[0075] A synthesis layer is a layer that inputs a group of values, such as a group of output values ​​of a layer or a group of pixel values ​​that make up an image, from multiple sources and performs processing to synthesize them by concatenating or adding them.

[0076] In addition, as parameters set in the convolution layer group included in the configuration 601 shown in Fig. 6, for example, the filter kernel size is set to 3 pixels wide and 3 pixels high, and the number of filters is set to 64, which enables image segmentation processing with a certain degree of accuracy. However, it should be noted that if the parameter settings for the layers and nodes constituting the neural network are different, the degree to which the tendency trained from the teacher data can be reproduced in the output data may differ. In other words, in many cases, the appropriate parameters differ depending on the form of implementation, and can be changed to preferred values ​​as necessary.

[0077] In addition to changing the parameters as described above, CNNs can sometimes have better characteristics by changing their configurations, such as higher accuracy in image segmentation processing, shorter image segmentation processing time, and shorter training time for machine learning models.

[0078] The CNN configuration 601 used in this embodiment is a U-net type machine learning model having an encoder function consisting of multiple layers including multiple downsampling layers and a decoder function consisting of multiple layers including multiple upsampling layers. In the U-net type machine learning model, the position information (spatial information) made ambiguous in multiple layers configured as an encoder can be used in layers of the same dimension (layers corresponding to each other) in multiple layers configured as a decoder (for example, by using skip connections).

[0079] Although not shown in the figure, as an example of a modification of the CNN configuration, for example, a batch normalization layer or an activation layer using a rectifier linear unit may be incorporated after the convolution layer.

[0080] When data is input to a trained model of such a machine learning model, data is output according to the design of the machine learning model. For example, output data that is likely to correspond to the input data according to the trends trained using teacher data is output.

[0081] In the learned model of the processing unit 222 according to this embodiment, when a tomographic image 401 is input, a boundary image 402 is output according to a tendency trained using teacher data. The processing unit 222 can detect the retinal layers and their boundaries in the tomographic image 401 based on the boundary image 402.

[0082] As shown in FIGS. 5(b) and 5(c), when the image regions are divided and learned, the processing unit 222 uses the learned model to obtain rectangular region images that are boundary images corresponding to each rectangular region. Therefore, the processing unit 222 can detect the retinal layer in each rectangular region. In this case, the processing unit 222 can generate a boundary image 402 corresponding to the input tomographic image 401 by arranging and combining each of the rectangular region images that are boundary images obtained using the learned model in the same positional relationship as each of the rectangular region images. In this case, the processing unit 222 can detect the retinal layer and its boundary in the tomographic image 401 based on the generated boundary image 402.

[0083] In step S304, when the processing unit 222 performs a detection process of the retinal layer, the process proceeds to step S305. In step S305, the display control unit 25 displays the boundary and the tomographic image detected by the processing unit 222 on the display unit 50. Here, an example of a screen displayed on the display unit 50 is shown in FIG.

[0084] 7 shows a display screen 700, which displays an SLO image 701, a thickness map 702 superimposed on the SLO image 701, an En-Face image 703, a tomographic image 711, and a retinal thickness graph 712. Retinal boundaries 715 and 716 are superimposed on the tomographic image 711.

[0085] In this embodiment, the range of the retina is from the boundary L1 between the internal limiting membrane and the nerve fiber layer to the retinal pigment epithelium layer L4, and the boundaries 715 and 716 correspond to the boundary L1 and the retinal pigment epithelium layer L4, respectively. The range of the retina is not limited to this, and may be, for example, from the boundary L1 between the internal limiting membrane and the nerve fiber layer to the choroid L6, and in this case, the boundaries 715 and 716 may correspond to the boundary L1 and the choroid L6, respectively.

[0086] Retinal thickness graph 712 is a graph showing the retinal thickness found from boundaries 715, 716. Furthermore, thickness map 702 is a representation of the retinal thickness found from boundaries 715, 716 as a color map. Note that, for the sake of explanation, color information corresponding to thickness map 702 is not shown in Fig. 7, but in reality, thickness map 702 can display the retinal thickness corresponding to each coordinate in SLO image 701 according to the corresponding color map.

[0087] The En-Face image 703 is a front image generated by projecting data in the range between the boundaries 715 and 716 in the XY direction. The front image is generated by projecting or integrating data corresponding to at least a part of the depth range of the volume data (three-dimensional tomographic image) obtained using optical interference and the depth range determined based on two reference planes onto a two-dimensional plane. The En-Face image 703 according to this embodiment is a front image generated by projecting data corresponding to a depth range (depth range between the boundaries 715 and 716) determined based on a detected retinal layer out of the volume data onto a two-dimensional plane. Note that, as a method of projecting data corresponding to a depth range determined based on two reference planes onto a two-dimensional plane, for example, a method of setting a representative value of the data within the depth range as a pixel value on the two-dimensional plane can be used. Here, the representative value can include values ​​such as an average value, a median value, or a maximum value of pixel values ​​within a range in the depth direction of the area surrounded by the two reference planes.

[0088] In addition, the depth range of the En-Face image 703 shown on the display screen 700 is not limited to the depth range between the boundaries 715, 716. The depth range of the En-Face image 703 may be, for example, a range including a predetermined number of pixels in a deeper or shallower direction based on one of the two layer boundaries 715, 716 related to the detected retinal layer. In addition, the depth range of the En-Face image 703 may be, for example, a range changed (offset) from the range between the two layer boundaries 715, 716 related to the detected retinal layer in response to an instruction from an operator.

[0089] The front image shown on the display screen 700 is not limited to an En-Face image based on a luminance value (En-Face image of luminance). The front image shown on the display screen 700 may be a motion contrast front image generated by projecting or integrating data corresponding to the depth range described above on a two-dimensional plane for motion contrast data between multiple volume data. Here, the motion contrast data is data showing changes between multiple volume data obtained by controlling the measurement light to scan multiple times in the same region (same position) of the subject's eye. At this time, the volume data is composed of multiple tomographic images obtained at different positions. Then, by obtaining data showing changes between multiple tomographic images obtained at approximately the same position at each of the different positions, the motion contrast data can be obtained as volume data. The motion contrast front image is also called an OCTA front image (En-Face image of OCTA) related to OCT angiography (OCTA) that measures the movement of blood flow, and the motion contrast data is also called OCTA data. The motion contrast data can be obtained, for example, as the decorrelation value, variance value, or maximum value divided by minimum value (maximum value / minimum value) between two tomographic images or the corresponding interference signals, and can be obtained by any known method. At this time, the two tomographic images can be obtained, for example, by controlling the measurement light to scan the same region (same position) of the subject's eye multiple times.

[0090] In addition, three-dimensional OCTA data (OCT volume data) used in generating an OCTA front image may be generated using at least a part of a common interference signal with volume data including a tomographic image for detecting retinal layers. In this case, the volume data (three-dimensional tomographic image) and the three-dimensional OCTA data can correspond to each other. Therefore, a motion contrast front image corresponding to a depth range determined based on the detected retinal layers can be generated using three-dimensional motion contrast data corresponding to the volume data.

[0091] Here, the thickness map 702, the En-Face image 703, the thickness graph 712, and the displays of the boundaries 715 and 716 are examples that can be generated by the image processing device 20 based on the boundaries and retinal layers detected by the processing unit 222. Any known method may be used as a method for generating these.

[0092] In addition to the above, the display screen 700 of the display unit 50 may be provided with a patient tab, an imaging tab, a report tab, a setting tab, etc. In this case, the contents shown on the display screen 700 of Fig. 7 will be displayed in the report tab. The display screen 700 may also display a patient information display section, an examination sort tab, an examination list, etc. In the examination list, thumbnails of fundus images, tomographic images, and OCTA images may be displayed.

[0093] Next, in step S306, the acquisition unit 21 acquires from the outside an instruction as to whether or not to end a series of processes related to the capturing of a tomographic image by the image processing system 1. This instruction can be input by the operator using the input unit 60. When the acquisition unit 21 acquires an instruction to end the processes, the image processing system 1 ends a series of processes related to this embodiment. On the other hand, when the acquisition unit 21 acquires an instruction not to end the processes, the process returns to step S302 and continues the capture.

[0094] As described above, the image processing device 20 according to this embodiment includes the acquisition unit 21 and the processing unit 222 (first processing unit). The acquisition unit 21 acquires a tomographic image of the subject's eye. The processing unit 222 executes a first detection process for detecting at least one retinal layer among a plurality of retinal layers of the subject's eye in the tomographic image by using a trained model.

[0095] When performing image segmentation processing using a trained model, for example, boundary detection can be performed appropriately according to the trained tendency even for changes in layer structure due to lesions in diseased eyes. Therefore, in the image processing device 20 according to this embodiment, by performing image segmentation processing using a trained model, boundary detection can be performed regardless of disease, site, etc., and the accuracy of boundary detection can be improved.

[0096] The image processing device 20 further includes an image processor 22 for generating a front image corresponding to at least a part of a depth range in a three-dimensional tomographic image of the subject's eye, the depth range being determined based on the detected at least one retinal layer. The image processor 22 can generate a motion contrast front image corresponding to the determined depth range by using three-dimensional motion contrast data corresponding to the three-dimensional tomographic image.

[0097] In this embodiment, a configuration has been described in which image segmentation processing is performed using one trained model, but image segmentation processing may also be performed using multiple trained models.

[0098] Since the trained model generates output data according to the tendency of learning using the teacher data as described above, the reproducibility of the learning tendency for the output data can be improved by performing learning using teacher data having similar tendencies in features. Therefore, for example, a more accurate boundary image can be generated by performing image segmentation processing on the tomographic images of the corresponding imaging sites using multiple trained models trained for each imaging site. In this case, the image processing system can detect the retinal layer with higher accuracy. In addition, in this case, it is also possible to increase the number of additional learning models, so that it is expected that the system can be upgraded to gradually improve performance.

[0099] Furthermore, the processing unit 222 may use a plurality of learning models that have been trained for each region, such as the vitreous side region, the retina region, and the sclera side region, in the tomographic image, and combine the outputs of the respective learning models to generate a final output of the processing unit 222. In this case, a more accurate boundary image can be generated for each region, and therefore the retinal layers can be detected with higher accuracy.

[0100] Furthermore, in this embodiment, the machine learning model that performs image segmentation processing has been described, but for example, a machine learning model that estimates the imaging site of a tomographic image can also be used.

[0101] In general, the configuration of a machine learning model is not limited to a configuration that outputs an image corresponding to an image that is input data. For example, a machine learning model may be configured to output a type of output data trained using teacher data for input data, or to output a probability for each of the types as a numerical value. Therefore, the format and combination of the input data and output data of the pair group that constitutes the teacher data can be suitable for the usage form, such as one being an image and the other being a numerical value, one being composed of a group of multiple images and the other being a character string, or both being images.

[0102] A specific example of training data for a machine learning model that estimates the imaging site is training data consisting of a group of pairs of a tomographic image acquired by OCT and an imaging site label corresponding to the tomographic image. Here, the imaging site label is a unique number or character string that represents the site. When a tomographic image acquired by OCT is input to a learned model trained using such training data, the imaging site label of the part captured in the image is output, or, depending on the design, the probability of each imaging site label is output.

[0103] The processing unit 222 may further use a trained model for estimating such an imaging site to estimate the imaging site of the tomographic image, and perform image segmentation processing using a trained model corresponding to the estimated imaging site or the most probable imaging site. In such a configuration, even if the acquisition unit 21 cannot acquire imaging conditions related to the imaging site of the tomographic image, the imaging site can be estimated from the tomographic image and image segmentation processing corresponding to the imaging site can be performed, thereby making it possible to detect the retinal layer with higher accuracy.

[0104] Example 2 In Example 1, an image segmentation process was performed to detect all target retinal layers from a tomographic image using a trained model. In contrast, in Example 2, boundary detection was performed using rule-based image features based on the detection result of the retinal region using the trained model.

[0105] Traditionally, when detecting the cup (optic disc cup) and disc (optic disc) in OCT images, it has been customary to detect the opening edge of Bruch's membrane. However, in cases of peripapillary chorioretinal atrophy, this detection can be difficult.

[0106] In addition, conventional rule-based image segmentation processing has low robustness against individual differences and lesions in the examined eye, and may initially misdetect the retinal region, which makes it difficult to properly detect the boundary of the inner retinal layer.

[0107] In response to this, the accuracy of boundary detection can be improved by performing image segmentation processing using a machine learning model. However, when recognizing an imaged part or detecting the boundary of a retinal layer using a machine learning model based on a machine learning algorithm such as deep learning, it is generally very difficult to collect a large number of cases of normal and diseased images with correct answers because this is a medical imaging field. Furthermore, it takes time to create correct answer data for learning.

[0108] Therefore, in this embodiment, the retinal area is detected using a trained model, and the detected retinal area is subjected to boundary detection using image features. This suppresses false detection of the retinal area and improves the detection accuracy of the inner retinal layer boundary. In addition, during the machine learning process, it is only necessary to create correct data for the retinal layer or other areas during learning, so learning can be performed efficiently.

[0109] An image processing system 8 according to this embodiment will be described below with reference to Figs. 8 to 13(d). Image processing by the image processing system according to this embodiment will be described below, focusing on the differences from the image processing according to embodiment 1. Note that the configuration and processing of the image processing system according to this embodiment that are similar to those of the image processing system 1 according to embodiment 1 will be indicated by the same reference numerals, and description thereof will be omitted.

[0110] 8 shows an example of a schematic configuration of an image processing system 8 according to this embodiment. In the image processing system 8, an image processing unit 82 of an image processing device 80 is provided with a first processing unit 822 and a second processing unit 823 instead of the processing unit 222.

[0111] The first processing unit 822 has a trained model for a machine learning model based on a machine learning algorithm such as deep learning, and detects a retinal region in a tomographic image using the trained model. The second processing unit 823 detects the boundary of the retinal layer by determining the result of image feature extraction based on a rule for the retinal region detected by the first processing unit 822.

[0112] Next, a series of processes according to this embodiment will be described with reference to Fig. 9(a) and (b). Fig. 9(a) is a flowchart of a series of processes according to this embodiment, and Fig. 9(b) is a flowchart of boundary detection processing in this embodiment. Note that the processes other than the boundary detection processing are the same as those in the first embodiment, and therefore their description will be omitted. When a tomographic image is generated in step S303, the process proceeds to step S904.

[0113] When the boundary detection process is started in step S904, the process proceeds to step S941. In step S941, the first processing unit 822 detects a retinal region in a tomographic image by using a trained model as a first boundary detection process.

[0114] Here, the machine learning model according to the present embodiment will be described with reference to Figs. 10(a) to 12. The learning data (teacher data) of the machine learning model according to the present embodiment is composed of a pair group of one or more input data and output data. An example of the teacher data is teacher data composed of a pair group of a tomographic image 1001 acquired by OCT imaging shown in Fig. 10(a) and a label image 1002 in which a label is given to an arbitrary layer from the tomographic image 1001 shown in Fig. 10(b).

[0115] Here, the labeled image is an image in which each pixel is labeled (annotated image), and in this embodiment, it refers to an image in which each pixel is given a label related to the image appearing (photographed) at that pixel. In the labeled image 1002, as examples of labels, a label 1003 on the superficial layer side (vitreous side) of the retina, a label 1004 on the inner retina, and a label 1005 on the deeper layer side (choroid side) of the retina are given. The first processing unit 822 in this embodiment detects the inner retina layer based on such a labeled image. In this embodiment, the range of the retina (range of the inner retina layer) is defined as the boundary L1 between the inner limiting membrane and the nerve fiber layer to the retinal pigment epithelium layer L4, but is not limited to this. For example, the range of the retina may be defined as the range of the boundary L1 between the inner limiting membrane and the nerve fiber layer to the inner / outer segment junction L3, the range of the boundary L1 between the inner limiting membrane and the nerve fiber layer to the Bruch's membrane L5, or the range of the boundary L1 between the inner limiting membrane and the nerve fiber layer to the choroid L6.

[0116] Furthermore, in FIG. 10(a) and (b), an example of one XZ cross section in the XY plane of the retina is shown, but the cross section is not limited to this. Although not shown, any multiple XZ cross sections in the XY plane can be learned in advance so as to be able to handle cross sections captured with various different scan patterns such as raster scan and radial scan. For example, when using data such as a tomographic image of a retina captured three-dimensionally by raster scan, volume data in which adjacent multiple tomographic images are aligned can be used as training data. In this case, it is possible to generate a pair image group at any angle from one volume data and one corresponding three-dimensional label data (three-dimensional label image). In addition, the machine learning model may be trained using images actually captured with various scan patterns as training images.

[0117] Next, images during learning will be described. A group of images, which constitute a pair group of a tomographic image 1001 and a label image 1002 constituting the training data of the machine learning model, is created by rectangular area images of a certain pixel size that correspond in positional relationship. The creation of the images will be described with reference to Figs. 11(a) to (c).

[0118] First, a case will be described where one of the pairs constituting the training data is a tomographic image 1001 and a label image 1002. In this case, as shown in Fig. 11(a), a pair is formed with a rectangular area image 1101, which is the entire tomographic image 1001, as input data, and a rectangular area image 1102, which is the entire label image 1002, as output data. Note that in the example shown in Fig. 11(a), a pair of input data and output data is formed by the entirety of each image, but the pairs are not limited to this.

[0119] 11(b), a pair may be formed by using a rectangular area image 1111 of a tomographic image 1001 as input data and a rectangular area image 1113, which is a corresponding photographed area in a label image 1002, as output data. The rectangular area images 1111 and 1113 are based on an A-scan unit. The A-scan unit may be a single A-scan unit or a number of A-scan units.

[0120] In Fig. 11(b), the A-scan unit is used as the basis, but instead of defining the entire depth direction of the image as the region, it is also possible to set parts outside the rectangular region above and below. In other words, the horizontal size of the rectangular region may be set to the size of several A-scans, and the depth size of the rectangular region may be set to be smaller than the depth size of the image.

[0121] 11(c), a pair may be formed by using a rectangular area image 1121 of the tomographic image 1001 as input data and a rectangular area image 1123, which is a corresponding photographed area in the label image 1002, as output data. In this case, the size of the rectangular area is set to a size that includes multiple labels within one rectangular area.

[0122] During learning, the scan range (photographing angle of view) and scan density (number of A-scans) are normalized to make the image sizes uniform, so that the rectangular region sizes during learning can be made constant. The rectangular region images shown in Figures 11(a) to 11(c) are examples of rectangular region sizes when learning separately.

[0123] The number of rectangular regions can be set to one in the example shown in FIG. 11(a), and to multiple in the examples shown in FIG. 11(b) and (c). For example, in the example shown in FIG. 11(b), a pair can be formed with a rectangular region image 1112 in the tomographic image 1001 as input data and a rectangular region image 1114, which is the corresponding photographed region in the label image 1002, as output data. Also, for example, in the example shown in FIG. 11(c), a pair can be formed with a rectangular region image 1122 in the tomographic image 1001 as input data and a rectangular region image 1124, which is the corresponding photographed region in the label image 1002, as output data. In this way, pairs of mutually different rectangular region images can be created from pairs of one tomographic image and one label image. Note that the pair group constituting the teacher data can be enriched by creating a large number of pairs of rectangular region images while changing the positions of the regions in the original tomographic image and label image to different coordinates.

[0124] In the examples shown in FIG. 11(b) and (c), the rectangular regions are shown discretely, but in reality, the original tomographic image and the label image can be divided into a group of rectangular region images of a certain image size that are continuous without gaps. The original tomographic image and the label image may also be divided into a group of rectangular region images at random positions that correspond to each other. In this way, by selecting images of smaller regions as rectangular regions (or strip regions) as pairs of input data and output data, a lot of pair data can be generated from the tomographic image 1001 and the label image 1002 that constitute the original pair. Therefore, the time required for training the machine learning model can be reduced. On the other hand, in a completed machine learning model that has been trained, the time required for image segmentation processing to be performed tends to be long.

[0125] Next, as an example of a machine learning model according to this embodiment, a configuration of a convolutional neural network (CNN) that performs a segmentation process on an input tomographic image will be described with reference to Fig. 12. Fig. 12 shows an example of a configuration 1201 of a machine learning model in the first processing unit 822. As the machine learning model according to this embodiment, for example, FCN or SegNet can be used as in the first embodiment. Also, depending on a desired configuration, a machine learning model that performs object recognition on a region-by-region basis as described in the first embodiment may be used.

[0126] The machine learning model shown in FIG. 12 is composed of a plurality of layers that process and output a group of input values, similar to the example of the machine learning model according to the first embodiment shown in FIG. 6. The types of layers included in the configuration 1201 of the machine learning model include a convolution layer, a downsampling layer, an upsampling layer, and a synthesis layer. Note that the configurations of these layers and the modified CNN configuration are similar to those of the machine learning model according to the first embodiment, and therefore detailed explanations thereof will be omitted. Note that the CNN configuration 1201 used in this embodiment is a U-net type machine learning model, similar to the CNN configuration 601 described in the first embodiment.

[0127] In the trained model of the first processing unit 822 according to this embodiment, when a tomographic image 1001 is input, a labeled image 1002 is output according to a tendency trained using teacher data. The first processing unit 822 can detect a retinal region in the tomographic image 1001 based on the labeled image 1002.

[0128] In addition, as shown in FIG. 11(b) and (c), when the image region is divided and learned, the first processing unit 822 uses the learned model to obtain rectangular region images that are label images corresponding to each rectangular region. Therefore, the first processing unit 822 can detect the retinal layer in each rectangular region. In this case, the first processing unit 822 arranges and combines each of the rectangular region images that are label images obtained using the learned model in the same positional relationship as each of the rectangular region images. In this way, the first processing unit 822 can generate a label image 1002 that corresponds to the input tomographic image 1001. In this case, the first processing unit 822 can also detect the retinal region in the tomographic image 1001 based on the generated label image 1002.

[0129] In step S941, when the retinal region is detected by the first processing unit 822, the process proceeds to step S942. In step S942, as a second detection process, the second processing unit 823 detects remaining boundaries in the inner retinal layer by rule-based image segmentation processing based on the retinal region detected by the first processing unit 822 in the tomographic image 1001 shown in FIG. 10(a).

[0130] The second boundary detection process by the second processing unit 823 will be described with reference to Fig. 13(a) to (d). Fig. 13(a) shows a tomographic image 1001, which is an example of an input tomographic image. Fig. 13(b) shows a labeled image 1002 output by the first processing unit 822, which is an image to which a label 1004 of the retinal region and labels 1003 and 1005 corresponding to the rest are assigned. The second processing unit 823 according to this embodiment sets the range of the retinal region indicated by the label 1004 in the labeled image 1002 as the target region for layer detection.

[0131] The second processing unit 823 can detect the target boundary by detecting the contour within the retinal region indicated by the label 1004 in the label image 1002. Fig. 13(c) shows an edge-enhanced image 1303 processed by the second processing unit 823. The processing by the second processing unit 823 will be described below. Note that, as shown in Figs. 13(c) and 13(d), the retinal layers are discontinued in the optic disc area, so boundary detection by the second processing unit 823 is not performed.

[0132] The second processing unit 823 performs noise removal and edge enhancement processing on the region corresponding to the label 1004 in the tomographic image 1001 to be processed. The second processing unit 823 applies, for example, a median filter or a Gaussian filter for the noise removal processing. The second processing unit 823 also applies a Sobel filter or a Hessian filter for the edge enhancement processing.

[0133] Here, edge enhancement processing for a two-dimensional tomographic image using a two-dimensional Hessian filter will be described. The Hessian filter can enhance the secondary local structure of a two-dimensional gray distribution based on the relationship between two eigenvalues ​​(λ1, λ2) of a Hessian matrix. Therefore, in this embodiment, the relationship between the eigenvalues ​​and eigenvectors (e1, e2) of the Hessian matrix is ​​used to enhance the two-dimensional line structure. Since the line structure in the two-dimensional tomographic image of the subject's eye corresponds to the structure of the retina layer, the structure of the retina layer can be enhanced by applying the Hessian filter.

[0134] In order to detect retinal layers with different thicknesses, the resolution of smoothing using a Gaussian function performed when calculating the Hessian matrix can be changed. In addition, when applying a two-dimensional Hessian filter, the data can be transformed to match the physical sizes of the XZ of the image before application. In the case of general OCT, the physical sizes in the XY and Z directions are different. Therefore, the filter is applied by matching the physical size of the retinal layer for each pixel. Note that the physical sizes in the XY and Z directions can be grasped from the design / configuration of the OCT device 10, so the data of the tomographic image can be transformed based on the physical sizes. In addition, if the physical size is not normalized, it can be approximately handled by changing the resolution of smoothing using a Gaussian function.

[0135] Although the processing of two-dimensional tomographic images has been described above, the subject to which the Hessian filter is applied is not limited to this. When the data structure when the tomographic image is captured is a three-dimensional tomographic image by raster scanning, a three-dimensional Hessian filter can also be applied. In this case, after the image processing unit 82 performs position alignment processing in the XZ direction between adjacent tomographic images, the second processing unit 823 can emphasize the secondary local structure of the three-dimensional gray distribution based on the relationship between the three eigenvalues ​​(λ1, λ2, λ3) of the Hessian matrix. Therefore, it is also possible to emphasize edges three-dimensionally by emphasizing the three-dimensional layer structure using the relationship between the eigenvalues ​​and eigenvectors (e1, e2, e3) of the Hessian matrix.

[0136] In the edge-enhanced image 1303, the portion where the edge is enhanced appears as a white line 1304. Note that a region in the tomographic image 1001 that does not correspond to the label 1004 can be treated as a region where edges are not detected. Also, although a configuration for performing edge enhancement processing using a Hessian filter has been described here, the processing method for edge enhancement processing is not limited to this, and any existing method may be used.

[0137] 13(d) shows a boundary image 1305 indicating the boundary of the retinal layer detected by the second processing unit 823 using the labeled image 1002 and the edge-enhanced image 1303. In the boundary image 1305, a black line 1306 indicates an example of the boundary. The process of detecting the boundary of the retinal layer from the labeled image 1002 and the edge-enhanced image 1303 by the second processing unit 823 will be described below.

[0138] The second processing unit 823 detects edge-enhanced boundaries from the edge-enhanced image 1303. In this embodiment, since the first processing unit 822 has already detected the boundary between the ILM and the NFL and the RPE, the second processing unit 823 subsequently detects the boundaries of the ISOS, the NFL, and the GCL. Although not shown, other boundaries may be detected, such as the boundary between the outer plexiform layer (OPL) and the outer nuclear layer (ONL), the boundary between the inner plexiform layer (IPL) and the inner nuclear layer (INL), the boundary between the INL and the OPL, and the boundary between the GCL and the IPL.

[0139] As a method of detecting the boundary, a plurality of points with strong edge strength are detected as boundary candidates in each A-scan, and points (points with strong edge strength) are connected as lines based on the continuity between the boundary candidates in adjacent A-scans. In addition, when points are connected as lines, the second processing unit 823 can remove outliers by evaluating the smoothness of the lines. More specifically, for example, the positions of the connected points in the Z direction are compared, and if the difference in the positions in the Z direction is greater than a predetermined threshold, the newly connected point can be determined as an outlier and excluded from the connecting process. In addition, when the outlier is removed, boundary candidates in the A-scan adjacent to the A-scan position of the excluded point may be connected as lines. Note that the method of removing outliers is not limited to this, and any existing method may be used.

[0140] The second processing unit 823 determines the corresponding boundary for each line formed by connecting the points based on the vertical distance and positional relationship in the Z direction of the boundary of the retina layer. If no boundary is detected as a result of removing outliers in each A-scan, the boundary may be obtained by interpolation from the surrounding boundaries. Alternatively, the boundary candidate may be searched for in the horizontal direction (X or Y direction) based on the edge from the surrounding boundaries, and the boundary may be determined again based on the boundary candidate searched from the surrounding boundaries.

[0141] Thereafter, the second processing unit 823 executes a process for correcting the detected boundary to make the boundary shape smooth. For example, the boundary shape may be smoothed using image features and shape features by an active contour model such as Snakes or Level Set method. In addition, the coordinate values ​​of the boundary shape may be regarded as time-series data based on a signal, and the shape may be smoothed by a smoothing process such as a Savitzky-Golay filter, a simple moving average, a weighted moving average, or an exponential moving average.

[0142] Through such processing, the second processing unit 823 can detect the retinal layer in the retinal region detected by the first processing unit 822. Note that the above-mentioned detection process of the retinal layer by the second processing unit 823 is an example, and the retinal layer can also be detected using any existing segmentation process. When the second processing unit 823 detects the retinal layer, the process proceeds to step S305. The subsequent processes are the same as those in the first embodiment, and therefore will not be described.

[0143] As described above, the image processing device 80 according to this embodiment includes the acquisition unit 21, the first processing unit 822, and the second processing unit 823. The acquisition unit 21 acquires a tomographic image of the subject's eye. The first processing unit 822 executes a first detection process for detecting at least one retinal layer of the multiple retinal layers of the subject's eye in the tomographic image by using a trained model. The second processing unit 823 executes a second detection process for detecting at least one retinal layer of the multiple retinal layers in the tomographic image without using a trained model.

[0144] Specifically, the second processing unit 823 detects, by the second detection process, at least one retinal layer other than the at least one retinal layer detected by the first detection process. In particular, in this embodiment, the first detection process is a process for detecting a layer from the boundary between the internal limiting membrane and the nerve fiber layer of the subject's eye to any one of the inner segment-outer segment junction, the retinal pigment epithelium layer, and the Bruch's membrane. The second detection process is performed after the first detection process, and is a process for detecting at least one retinal layer included in the layer detected by the first detection process, i.e., between the detected layers.

[0145] The image processing device 80 according to this embodiment can perform boundary detection regardless of disease, site, etc. Also, the accuracy of boundary detection can be improved by combining boundary detection based on image features for the area output by the machine learning model. Furthermore, in the process of machine learning, since it is only necessary to create correct answer data for the retinal layer or other parts during learning, learning can be performed efficiently.

[0146] In addition, when the number of boundaries detected by the machine learning model increases, the possibility of false positives occurring in the output label image and boundary image may increase. In contrast, in the detection of the retinal region using the machine learning model according to the present embodiment, the number of boundaries to be detected is small, so that false positives in the output label image and boundary image can be suppressed.

[0147] In this embodiment, similarly to the first embodiment, the first processing unit 822 may be configured to detect the retinal area using a plurality of machine learning models. In this case, the accuracy of the detection of the retinal area can be improved. In addition, since it is possible to increase the number of learning models, it is expected that the performance can be gradually improved by upgrading the version.

[0148] (Modification of the second embodiment) In the above-described second embodiment, an example was shown in which the first processing unit 822 detects a retinal area using a trained model as a preliminary step, prior to the detection of the retinal boundary by the second processing unit 823 based on a rule base. However, the order of the processing by the first processing unit 822 and the processing by the second processing unit 823 is not limited to this. For example, if it takes a very long time to detect the retinal area by the first processing unit 822, it is also possible to cause the second processing unit 823 to perform the detection processing of the retinal area first based on a rule base.

[0149] In such a case, the second processing unit 823 first detects the boundary between the ILM and the NFL and the RPE or ISOS using a method similar to that by the second processing unit 823 shown in the second embodiment. This is because these boundaries are areas of high brightness in the retina and are located in the shallow and deep layers of the retina. When detecting the boundary between the ILM and the NFL and the RPE or ISOS, these boundaries may be detected based on a blurred image that has been subjected to noise processing several times, since the features are more likely to be seen than other boundaries. In this case, only global features can be detected, so that erroneous detection of other boundaries can be prevented. In addition, binarization processing using a dynamic threshold may be performed on the tomographic image to limit the retinal region, and the boundary between the ILM and the NFL and the RPE or ISOS may be detected from within the retinal region. The second processing unit 823 may detect the boundary between the ILM and the NFL and the BM.

[0150] However, when detecting the retinal region, which is the first detection target, based on such rules, as described above, the robustness against individual differences and lesions in the examined eye is low, and the retinal region may be erroneously detected at first. In such a case, the subsequent detection of the inner retinal layer boundary may not be performed appropriately.

[0151] As a countermeasure, in this modification, the image processing unit 82 compares parameters for checking for erroneous detection of the retinal region, such as the discontinuity of the retinal region boundary, the local curvature, or the variance of boundary coordinates in a local region, with predetermined thresholds. If these parameters exceed the predetermined thresholds, the image processing unit 82 determines that the detection of the retinal region by the second processing unit 823 is an erroneous detection. Then, when the image processing unit 82 determines that the detection of the retinal region by the second processing unit 823 is an erroneous detection, the first processing unit 822 is configured to detect the retinal region.

[0152] According to this modified example, even if it takes a very long time for the first processing unit 822 to detect the retinal area, it is possible to ensure robustness against individual differences and lesions in the test eyes while reducing the actual processing waiting time for an examiner (operator) performing multiple examinations.

[0153] In addition, in this modification, the configuration in which the processing of the second processing unit 823 is performed before the processing of the first processing unit 822 has been described, but these processes may be started simultaneously. In this case, when the image processing unit 82 determines that the detection of the retinal area by the second processing unit 823 is an erroneous detection, the second processing unit 823 waits for the detection of the retinal area by the first processing unit 822 and performs boundary detection of the inner retinal layer. Note that, when the detection of the retinal area by the second processing unit 823 is properly performed, the processing by the first processing unit 822 can be interrupted or the processing result by the first processing unit 822 can be discarded.

[0154] Furthermore, when the first processing unit 822 and the second processing unit 823 detect a retinal region (the same retinal layer), the display control unit 25 may cause the display unit 50 to display the processing results of the detection processing by the first processing unit 822 and the second processing unit 823. In this case, the second processing unit 823 may perform boundary detection of the inner retinal layer for either of the processing results of the detection processing by the first processing unit 822 and the second processing unit 823 in response to an instruction from the operator for the processing results displayed on the display unit 50. In this case, the detection processing of the retinal region by the second processing unit 823 may be defined as a second detection processing, and the subsequent detection processing of the boundary of the inner retinal layer by the second processing unit 823 may be defined as a third detection processing.

[0155] Example 3 In the second embodiment, a retinal area is detected using a trained model, and a boundary of the inner retinal layer is detected for the detected retinal area. In contrast, in the present embodiment, the area to be detected using the trained model is not limited to the retinal area, but a characteristic area is detected with respect to the photographed site of the image, which is the input data.

[0156] Image processing by the image processing system according to this embodiment will be described below, focusing on the differences from image processing according to embodiment 2. Note that the configuration and processing procedure of the image processing system according to this embodiment are similar to those of the image processing system 8 according to embodiment 2, and therefore the same reference numerals are used to indicate them, and the description will be omitted.

[0157] The area detected by the first processing unit 822 according to this embodiment using the trained model will be described with reference to Fig. 14(a) to (d). Fig. 14(a) to (d) show examples of images of each part of the subject's eye and label images of the processing results of the first processing unit 822.

[0158] 14A shows a tomographic image 1401 of the macular region and a labeled image 1402 of the macular region obtained using a trained model. The labeled image 1402 shows a vitreous label 1403, a label 1404 ranging from ILM to ISOS, a label 1405 ranging from ISOS to RPE, a choroid label 1406, and a sclera label 1407.

[0159] Regarding the macula, in order to capture, for example, the thickness of the entire retina due to bleeding or neovascularization, photoreceptor cell defects related to visual acuity, or choroidal thinning due to pathological myopia, a label is set for each region where morphological changes are likely to occur, and the machine learning model is trained in advance. As training data, a tomographic image of the macula is used as input data, and, for example, a label image with a vitreous label, a label in the range from ILM to ISOS, a label in the range from ISOS to RPE, a choroid label, and a sclera label is used as output data. As a result, the first processing unit 822 can input a tomographic image of the macula to the trained model, thereby acquiring a label image indicating a label for each region where the above-mentioned morphological changes are likely to occur, and detecting the region in units of these labels.

[0160] FIG. 14(b) shows a tomographic image 1411 obtained by photographing the optic disc and a label image 1412 of the optic disc obtained using a trained model. The label image 1412 shows a vitreous body label 1413, a label 1414 in the range from the ILM to the NFL and GCL boundary, and a label 1415 in the range from the boundary between the NFL and GCL to the boundary between the GCL and IPL. Furthermore, the label image 1412 shows a label 1416 in the range from the boundary between the GCL and IPL to the ISOS, and a label 1417 in the range from the ISOS to the RPE. Furthermore, the label image 1412 shows a label 1418 in the range from the RPE to the deep layers, and a label 1419 in the range of the lamina cribrosa. In this case, the input data for the learning data is a tomographic image of the optic disc, and the output data is a labeled image with, for example, a vitreous label, a label for the range from the ILM to the boundary between the NFL and GCL, a label for the range from the boundary between the NFL and GCL to the boundary between the GCL and IPL, a label for the range from the boundary between the GCL and IPL to the ISOS, a label for the range from the ISOS to the RPE, a label for the range from the RPE to the deep layers, and a label for the range of the lamina cribrosa.

[0161] The front image 1421 in Fig. 14(c) is an image seen from the front direction in the XY plane when photographing the optic disc, and is an image photographed using the fundus image photographing device 30. Moreover, the labeled image 1422 in Fig. 14(c) is a labeled image obtained by using a trained model for the front image 1421 of the optic disc. The labeled image 1422 shows a label 1423 of the periphery of the optic disc, a Disc label 1424, and a Cup label 1425.

[0162] In the optic disc, loss of ganglion cells and morphological changes such as the end of the RPE (RPE-tip) or the opening of Bruch's membrane (BMO), the lamina cribrosa, and the cup and disc are likely to occur due to glaucoma. Therefore, labels are set for each of these regions and the machine learning model is trained in advance. As training data, a front image of the optic disc is used as input data, and a label image with, for example, a label of the peripheral part of the optic disc, a disc label, and a cup label is used as output data. As a result, the first processing unit 822 can input an image of the optic disc to the trained model, obtain a label image indicating a label for each region where the above-mentioned morphological changes are likely to occur, and detect the region in units of these labels.

[0163] FIG. 14(d) shows a tomographic image 1431 when the anterior segment is photographed and a label image 1432 in the anterior segment photographed using the trained model. The label image 1432 shows a cornea label 1433, an anterior chamber label 1434, an iris label 1435, and a crystalline lens label 1436. Unlike the posterior segment image, the anterior segment tomographic image 1431 is trained in advance in the machine learning model for the main regions as described above. As training data, the anterior segment tomographic image is used as input data, and for example, a label image with a cornea label, an anterior chamber label, an iris label, and a crystalline lens label is used as output data. As a result, the first processing unit 822 can input the anterior segment image to the trained model to obtain a label image indicating a label for each region where the above-mentioned morphological change is likely to occur, and detect the region in units of those labels.

[0164] 14(a), (b), and (d), the second processing unit 823 according to this embodiment detects remaining boundaries based on the regions detected by the first processing unit 822. The second processing unit 823 may also measure the thickness of the detected boundaries, layer regions sandwiched between the boundaries, or regions detected by the first processing unit 822.

[0165] In addition, the second processing unit 823 can measure each area classified by the label in the front image of Fig. 14(c) and calculate the height, width, area, and Cup / Disc ratio of each area. Note that any existing method may be used for these measurements and ratio calculations.

[0166] In this way, the second processing unit 823 according to this embodiment can execute an image processing algorithm corresponding to the region detected by the first processing unit 822, and can change the rule for applying the image processing algorithm for each region. The rule here includes, for example, the type of boundary to be detected. For example, the second processing unit 823 may perform additional boundary detection in the range from ILM to ISOS indicated by the label 1404 for the tomographic image 1401 of the macular region shown in FIG. 14(a) in the same manner as in the first embodiment. The boundary detection method may be the same as the boundary detection method in the first embodiment. In addition, the second processing unit 823 may apply an image processing algorithm and a rule to the tomographic image 1401 of the macular region to detect the boundary between ILM and NFL, the boundary between OPL and ONL, the boundary between IPL and INL, the boundary between INL and OPL, and the boundary between GCL and IPL. In this way, in the second processing unit 823, any image processing algorithm and a rule may be set for each region according to a desired configuration.

[0167] As described above, in the image processing device 80 according to this embodiment, the first processing unit 822 detects a predetermined boundary for each imaging site for the input image. Therefore, in this embodiment, the area detected using the trained model is not limited to the retina area, but is a characteristic area for the imaging site, so that it can also handle variations such as diseases. If the input image is a tomographic image of the retina, the first processing unit 822 can detect a predetermined boundary for each imaging site as a first detection process for detecting at least one retinal layer. If the input image is an image other than a tomographic image of the retina, the first processing unit 822 can detect a predetermined boundary for each imaging site as a process different from the first detection process. As in the first embodiment, the final output of the first processing unit 822 may be generated by combining the outputs of the multiple learning models trained for each area of ​​the tomographic image.

[0168] Furthermore, in the image processing device 80 according to this embodiment, predetermined shape characteristics such as the height (thickness), width, area, Cup / Disc ratio, etc. of each region of the subject eye can be measured based on the results of the first detection process or the second detection process.

[0169] Example 4 In the third embodiment, an example was shown in which the area detected using the trained model is not limited to the retina area, but a characteristic area in the part to be photographed is detected. In contrast, in the fourth embodiment, the execution of processing using the trained model is selected according to the photographing conditions under which the image was photographed, and further the area to be detected is narrowed down using the trained model.

[0170] 15 to 16(b), image processing by the image processing system 150 according to this embodiment will be described, focusing on the differences from the image processing according to embodiment 2. Note that the configuration and processing of the image processing system according to this embodiment that are similar to the configuration and processing of the image processing system 8 according to embodiment 2 will be indicated by the same reference numerals, and description thereof will be omitted.

[0171] 15 shows an example of a schematic configuration of an image processing system 150 according to this embodiment. In the image processing system 150 according to this embodiment, an image processing unit 1520 of an image processing device 152 is provided with a selection unit 1524 in addition to a tomographic image generating unit 221, a first processing unit 822, and a second processing unit 823.

[0172] The selection unit 1524 selects image processing to be performed on the tomographic image based on the group of imaging conditions acquired by the acquisition unit 21 and learning contents (teacher data) related to the trained model of the first processing unit 822. Specifically, based on the group of imaging conditions and learning of the trained model, the selection unit 1524 selects whether to detect the retinal layer only by the first processing unit 822, to detect the retinal region by the first processing unit 822 and detect the retinal layer by the second processing unit 823, or to detect the retinal layer only by the second processing unit 823. In addition, when the first processing unit 822 has a plurality of trained models, the selection unit 1524 can select the trained model to be used in the detection process by the first processing unit 822 based on the group of imaging conditions and learning contents related to the trained model of the first processing unit 822.

[0173] Next, a series of processes according to this embodiment will be described with reference to Fig. 16(a) and (b). Fig. 16(a) is a flowchart of a series of processes according to this embodiment, and Fig. 16(b) is a flowchart of boundary detection processing according to this embodiment. Note that the processes other than the boundary detection processing are similar to those in the second embodiment, and therefore will not be described. When a tomographic image is generated in step S303, the process proceeds to step S1604. When the process proceeds to step S1604, the boundary detection processing is started, and the process proceeds to step S1641.

[0174] In step S1641, the acquisition unit 21 acquires a group of imaging conditions for the generated tomographic image, and the image processing unit 1520 acquires, from the acquired group of imaging conditions, information required for executing selection of processing by the first processing unit 822 and the second processing unit 823. For example, these conditions may include an imaging part, an imaging method, an imaging area, an imaging angle of view, and an image resolution.

[0175] In step S1642, the selection unit 1524 selects whether or not to execute processing by the first processing unit 822 based on the shooting conditions acquired in step S1641. Here, as an example, a case is considered in which only two trained models for the first processing unit 822 are prepared, an optic disc model and a macular model trained using teacher data for each shooting site of the optic disc and the macular region. In this case, the first processing unit 822 is described as being unable to handle a wide-angle image (an image in which both the optic disc and the macular region are shot).

[0176] In this example, when the selection unit 1524 determines that the input image is an image of the optic disc or the macular region alone based on, for example, the name of the imaging part and the imaging angle of view information among the imaging conditions, the selection unit 1524 selects to execute processing by the first processing unit 822. As a result, the processing proceeds to step S1643. On the other hand, when the selection unit 1524 determines that the input image is an image of an area other than the optic disc or the macular region, or a wide-angle image including both the optic disc and the macular region, the selection unit 1524 selects not to execute processing by the first processing unit 822. As a result, the processing proceeds to step S1645.

[0177] In step S1643, the selection unit 1524 selects an appropriate trained model to be used by the first processing unit 822 based on the imaging conditions acquired in step S1641. In the above example, when the selection unit 1524 determines that the input image is an image of the optic disc based on, for example, the imaging part name and the imaging angle of view information, the selection unit 1524 selects a model for the optic disc. Similarly, when the selection unit 1524 determines that the input image is an image of the macular region, the selection unit 1524 selects a model for the macular region.

[0178] Although an example is shown here in which the trained model has learned only images of the optic disc and the macula, the learning content of the trained model is not limited to this. For example, a trained model trained on other parts or a trained model trained using wide-angle images including the optic disc and the macula may be used.

[0179] In addition, when trained models corresponding to the imaging method, not the imaging site, are separately prepared, the selection of the process or the selection of the trained model may be performed according to the imaging method. Examples of the imaging method include the imaging methods of SD-OCT and SS-OCT, and the image quality, imaging range, and depth of penetration in the depth direction are different depending on the difference between the two imaging methods. Therefore, appropriate processing or a trained model may be selected for images of these different imaging methods. Note that, when learning is performed together regardless of the imaging method during learning, there is no need to change the process according to the imaging method. In addition, when there is only one trained model, there is no need to select a learning model in step S1643, and therefore this process can be skipped.

[0180] In step S1644, the first processing unit 822 performs a first boundary detection process using the trained model selected in step S1643. Note that, for this process, the one described in the first to third embodiments can be used. For example, when the model for the macular area has learned about image segmentation processing of each retinal layer in the macular area, the first processing unit 822 can detect all boundaries to be detected as the first detection process, similar to the first embodiment. Also, for example, when the model for the optic disc has learned about processing to detect a retinal area in the optic disc, the first processing unit 822 can detect a retinal area as the first detection process, similar to the second embodiment. Similarly, when the model for the macular area has learned processing to detect a characteristic area of ​​the macular area as in the third embodiment, the first processing unit 822 can detect a characteristic area as the first detection process, similar to the third embodiment. Note that, the specific detection method is the same as the detection method in the first to third embodiments, and therefore will not be described.

[0181] In step S1645, the selection unit 1524 selects whether or not to execute processing by the second processing unit 823 based on the shooting conditions acquired in step S1641. If the selection unit 1524 selects to execute processing by the second processing unit 823, the process proceeds to step S1646. On the other hand, if the selection unit 1524 selects not to execute processing by the second processing unit 823, the process of step S1604 ends, and the process proceeds to step S305.

[0182] Here, an example of the selection process of the selection unit 1524 in step S1645 will be described. The case where the process is executed by the second processing unit 823 is, for example, the case where the second processing unit 823 detects a boundary based on the area detected by the first processing unit 822 as described in the second and third embodiments.

[0183] In addition, when an unlearned image is input to the first processing unit 822, the second processing unit 823 also executes processing. In this case, in step S1642, it is selected to skip the processing by the first processing unit 822, and therefore the second processing unit 823 performs boundary detection by rule-based image segmentation processing without using a learned model.

[0184] On the other hand, a case where the processing by the second processing unit 823 is not executed is, for example, a case where the first processing unit 822 can detect all target boundaries using a trained model. In this case, the processing is completed only by the first processing unit 822, so that the processing by the second processing unit 823 can be skipped.

[0185] However, even if the first processing unit 822 can detect all target boundaries using the learned model, if the second processing unit 823 performs measurement of layer thickness based on the detected boundaries, execution of processing by the second processing unit 823 may be selected. On the other hand, measurement of layer thickness based on the detected boundaries is not limited to being performed by the second processing unit 823, and may be performed in the image processing unit 1520. Therefore, even if measurement of layer thickness is performed, execution of processing by the second processing unit 823 may not be selected.

[0186] In step S1646, the selection unit 1524 selects image processing required for executing processing by the second processing unit 823 and a rule for applying the image processing, based on the imaging conditions acquired in step S1641. For example, when the input image is a tomographic image as shown in Fig. 14(a), (b), and (d), the selection unit 1524 selects processing and a rule for detecting the remaining boundary based on the region detected by the first processing unit 822.

[0187] Specifically, for example, when the input image is a tomographic image 1401 of the macular region shown in Fig. 14(a), the selection unit 1524 selects image processing and rules that allow for correct layer recognition of the macular region. Also, for example, when the input image is a tomographic image 1411 of the optic disc shown in Fig. 14(b), the selection unit 1524 selects image processing and rules that take into account exceptional processing of the optic disc region, taking into account the influences of the Bruch membrane opening (BMO), the lamina cribrosa, the Cup, the Disc, etc. Furthermore, for example, when the input image is a tomographic image 1431 of the anterior segment shown in Fig. 14(d), the selection unit 1524 selects image processing and rules that allow for further layer recognition of the cornea region.

[0188] Furthermore, if, in addition to or solely in addition to detecting layer boundaries in step S1647, measurement of the thickness of the detected boundaries or layer regions sandwiched between the boundaries is performed, the selection unit 1524 can also select image processing required for such image measurement function.

[0189] In step S1647, the second processing unit 823 detects the boundary and / or measures the detected boundary or area. As in the second and third embodiments, the process of detecting the boundary in the area based on the area detected by the first processing unit 822 and the process of measuring the boundary will not be described.

[0190] Here, an example of processing performed by the second processing unit 823 when an image for which the trained model of the first processing unit 822 has not yet been trained is input will be described. In this case, since there is no candidate for the retinal region detected from the input image, the second processing unit 823 first detects the boundary between the ILM and the NFL and the RPE or ISOS as described in the modified example of the second embodiment.

[0191] After detecting the boundary between the ILM and the NFL and the RPE or ISOS, the second processing unit 823 detects the remaining boundaries based on the area between these boundaries. This detection process is similar to the detection process in the second and third embodiments, and therefore a description thereof will be omitted. After the second processing unit 823 executes these processes, the process proceeds to step S305. Note that, as in the modified example of the second embodiment, the second processing unit 823 may first detect the boundary between the ILM and the NFL and the BM. Further, since the subsequent processes are similar to the processes in the first to third embodiments, a description thereof will be omitted.

[0192] As described above, in the image processing device 152 according to this embodiment, the acquisition unit 21 acquires the imaging conditions related to the tomographic image of the subject's eye. The image processing device 152 further includes a selection unit 1524 that selects a process based on the imaging conditions. The selection unit 1524 selects at least one of the first detection process and the second detection process based on the imaging conditions.

[0193] For this reason, in the image processing device 152 according to the present embodiment, whether or not boundary detection is possible using the trained model executed by the first processing unit 822, and whether or not boundary detection processing using image features executed by the second processing unit 823 needs to be performed are determined based on the shooting conditions. This allows appropriate processing to be performed according to the input image even when boundary detection using the trained model only corresponds to a specific image. Therefore, even if the trained model does not correspond to various image patterns, the boundary detection processing can be reliably performed. Therefore, the accuracy of boundary detection can be improved by using at least one of the machine learning models at various stages of maturity created by machine learning in combination with an image processing method that performs boundary detection of the retinal layer by judging the result of image feature extraction on a rule basis.

[0194] The first processing unit 822 also includes a plurality of trained models that have been machine-learned using different training data. Furthermore, the first processing unit 822 executes the first detection process using a trained model that has been machine-learned using training data corresponding to the shooting conditions, among the plurality of trained models.

[0195] According to this embodiment, the retinal layer can be detected using an appropriate learning model based on the shooting conditions, and more appropriate processing can be performed according to the input image. In addition, since it is possible to add additional learning models, it is expected that the performance can be gradually improved by upgrading. Furthermore, according to the selection unit 1524, the image processing and rules used in the second processing unit 823 can be selected based on the shooting conditions, and more appropriate processing can be performed according to the input image.

[0196] In this embodiment, in steps S1642 and S1645, the selection of the processes by the first processing unit 822 and the second processing unit 823 is performed separately, but the procedure for selecting the processes is not limited to this. For example, the selection unit 1524 may be configured to select, in one step, the process by only the first processing unit 822, the process by only the second processing unit 823, or the processes by the first processing unit 822 and the second processing unit 823.

[0197] (Modification of the fourth embodiment) In the fourth embodiment, it has been shown that appropriate processing is possible in various cases, such as when the first processing unit 822 and the second processing unit 823 share the boundary detection or the like, or when only one of them can complete the processing, etc. In contrast, the first processing unit 822 and the second processing unit 823 may execute the same processing, for example, processing to detect the same boundary, in parallel.

[0198] As an example of such a case, consider a case where the first processing unit 822 can detect all target boundaries using a trained model, and the second processing unit 823 can detect all target boundaries including the retina region that is the first detection target. In this case, the first processing unit 822 and the second processing unit 823 output the results of detecting the respective boundaries separately.

[0199] Since these detection results are the results of processing by a trained model and image processing by a rule base, respectively, there may be a difference between the two results. Therefore, in this modified example, the display control unit 25 can display the two results side by side on the display unit 50, switch between them, or display them overlapping each other. In addition, the image processing unit 1520 can determine whether the two results match or mismatch, and the display control unit 25 can highlight the mismatched part and display it on the display unit 50. In this case, the reliability of the layer detection can be shown to the operator. Furthermore, the display control unit 25 may display the mismatched part on the display unit 50 so that the operator can select a more satisfactory result according to the operator's instructions.

[0200] Example 5 In the second to fourth embodiments, a retinal region is detected using a trained model, and the boundary of the inner retinal layer is detected for the detected retinal region on a rule-based basis. In contrast, in the fifth embodiment, the region detected using the trained model is corrected based on medical features.

[0201] 17 to 19(d), image processing by the image processing system 170 according to this embodiment will be described, focusing on the differences from the image processing according to embodiment 2. Note that the configuration and processing of the image processing system according to this embodiment that are similar to the configuration and processing of the image processing system 8 according to embodiment 2 will be indicated by the same reference numerals, and the description will be omitted.

[0202] 17 shows an example of a schematic configuration of an image processing system 170 according to this embodiment. In an image processing unit 1720 of an image processing device 172 in the image processing system 170 according to this embodiment, a correction unit 1724 is provided in addition to a tomographic image generating unit 221, a first processing unit 822, and a second processing unit 823.

[0203] The correction unit 1724 corrects the labeled area based on the medical characteristics of the eye for the labeled image obtained by the first processing unit 822 using the trained model. This enables the image processing unit 1720 to more appropriately detect the retina area and the feature area.

[0204] Next, a series of processes according to this embodiment will be described with reference to Figures 18(a) and 18(b). Figure 18(a) is a flowchart of a series of processes according to this embodiment, and Figure 18(b) is a flowchart of boundary detection according to this embodiment. Note that the processes other than the boundary detection process are similar to those in the second embodiment, and therefore will not be described. When a tomographic image is generated in step S303, the process proceeds to step S1804. When the process proceeds to step S1804 and the process by the first processing unit 822 is executed in step S941, the process proceeds to step S1841.

[0205] In step S1841, the correction unit 1724 corrects the retinal region detected in step S941 by the first processing unit 822. More specifically, for the labeled image obtained by the first processing unit 822 using the trained model, the correction unit 1724 corrects the labeled region based on the medical characteristics of the eye.

[0206] Here, the correction process by the correction unit 1724 according to this embodiment will be described with reference to Fig. 19(a) to (d). Fig. 19(a) shows an example of a tomographic image 1901 to be input to the first processing unit 822. Fig. 19(b) shows an example of a labeled image 1902 obtained by the first processing unit 822 using a trained model with the tomographic image 1901 as input. The labeled image 1902 shows a label 1904 of the inner retina, a label 1903 on the superficial layer side (vitreous side) of the retina, and a label 1905 on the deeper layer side (choroid side) of the retina.

[0207] The labeling is based on the label setting at the time of learning of the trained model. Therefore, the type of label is not limited to this, and multiple labels may be set in the retinal layer as shown in Example 3. Even in this case, the correction process according to this embodiment can be applied.

[0208] In this embodiment, the first processing unit 822 performs image segmentation processing on a pixel-by-pixel basis using a trained model. Therefore, partial erroneous detection may occur as shown by labels 1903' and 1904' in Fig. 19(b). The correction unit 1724 corrects these erroneous detections based on medical characteristics of the eye.

[0209] In the first processing unit 822, a labeling process is performed for each detected label, and adjacent pixels having the same label are integrated as one region. In this embodiment, three types of labels are assigned: a label 1904 for the inner layer of the retina, a label 1903 for the superficial layer side (vitreous side) of the retina, and a label 1905 for the deeper layer side (choroid side) of the retina. Since the target image is an image obtained by capturing a tomographic image of the retina, the order in which these labels appear is label 1903, label 1904, and label 1905 from the top of the image. In the case of an EDI (Enhanced Depth Imaging) mode in which the choroid side is emphasized and captured, the retina is inverted and captured, so the order in which the labels appear is label 1905, label 1904, and label 1903 from the top of the image.

[0210] As described above, since the image input to the first processing unit 822 is a tomographic image of the retina, the positional relationship of the labels can be estimated based on medical characteristics from the conditions at the time of shooting and the shooting site. Therefore, the correction unit 1724 specifies the detection result for each labeled area, and corrects the area considered to be a false positive to an area estimated based on the medical characteristics.

[0211] Specifically, the correction unit 1724 identifies regions, for example, starting with the largest labeled region, and determines that a labeled region whose area is equal to or smaller than a threshold or that is spatially far from an already identified region is a false positive. The correction unit 1724 then resets the label information of the region determined to be a false positive. An example of this case is shown in FIG. 19(c). The region 1910 shown in FIG. 19(c) indicates a region in which the label information of the region indicated by the labels 1903' and 1904', which is a region determined to be a false positive by the correction unit 1724, has been reset.

[0212] The correction unit 1724 assigns label information estimated from surrounding label information to the region 1910 whose label information has been reset. In the example shown in Fig. 19(c), the label 1903 is assigned to the region 1910 surrounded by the label 1903, and the label 1905 is assigned to the region 1910 surrounded by the label 1905.

[0213] As a result of these processes by the correction unit 1724, a final labeled image 1920 is output as shown in Fig. 19(d), which enables the image processing unit 1720 to more appropriately detect the retina region.

[0214] After the correction process by the correction unit 1724 is performed, the process proceeds to step S942. In step S942, the second processing unit 823 performs a second boundary detection process based on the corrected retinal area, similar to the second embodiment. The subsequent processes are similar to those in the second embodiment, and therefore will not be described.

[0215] As described above, the image processing device 172 according to this embodiment further includes the correction unit 1724 that corrects the structure of the retinal layer detected by the first processing unit 822 based on medical characteristics of the retinal layer.

[0216] Therefore, in the image processing device 172 according to the present embodiment, it is possible to correct the area detected using the trained model by using medical features. Therefore, even when detecting an image on a pixel-by-pixel basis, it is possible to reduce false positives.

[0217] In this embodiment, the correction process by the correction unit 1724 is added to the process according to the second embodiment, but the correction process may be added to the process according to the third or fourth embodiment.

[0218] Example 6 In Examples 1 to 5, the boundary of the inner retina and the retinal region are detected for the captured tomographic image using a trained model. In contrast, in this embodiment, a high-quality image is generated by improving the image quality of the tomographic image using another trained model, and boundary detection and region detection are performed on the high-quality image using the trained model according to Examples 1 and 2. Note that the improvement in image quality in this embodiment includes noise reduction, conversion to colors and gradations that make it easier to observe the subject, improvement in resolution and spatial resolution, and enlargement of the image size while suppressing a decrease in resolution.

[0219] Image processing by the image processing system according to this embodiment will be described below, focusing on the differences from image processing according to embodiment 2. Note that the configuration and processing procedure of the image processing system according to this embodiment are similar to those of the image processing system 8 according to embodiment 2, and therefore the same reference numerals are used to indicate them, and the description will be omitted.

[0220] In this embodiment, the first processing unit 822 performs processing to improve the image quality of the input image using a trained model for an image quality improvement model, which is a machine learning model different from the machine learning model for detecting the retina region. The image quality improvement model is a trained model that is trained in advance using appropriate teacher data for a machine learning model based on an arbitrary machine learning algorithm, and outputs an image with improved image quality of the input image.

[0221] An example of training data for the image quality improvement model according to this embodiment is shown in Fig. 20. In Fig. 20, a tomographic image 2001 is an example of a tomographic image acquired by OCT imaging, and a tomographic image 2002 is an example of a tomographic image obtained by image quality improvement processing of the tomographic image 2001. The tomographic image 2001 is an example of input data, and the tomographic image 2002 is an example of output data, and the training data is made up of a group of pairs made up of these images.

[0222] In addition, the high image quality processing may include aligning tomographic images captured multiple times at the same spatial position, and averaging the aligned tomographic images. In addition, the high image quality processing is not limited to averaging, and may be, for example, processing using a smoothing filter, maximum a posteriori probability estimation processing (MAP estimation processing), tone conversion processing, etc. In addition, the high image quality processed image may be, for example, an image subjected to filter processing such as noise removal and edge emphasis, or an image with contrast adjusted to change a low brightness image to a high brightness image. Furthermore, the output data of the teacher data related to the high image quality model may be a high image quality image, and may be a tomographic image captured using an OCT device with higher performance than the OCT device used to capture the tomographic image that is the input data, or a tomographic image captured under high load settings.

[0223] The first processing unit 822 inputs the tomographic image acquired by OCT imaging into the high image quality model trained using such teacher data, and acquires a high image quality tomographic image. The first processing unit 822 can acquire a high image quality volume tomographic image by inputting a volume tomographic image obtained by three-dimensionally scanning the retina by raster scanning into the high image quality model.

[0224] The first processing unit 822 receives a high-quality image acquired using the image quality improvement model as an input, and detects a retina region or a feature region using a trained model, as in the second to fifth embodiments.

[0225] Furthermore, the second processing unit 823 can detect a retinal layer based on the high-quality image acquired by the first processing unit 822 and the detected retinal region and characteristic region.

[0226] As described above, in the image processing device 80 according to this embodiment, the first processing unit 822 performs a first detection process on a tomographic image whose image quality has been improved using the learned model.

[0227] As a result, the image processing device 80 according to the present embodiment can improve the image quality of the input image using the trained model of the machine learning model, and detect the retinal layer for the image with the improved image quality. Therefore, the retinal layer can be detected using an image with improved image quality such as noise reduction, and false detection can be reduced.

[0228] In this embodiment, the process of improving the image quality of the tomographic image, which is the input image, is added to the process of the second embodiment. However, the process of improving the image quality may be added to the processes of the first embodiment or any of the third to fifth embodiments.

[0229] In the present embodiment, the image quality improvement model that performs image quality improvement is a machine learning model different from the machine learning model that performs the detection process. However, the machine learning model may be configured to perform both the image quality improvement and the detection process by having the machine learning model that performs the detection process learn the image quality improvement process.

[0230] In this embodiment, the first processing unit 822 generates a high-quality image by improving the image quality of the tomographic image using a trained model (high-quality model) related to image quality improvement processing. However, the component that generates a high-quality image using the high-quality model is not limited to the first processing unit 822. For example, a third processing unit (high-quality unit) separate from the first processing unit 822 may be provided, and the third processing unit may generate a high-quality image using the high-quality model. Therefore, the first processing unit 822 or the third processing unit can function as an example of a generating unit that generates a tomographic image with higher image quality than the tomographic image from the tomographic image using a trained model for image quality improvement. The third processing unit and the high-quality model may be configured as a software module executed by a processor such as a CPU, MPU, GPU, or FPGA, or may be configured as a circuit that performs a specific function such as an ASIC.

[0231] Example 7 Next, an image processing device 80 according to a seventh embodiment will be described with reference to Figs. 21(a) to 23. In the sixth embodiment, the first processing unit 822 performed a first detection process on a tomographic image whose image quality has been improved using the high image quality model, and detected a retinal region or a characteristic region. In this regard, the first processing unit 822 may perform high image quality processing on other images using the high image quality model, and the display control unit 25 may display various high image quality images on the display unit 50. For example, the first processing unit 822 may perform high image quality processing on a luminance En-Face image or an OCTA front image, etc., generated based on information on the retinal layer (e.g., a boundary image) detected by the first detection process and the second detection process. In addition, the display control unit 25 can display at least one of the tomographic image, the luminance En-Face image, and the OCTA front image, which have been subjected to high image quality processing by the first processing unit 822, on the display unit 50. The image to be displayed after being improved in image quality may be an SLO image, a fundus image acquired by a fundus camera, etc., or a fluorescent fundus image. The SLO image here is a front image of the fundus acquired by an SLO (Scanning Laser Ophthalmoscope) optical system (not shown).

[0232] Here, the learning data of the high image quality model for performing the high image quality processing of various images is, for various images, similar to the learning data of the high image quality model according to the sixth embodiment, with the image before the high image quality processing as input data and the image after the high image quality processing as output data. Note that, as with the sixth embodiment, the high image quality processing on the learning data may be, for example, an averaging process, a process using a smoothing filter, a maximum a posteriori probability estimation process (MAP estimation process), a tone conversion process, or the like. In addition, the image after the high image quality processing may be, for example, an image subjected to a filter process such as noise removal and edge emphasis, or an image with contrast adjusted to change a low brightness image to a high brightness image. Furthermore, the output data of the teacher data related to the high image quality model may be an image of high image quality, and may be an image captured using an OCT device with higher performance than the OCT device used to capture the image as the input data, or an image captured under high load settings.

[0233] Further, the image quality improvement model may be prepared for each type of image to be subjected to image quality improvement processing. For example, an image quality improvement model for a tomographic image, an image quality improvement model for a luminance En-Face image, and an image quality improvement model for an OCTA front image may be prepared. Furthermore, the image quality improvement model for the luminance En-Face image and the image quality improvement model for the OCTA front image may be a trained model that comprehensively trains images of different depth ranges for the depth range (generation range) related to image generation. For example, as shown in FIG. 21(a), images of the superficial layer (Im2110), deep layer (Im2120), outer layer (Im2130), and choroidal vascular network (Im1940) may be included as images of the different depth ranges. Furthermore, the image quality improvement model for the luminance En-Face image and the image quality improvement model for the OCTA front image may be prepared as a plurality of image quality improvement models that train images for each different depth range.

[0234] In addition, when preparing a high quality model for a tomographic image, it may be a trained model that comprehensively trains tomographic images obtained at different positions in the sub-scanning (Y) direction. The tomographic images Im2151 to Im2153 shown in FIG. 21(b) are examples of tomographic images obtained at different positions in the sub-scanning direction. However, in the case of images obtained by photographing different locations (for example, the center of the macula, the center of the optic disc), the images may be trained separately for each location, or may be trained together regardless of the photographed location. The tomographic images to be trained may include a luminance tomographic image and a motion contrast data tomographic image. However, since the image feature values ​​are significantly different between the luminance tomographic image and the motion contrast data tomographic image, they may be trained separately as high quality models.

[0235] Image processing by the image processing system according to this embodiment will be described below, focusing on the differences from image processing according to embodiment 6. Note that the configuration and processing procedure of the image processing system according to this embodiment are similar to those of the image processing system 8 according to embodiment 6, so they are indicated by the same reference numerals and descriptions thereof will be omitted.

[0236] In this embodiment, an example in which the display control unit 25 displays on the display unit 50 an image that has been subjected to image quality improvement processing by the first processing unit 822 will be described. In this embodiment, the description will be given using Figs. 22(a) and (b), but the display screen is not limited to this. Image quality improvement processing (image quality improvement processing) can be applied to a display screen that displays multiple images obtained at different dates and times side by side, such as in a follow-up observation. Image quality improvement processing can also be applied to a display screen on which an examiner checks whether or not image capture has been successful immediately after capture, such as an image capture confirmation screen. The display control unit 25 can display on the display unit 50 multiple high-quality images generated by the first processing unit 822 and low-quality images that have not been subjected to image quality improvement. In addition, the display control unit 25 can display on the display unit 50 low-quality images and high-quality images selected in response to an examiner's instruction from among multiple high-quality images and low-quality images that have not been subjected to image quality improvement displayed on the display unit 50. In addition, the image processing device 80 can output the low-quality images and high-quality images selected in response to an examiner's instruction to the outside.

[0237] 22(a) and (b), an example of a display screen 2200 of the interface according to this embodiment will be shown below. The display screen 2200 shows the entire screen, and displays a patient tab 2201, an imaging tab 2202, a report tab 2203, and a setting tab 2204. The diagonal lines in the report tab 2203 indicate the active state of the report screen. In this embodiment, an example of displaying the report screen will be described.

[0238] The report screen shown in Fig. 22(a) shows an SLO image Im2205, OCTA front images Im2207 and 2208, an En-Face image Im2209 of luminance, cross-sectional images Im2211 and 2212, and a button 2220. In addition, an OCTA front image Im2206 corresponding to the OCTA front image Im2207 is superimposed on the SLO image Im2205. Furthermore, boundary lines 2213 and 2214 of the depth ranges of the OCTA front images Im2207 and Im2208 are superimposed on the cross-sectional images Im2211 and 2212, respectively. The button 2220 is a button for designating the execution of image quality improvement processing. The button 2220 may be a button for instructing the display of a high-image-quality image, as described later.

[0239] In this embodiment, image quality improvement processing is performed by specifying button 2220, or whether or not to perform the processing is determined based on information saved (stored) in a database. First, an example will be described in which the display of a high-image-quality image and a low-image-quality image are switched by specifying button 2220 in response to an instruction from the examiner. Note that, hereinafter, the target image for image quality improvement processing will be described as an OCTA front image.

[0240] The depth range of the OCTA front images Im2207 and Im2208 may be determined using information on the retinal layers detected by the first detection process and the second detection process. The depth range may be, for example, a range between two layer boundaries related to the detected retinal layers, or may be a range including a predetermined number of pixels in a deeper or shallower direction based on one of the two layer boundaries related to the detected retinal layers. The depth range may be, for example, a range changed (offset) from the range between the two layer boundaries related to the detected retinal layers in response to an instruction from an operator.

[0241] When the examiner designates the report tab 2203 and transitions to the report screen, the display control unit 25 displays the low-quality OCTA front images Im2207 and Im2208. After that, when the examiner designates the button 2220, the first processing unit 822 executes image quality improvement processing on the OCTA front images Im2207 and Im2208 displayed on the screen. After the image quality improvement processing is completed, the display control unit 25 displays the high-quality image generated by the first processing unit 822 on the report screen. Note that since the OCTA front image Im2206 is an image in which the OCTA front image Im2207 is superimposed on the SLO image Im2205, the display control unit 25 can also display an image that has been subjected to image quality improvement processing for the OCTA front image Im2206. In addition, the display control unit 25 can change the display of the button 2220 to an active state, and display such that it is clear that the image quality improvement processing has been executed.

[0242] Here, the execution of the process in the first processing unit 822 does not need to be limited to the timing when the examiner designates the button 2220. Since the types of OCTA front images Im2207 and Im2208 to be displayed when the report screen is opened are known in advance, the first processing unit 822 may execute the image quality improvement process when the displayed screen transitions to the report screen. Then, the display control unit 25 may display the high-quality image on the report screen at the timing when the button 2220 is pressed. Furthermore, the types of images to be subjected to the image quality improvement process in response to an instruction from the examiner or when transitioning to the report screen do not need to be two types. The process may be performed on multiple OCTA front images such as the surface layer (Im2110), deep layer (Im2120), outer layer (Im2130), and choroidal vascular network (Im2140) shown in FIG. 21(a) that are likely to be displayed. In this case, the image that has been subjected to the image quality improvement process may be temporarily stored in a memory or in a database.

[0243] Next, a case where the image quality improvement process is performed based on information stored (recorded) in the database will be described. When the state of performing the image quality improvement process is stored in the database, when the report screen is transitioned to, the display control unit 25 causes the display unit 50 to display the high-image-quality image obtained by the first processing unit 822 performing the image quality improvement process by default. Then, the display control unit 25 can be configured to display the button 2220 in an active state by default, so that the examiner can understand that the high-image-quality image obtained by performing the image quality improvement process is being displayed. If the examiner wants to display a low-image-quality image before the image quality improvement process, the examiner can cancel the active state by designating the button 2220, so that the display control unit 25 can display the low-image-quality image on the display unit 50. At this time, if the examiner wants to return the displayed image to a high-image-quality image, the examiner can designate the button 2220 to activate it, so that the display control unit 25 can display the high-image-quality image on the display unit 50 again.

[0244] The execution of high image quality processing for the database is specified by hierarchy, such as common to all data stored in the database, and for each piece of photographed data (each examination). For example, when a state in which high image quality processing is executed for the entire database is stored, a state in which the examiner does not execute high image quality processing for each piece of photographed data (each examination) can be stored. In this case, the individual photographed data in which a state in which high image quality processing is not executed is stored can be displayed in a state in which high image quality processing is not executed the next time it is displayed. According to this configuration, when the execution of high image quality processing is not specified for each piece of photographed data (each examination), processing can be executed based on information specified for the entire database. Also, when it is specified for each piece of photographed data (each examination), processing can be executed individually based on that information.

[0245] In order to save the execution state of the image quality improvement process for each piece of photographed data (each examination), a user interface (e.g., a save button) not shown may be used. When transitioning to other photographed data (other examination) or other patient data (e.g., changing to a display screen other than the report screen in response to an instruction from the examiner), the execution state of the image quality improvement process may be saved based on the display state (e.g., the state of the button 2220).

[0246] In this embodiment, an example is shown in which OCTA front images Im2207 and Im2208 are displayed as OCTA front images, but the OCTA front images to be displayed can be changed according to the examiner's designation. Therefore, a description will be given of how to change the image to be displayed when the execution of image quality improvement processing is designated (when the button 2220 is in an active state).

[0247] The image to be displayed can be changed using a user interface (e.g., a combo box) not shown. For example, when the examiner changes the type of image from the superficial layer to the choroidal vascular network, the first processing unit 822 performs image quality improvement processing on the choroidal vascular network image, and the display control unit 25 displays the high-quality image generated by the first processing unit 822 on the report screen. That is, the display control unit 25 may change the display of the high-quality image in the first depth range to the display of the high-quality image in the second depth range that is at least partially different from the first depth range, in response to an instruction from the examiner. At this time, the display control unit 25 may change the display of the high-quality image in the first depth range to the display of the high-quality image in the second depth range by changing the first depth range to the second depth range in response to an instruction from the examiner. Note that, for an image that is likely to be displayed at the time of transition to the report screen as described above, if a high-quality image has already been generated, the display control unit 25 may display the generated high-quality image.

[0248] The method of changing the type of image is not limited to the above, and it is also possible to generate an OCTA front image with a different depth range set by changing the reference layer or offset value, and to display a high-quality image obtained by performing image quality improvement processing on the generated OCTA front image. In this case, when the reference layer or offset value is changed, the first processing unit 822 performs image quality improvement processing on an arbitrary OCTA front image, and the display control unit 25 displays the high-quality image on the report screen. Note that the reference layer and offset value can be changed using a user interface (e.g., a combo box or a text box) not shown. The depth range (generation range) of the OCTA front image can also be changed by dragging (moving the layer boundary) one of the boundary lines 2213 and 2214 superimposed and displayed on the tomographic images Im2211 and Im2212, respectively.

[0249] When the boundary line is changed by dragging, the execution command of the image quality improvement process is executed continuously. Therefore, the first processing unit 822 may always process the execution command, or may execute the command after the layer boundary is changed by dragging. Alternatively, the execution command of the image quality improvement process may be executed continuously, but when the next command arrives, the previous command may be canceled and the latest command may be executed.

[0250] Note that the high image quality processing may take a relatively long time. Therefore, regardless of the timing at which the command is executed, it may take a relatively long time until the high image quality image is displayed. Therefore, from the time when the depth range for generating the OCTA front image is set according to an instruction from the examiner until the high image quality image is displayed, a low image quality OCTA front image (low image quality image) corresponding to the set depth range may be displayed. That is, when the depth range is set, a low image quality OCTA front image (low image quality image) corresponding to the set depth range is displayed, and when the high image quality processing is completed, the display of the low image quality OCTA front image may be changed to the display of a high image quality image. Also, from the time when the depth range is set until the high image quality image is displayed, information indicating that the high image quality processing is being performed may be displayed. Note that these processes are not limited to the configuration applied when it is assumed that the execution of the high image quality processing has already been specified (the button 2220 is in an active state). For example, when the execution of the high image quality processing is instructed according to an instruction from the examiner, these processes can be applied even before the high image quality image is displayed.

[0251] In this embodiment, an example has been shown in which OCTA front images Im2207 and 2108 relating to different layers are displayed as OCTA front images, and low-image-quality and high-image-quality images are switched between and displayed, but the images displayed are not limited to this. For example, an OCTA front image with low image quality may be displayed side by side as the OCTA front image Im2207, and an OCTA front image with high image quality may be displayed side by side as the OCTA front image Im2208. When images are switched between and displayed, it is easy to compare parts that have changed because the images are switched at the same location, and when images are displayed side by side, it is easy to compare the entire images because the images can be displayed simultaneously.

[0252] Next, the execution of image quality improvement processing during screen transition will be described with reference to Figs. 22(a) and (b). Fig. 22(b) is an example of a screen in which the OCTA front image Im2207 in Fig. 22(a) is enlarged and displayed. In Fig. 22(b), a button 2220 is displayed similarly to Fig. 22(a). The screen transition from Fig. 22(a) to Fig. 22(b) is performed by, for example, double-clicking the OCTA front image Im2207, and the screen transition from Fig. 22(b) to Fig. 22(a) is performed by pressing the close button 2230. Note that the method of screen transition is not limited to that shown here, and a user interface (not shown) may also be used.

[0253] If execution of high image quality processing is specified at the time of screen transition (button 2220 is active), that state is maintained even when the screen transitions. That is, when a high image quality image is displayed on the screen of FIG. 22(a) and a transition is made to the screen of FIG. 22(b), the high image quality image is also displayed on the screen of FIG. 22(b). Then, button 2220 is made active. The same is true when transitioning from FIG. 22(b) to FIG. 22(b). In FIG. 22(b), it is also possible to switch the display to a low image quality image by specifying button 2220.

[0254] Regarding the screen transition, the transition is not limited to the screen shown here, but can be made while maintaining the display state of the high-quality image, as long as the transition is to a screen that displays the same photographed data, such as a display screen for follow-up observation or a display screen for panorama. That is, an image corresponding to the state of the button 2220 on the display screen before the transition can be displayed on the display screen after the transition. For example, if the button 2220 on the display screen before the transition is in an active state, a high-quality image is displayed on the display screen after the transition. Also, for example, if the active state of the button 2220 on the display screen before the transition is released, a low-quality image is displayed on the display screen after the transition. Note that, when the button 2220 on the display screen for follow-up observation becomes active, multiple images obtained at different dates and times (different examination dates) displayed side by side on the display screen for follow-up observation may be switched to a high-quality image. That is, when the button 2220 on the display screen for follow-up observation becomes active, the multiple images obtained at different dates and times may be configured to be reflected all at once.

[0255] An example of the display screen for follow-up observation is shown in FIG. 23. When a tab 2301 is selected in response to an instruction from the examiner, a display screen for follow-up observation is displayed as shown in FIG. 23. At this time, the examiner can change the depth range of the OCTA front image by selecting a desired set from the preset depth range sets displayed in list boxes 2302 and 2303. For example, the superficial retina is selected in list box 2302, and the deep retina is selected in list box 2303. The analysis results of the OCTA front image of the superficial retina are displayed in the upper display area, and the analysis results of the OCTA front image of the deep retina are displayed in the lower display area. When a depth range is selected, the analysis results of multiple OCTA front images in the selected depth range are collectively changed for multiple images at different dates and times to a parallel display.

[0256] At this time, when the display of the analysis results is deselected, the display may be changed to a parallel display of multiple OCTA front images taken at different times. Then, when a button 2220 is selected in response to an instruction from the examiner, the display of multiple OCTA front images is changed to a display of multiple high-quality images in a batch.

[0257] Furthermore, when the display of the analysis results is in a selected state, when the button 2220 is designated in response to an instruction from the examiner, the display of the analysis results of the multiple OCTA front images is collectively changed to the display of the analysis results of the multiple high-quality images. Here, the display of the analysis results may be a display in which the analysis results are superimposed on the image with an arbitrary transparency. In this case, the change from the display of the image to the display of the analysis results may be, for example, a change to a state in which the analysis results are superimposed on the displayed image with an arbitrary transparency. Furthermore, the change from the display of the image to the display of the analysis results may be, for example, a change to a display of an image (e.g., a two-dimensional map) obtained by blending the analysis results and the image with an arbitrary transparency.

[0258] In addition, the type of layer boundary and the offset position used to specify the depth range can be changed collectively from the user interfaces 2305 and 2306, respectively. Note that the user interfaces 2305 and 2306 for changing the type of layer boundary and the offset position are merely examples, and any other type of interface may be used. Note that the depth ranges of multiple OCTA front images at different dates and times may be changed collectively by displaying a tomographic image together and moving the layer boundary data superimposed on the tomographic image according to an instruction from the examiner. In this case, multiple tomographic images at different dates and times may be displayed side by side, and when the above-mentioned movement is made on one tomographic image, the layer boundary data may be moved in the same way on the other tomographic images.

[0259] Moreover, the image projection method and the presence or absence of projection artifact suppression processing may be changed by selecting from a user interface such as a context menu.

[0260] Alternatively, a selection button 2307 may be selected to display a selection screen (not shown), and an image selected from an image list displayed on the selection screen may be displayed. Note that an arrow 2304 displayed at the top of Fig. 23 is a mark indicating the currently selected examination, and the baseline examination is the examination selected during follow-up imaging (the image at the leftmost side in Fig. 23). Of course, a mark indicating the baseline examination may be displayed on the display unit.

[0261] Also, when the "Show Difference" checkbox 2308 is specified, a measurement value distribution (map or sector map) for the reference image is displayed on the reference image. Furthermore, in this case, a difference measurement value map between the measurement value distribution calculated for the reference image and the measurement value distribution calculated for the image displayed in the area is displayed in the area corresponding to the other examination dates. As the measurement result, a trend graph (a graph of the measurement value for the image on each examination date obtained by measuring the change over time) may be displayed on the report screen. That is, time series data (for example, a time series graph) of multiple analysis results corresponding to multiple images at different dates and times may be displayed. In this case, analysis results for dates and times other than the multiple dates and times corresponding to the multiple images displayed may also be displayed as time series data in a state that can be distinguished from the multiple analysis results corresponding to the multiple images displayed (for example, the color of each point on the time series graph differs depending on whether the image is displayed or not). Also, a regression line (curve) of the trend graph or a corresponding formula may be displayed on the report screen.

[0262] In this embodiment, the OCTA front image has been described, but the image to which the processing according to this embodiment is applied is not limited to this. The image related to the processing such as display, image quality improvement, and image analysis according to this embodiment may be an En-Face image of luminance. Furthermore, it may be not only an En-Face image but also a different image such as a tomographic image by B-scan, an SLO image, a fundus image, or a fluorescent fundus image. In this case, the user interface for executing the image quality improvement processing may be one that instructs the execution of the image quality improvement processing for a plurality of different types of images, or one that instructs the execution of the image quality improvement processing by selecting an arbitrary image from a plurality of different types of images.

[0263] For example, when a tomographic image obtained by a B-scan is displayed with high image quality, the tomographic images Im2211 and Im2212 shown in FIG. 22(a) may be displayed with high image quality. Furthermore, the tomographic images with high image quality may be displayed in the area where the OCTA front images Im2207 and Im2208 are displayed. Only one or more tomographic images with high image quality may be displayed. When multiple tomographic images are displayed, tomographic images acquired at different positions in the sub-scanning direction may be displayed, and when multiple tomographic images obtained by, for example, a cross scan are displayed with high image quality, images in different scanning directions may be displayed. Furthermore, when multiple tomographic images obtained by, for example, a radial scan are displayed with high image quality, a partially selected multiple tomographic images (for example, two tomographic images at positions symmetrical to each other with respect to a reference line) may be displayed. Furthermore, multiple tomographic images may be displayed on a display screen for follow-up observation as shown in FIG. 23, and instructions for image quality improvement and analysis results (for example, the thickness of a specific layer) may be displayed by a method similar to the above-mentioned method. Furthermore, image quality improvement processing may be performed on the tomographic images based on information stored in a database using a method similar to that described above.

[0264] Similarly, when the SLO image is to be displayed with high image quality, for example, the SLO image Im2205 may be displayed with high image quality. Furthermore, when the luminance En-Face image is to be displayed with high image quality, for example, the luminance En-Face image 2209 may be displayed with high image quality. Furthermore, a plurality of SLO images and luminance En-Face images may be displayed on a display screen for follow-up observation as shown in FIG. 23, and instructions for image quality improvement and analysis results (for example, the thickness of a specific layer, etc.) may be displayed by a method similar to the above-mentioned method. Furthermore, image quality improvement processing may be performed on the SLO image and the luminance En-Face image based on information stored in a database by a method similar to the above-mentioned method. Note that the display of the tomographic image, the SLO image, and the luminance En-Face image is an example, and these images may be displayed in any manner according to a desired configuration. Furthermore, at least two or more of the OCTA front image, the tomographic image, the SLO image, and the luminance En-Face image may be displayed with high image quality by a single instruction.

[0265] With this configuration, the display control unit 25 can display the image that has been subjected to high image quality processing by the first processing unit 822 according to this embodiment on the display unit 50. At this time, as described above, if at least one of a plurality of conditions related to the display of a high image quality image, the display of an analysis result, and the depth range of a displayed front image is in a selected state, the selected state may be maintained even if the display screen is transitioned.

[0266] Also, as described above, when at least one of the multiple conditions is in a selected state, the at least one selected state may be maintained even if another condition is changed to a selected state. For example, when the display of the analysis result is in a selected state, the display control unit 25 may change the display of the analysis result of the low image quality image to the display of the analysis result of the high image quality image in response to an instruction from the examiner (for example, when the button 2220 is designated). Also, when the display of the analysis result is in a selected state, the display control unit 25 may change the display of the analysis result of the high image quality image to the display of the analysis result of the low image quality image in response to an instruction from the examiner (for example, when the designation of the button 2220 is released).

[0267] Furthermore, when the display of the high-quality image is in a non-selected state, the display control unit 25 may change the display of the analysis result of the low-quality image to that of the low-quality image in response to an instruction from the examiner (for example, when the designation of the display of the analysis result is released). When the display of the high-quality image is in a non-selected state, the display control unit 25 may change the display of the low-quality image to that of the analysis result of the low-quality image in response to an instruction from the examiner (for example, when the designation of the display of the analysis result is released). When the display of the high-quality image is in a selected state, the display control unit 25 may change the display of the analysis result of the high-quality image to that of the high-quality image in response to an instruction from the examiner (for example, when the designation of the display of the analysis result is released). When the display of the high-quality image is in a selected state, the display control unit 25 may change the display of the high-quality image to that of the analysis result of the high-quality image in response to an instruction from the examiner (for example, when the designation of the display of the analysis result is released).

[0268] Consider also a case where the display of the high-quality image is in a non-selected state and the display of the first type of analysis result is in a selected state. In this case, the display control unit 25 may change the display of the first type of analysis result of the low-quality image to the display of the second type of analysis result of the low-quality image in response to an instruction from the examiner (for example, when the display of the second type of analysis result is specified). Also consider a case where the display of the high-quality image is in a selected state and the display of the first type of analysis result is in a selected state. In this case, the display control unit 25 may change the display of the first type of analysis result of the high-quality image to the display of the second type of analysis result of the high-quality image in response to an instruction from the examiner (for example, when the display of the second type of analysis result is specified).

[0269] In addition, as described above, the display screen for follow-up observation may be configured so that these display changes are reflected collectively on a plurality of images obtained at different dates and times. Here, the display of the analysis result may be a display in which the analysis result is superimposed on the image with an arbitrary transparency. In this case, the change to the display of the analysis result may be, for example, a change to a state in which the analysis result is superimposed on the displayed image with an arbitrary transparency. In addition, the change to the display of the analysis result may be, for example, a change to a display of an image (e.g., a two-dimensional map) obtained by blending the analysis result and the image with an arbitrary transparency.

[0270] In this embodiment, the first processing unit 822 generates a high-quality image by improving the image quality of the tomographic image using a trained model (high-quality model) related to image quality improvement processing. However, the component that generates a high-quality image using the high-quality model is not limited to the first processing unit 822. For example, a third processing unit (high-quality unit) separate from the first processing unit 822 may be provided, and the third processing unit may generate a high-quality image using the high-quality model. In this case, the third processing unit and the high-quality model may be configured as a software module executed by a processor such as a CPU, MPU, GPU, or FPGA, or may be configured as a circuit that performs a specific function such as an ASIC.

[0271] (Modifications of Examples 6 and 7) In the sixth and seventh embodiments, the display control unit 25 can display on the display unit 50 an image selected from the high-quality image generated by the first processing unit 822 and the input image in response to an instruction from the examiner. The display control unit 25 may also switch the display on the display unit 50 from the captured image (input image) to a high-quality image in response to an instruction from the examiner. That is, the display control unit 25 may change the display of a low-quality image to a high-quality image in response to an instruction from the examiner. The display control unit 25 may also change the display of a high-quality image to a low-quality image in response to an instruction from the examiner.

[0272] Furthermore, the first processing unit 822 may start image quality improvement processing (input of an image to the image quality improvement engine) by the image quality improvement engine (image quality improvement model) in response to an instruction from the examiner, and the display control unit 25 may display the generated high-image quality image on the display unit 50. On the other hand, when an input image is captured by the imaging device (OCT device 10), the first processing unit 822 may automatically generate a high-image quality image based on the input image using the image quality improvement engine, and the display control unit 25 may display the high-image quality image on the display unit 50 in response to an instruction from the examiner. Here, the image quality improvement engine includes a learned model that performs the above-mentioned image quality improvement processing (image quality improvement processing).

[0273] In addition, these processes can be performed similarly for the output of the analysis results. That is, the display control unit 25 may change the display of the analysis results of the low-quality image to the display of the analysis results of the high-quality image in response to an instruction from the examiner. Also, the display control unit 25 may change the display of the analysis results of the high-quality image to the display of the analysis results of the low-quality image in response to an instruction from the examiner. Furthermore, the display control unit 25 may change the display of the analysis results of the low-quality image to the display of the low-quality image in response to an instruction from the examiner. Also, the display control unit 25 may change the display of the analysis results of the low-quality image to the display of the analysis results of the low-quality image in response to an instruction from the examiner. Furthermore, the display control unit 25 may change the display of the analysis results of the high-quality image to the display of the high-quality image in response to an instruction from the examiner. Also, the display control unit 25 may change the display of the analysis results of the high-quality image to the display of the analysis results of the high-quality image in response to an instruction from the examiner.

[0274] Furthermore, the display control unit 25 may change the display of the analysis results of the low-quality image to a display of other types of analysis results of the low-quality image in response to an instruction from the examiner. Also, the display control unit 25 may change the display of the analysis results of the high-quality image to a display of other types of analysis results of the high-quality image in response to an instruction from the examiner.

[0275] Here, the display of the analysis result of the high-quality image may be a display in which the analysis result of the high-quality image is superimposed on the high-quality image with an arbitrary transparency. Also, the display of the analysis result of the low-quality image may be a display in which the analysis result of the low-quality image is superimposed on the low-quality image with an arbitrary transparency. In this case, the change to the display of the analysis result may be, for example, a change to a state in which the analysis result is superimposed on the displayed image with an arbitrary transparency. Also, the change to the display of the analysis result may be, for example, a change to a display of an image (e.g., a two-dimensional map) obtained by blending the analysis result and the image with an arbitrary transparency.

[0276] In this modification, the first processing unit 822 generates a high-quality image by improving the image quality of the tomographic image using a trained model (high-quality model) related to image quality improvement processing. However, the component that generates a high-quality image using the high-quality model is not limited to the first processing unit 822. For example, a third processing unit other than the first processing unit 822 may be provided, and the third processing unit may generate a high-quality image using the high-quality model. In this case, the third processing unit and the high-quality model may be configured as a software module executed by a processor such as a CPU, MPU, GPU, or FPGA, or may be configured as a circuit that performs a specific function such as an ASIC.

[0277] In the seventh embodiment, an image that has been subjected to image quality improvement processing using an image quality improvement model is displayed according to the active state of the button 2220 on the display screen. In response to this, an analysis value using the result of the image segmentation processing using the trained model may be configured to be displayed according to the active state of the button 2220. In this case, for example, when the button 2220 is inactive (the image segmentation processing using the trained model is not selected), the display control unit 25 causes the display unit 50 to display an analysis result using the result of the image segmentation processing performed by the second processing unit 823. In response to this, when the button 2220 is activated, the display control unit 25 causes the display unit 50 to display an analysis result using the result of the image segmentation processing performed by the first processing unit 822 alone or by the first processing unit 822 and the second processing unit 823.

[0278] In this configuration, analysis results using the results of image segmentation processing without using a trained model and analysis results using the results of image segmentation processing using a trained model are switched and displayed depending on the active state of the button. Since these analysis results are based on the results of processing using a trained model and rule-based image processing, respectively, there may be differences between the two results. Therefore, by switching between these analysis results, the examiner can compare the two and use the more convincing analysis result for diagnosis.

[0279] In addition, when the image segmentation process is switched, for example, when the image to be displayed is a tomographic image, the numerical value of the layer thickness analyzed for each layer may be switched and displayed. In addition, for example, when a tomographic image divided by color or hatching pattern for each layer is displayed, a tomographic image in which the shape of the layer has changed according to the result of the image segmentation process may be switched and displayed. Furthermore, when a thickness map is displayed as the analysis result, a thickness map in which the color indicating the thickness has changed according to the result of the image segmentation process may be displayed. In addition, a button for specifying the image quality improvement process and a button for specifying the image segmentation process using a trained model may be provided separately, or either one of them may be provided, or both buttons may be provided as a single button.

[0280] Also, the switching of the image segmentation process may be performed based on information stored (recorded) in a database, similar to the switching of the image quality improvement process described above. Note that, with regard to the process at the time of screen transition, the switching of the image segmentation process may be performed similar to the switching of the image quality improvement process described above.

[0281] In the first to seventh embodiments, the acquisition unit 21 acquires the interference signal acquired by the OCT device 10 and the three-dimensional tomographic data generated by the tomographic image generating unit 221. However, the configuration in which the acquisition unit 21 acquires these signals and data is not limited to this. For example, the acquisition unit 21 may acquire these signals from a server or an imaging device connected to the image processing device 20 via a LAN, a WAN, the Internet, or the like. In this case, the process related to imaging can be omitted and the three-dimensional tomographic data that has already been captured can be acquired. Then, the boundary detection process can be performed in steps S304, S904, and the like. Therefore, the series of processing times from the acquisition of tomographic information to the display of a front image, a thickness map, and the like can be shortened.

[0282] The trained model for image segmentation and the trained model for image quality improvement used by the processing unit 222 and the first processing unit 822 can be provided in the image processing device 20, 80, 152. The trained model may be configured, for example, by a software module executed by a processor such as a CPU, MPU, GPU, or FPGA, or may be configured by a circuit that performs a specific function such as an ASIC. In addition, these trained models may be provided in another server device connected to the image processing device 20, 80, 152. In this case, the image processing device 20, 80, 152 can use the trained model by connecting to a server or the like equipped with the trained model via any network such as the Internet. Here, the server equipped with the trained model may be, for example, a cloud server, a fog server, an edge server, or the like.

[0283] In addition, in the second to seventh embodiments, the label image in which each pixel is labeled has been described as the label image. However, the label image in which each region is labeled may be used as the label image.

[0284] The configuration of the OCT device 10 is not limited to the above configuration, and some of the components included in the OCT device 10 may be separate from the OCT device 10. In addition, the user interface such as buttons and the display layout are not limited to those shown above.

[0285] According to the above-described first to seventh embodiments and their modifications, it is possible to detect the boundaries of retinal layers regardless of the disease, the site, and the like.

[0286] Example 8 In the medical field, image diagnosis using images acquired by various imaging devices is performed to identify the disease of a subject or observe the extent of the disease. In the radiology field, the types of imaging devices include X-ray devices, X-ray computed tomography devices (CT), magnetic resonance imaging devices (MRI), and positron emission tomography devices (PET). In the ophthalmology field, the types of imaging devices include fundus cameras, scanning laser ophthalmoscopes (SLO), optical coherence tomography (OCT) devices, and OCT angiography (OCTA) devices.

[0287] Image diagnosis is basically performed by medical professionals by observing lesions depicted in images, but in recent years, improvements in image analysis technology have made it possible to obtain various information useful for diagnosis. For example, image analysis can now detect small lesions that may be overlooked and assist medical professionals, quantitatively measure the shape and volume of lesions, and even identify diseases without medical professionals' observation.

[0288] There are various image analysis techniques, but a process called image segmentation processing, which is a process for identifying regions such as organs and lesions depicted in an image, is necessary for implementing many image analysis techniques. Hereinafter, for the sake of simplicity, image segmentation processing will also be simply referred to as segmentation processing.

[0289] Conventional image segmentation processing is performed by an image processing algorithm based on medical knowledge and image characteristics regarding target organs and lesions, as in Patent Document 1. However, in actual medical settings, images acquired from an imaging device may not be captured clearly due to various factors such as the pathology of the subject, the imaging environment of the imaging device, and the photographer's lack of skill. Therefore, in conventional image segmentation processing, target organs and lesions are not depicted as expected, and specific regions cannot be extracted with high accuracy.

[0290] Specifically, for example, in an image obtained by photographing a diseased eye with retinal layer loss, bleeding, white spots, neovascularization, etc., using an OCT device, the shape of the retinal layer may be depicted irregularly. In such a case, erroneous detection may occur in the area detection process of the retinal layer, which is a type of image segmentation process.

[0291] One of the purposes of the following embodiments 8 to 19 is to provide a medical image processing device, a medical image processing method, and a program that can perform image segmentation processing with higher accuracy than conventional image segmentation processing.

[0292] <Terminology explanation> Here, terms used in this disclosure will be explained.

[0293] In the network in this specification, each device may be connected by a wired or wireless line. Here, the lines connecting each device in the network include, for example, a dedicated line, a local area network (hereinafter referred to as LAN) line, a wireless LAN line, an Internet line, Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0294] The medical image processing device may be composed of two or more devices capable of communicating with each other, or may be composed of a single device. Each component of the medical image processing device may be composed of a software module executed by a processor such as a CPU, MPU, GPU, or FPGA. Each component may be composed of a circuit that performs a specific function such as an ASIC. Each component may be composed of a combination of any other hardware and any software.

[0295] Moreover, the medical image processed by the medical image processing device or the medical image processing method is a tomographic image of the subject acquired by using an OCT device. Here, the OCT device may include a time domain OCT (TD-OCT) device or a Fourier domain OCT (FD-OCT) device. The Fourier domain OCT device may include a spectral domain OCT (SD-OCT) device or a swept wavelength OCT (SS-OCT) device. The OCT device may include any OCT device such as a wavefront compensation OCT (AO-OCT) device using a wavefront compensation optical system, a line OCT device in which the measurement light irradiated on the subject is formed in a line shape, or a full-field OCT device in which the measurement light is formed in a planar shape.

[0296] The medical images include tomographic images of the subject's eye (examined eye). The tomographic images of the examined eye are not limited to tomographic images of the retina in the posterior part of the examined eye, but also include tomographic images of the anterior part of the examined eye, the eye chamber, etc. In addition, when the OCT device is used in an endoscope, etc., a tomographic image of the subject's skin or organs may be used as a medical image to be processed by the medical image processing device or medical image processing method according to the following embodiments.

[0297] The image management system is a device and system that receives and stores images captured by an imaging device such as an OCT device or processed images. The image management system can transmit images in response to a request from a connected device, perform image processing on stored images, and request image processing from other devices. The image management system can include, for example, a picture archiving and communication system (PACS). In particular, the image management system according to the following embodiment includes a database that can store various information such as information on subjects associated with received images and imaging times. The image management system is connected to a network, and can transmit and receive images, convert images, and transmit and receive various information associated with stored images in response to a request from other devices.

[0298] The imaging conditions are various information at the time of imaging of an image acquired by an imaging device. The imaging conditions include, for example, information on the imaging device, information on the facility where imaging was performed, information on the examination related to imaging, information on the photographer, and information on the subject. The imaging conditions also include, for example, imaging date and time, imaging site name, imaging area, imaging angle of view, imaging method, image resolution and gradation, image size, applied image filter, and information on the data format of the image. The imaging area may include a peripheral area shifted from a specific imaging site, an area including multiple imaging sites, and the like. The imaging method may include any imaging method of OCT, such as a spectral domain method or a wavelength sweep method.

[0299] The photographing conditions can be stored in a data structure constituting an image, or can be stored as photographing condition data separate from the image, or can be stored in a database or image management system related to the photographing device. Therefore, the photographing conditions can be acquired by a procedure corresponding to a storage means of the photographing conditions of the photographing device. Specifically, the photographing conditions can be acquired, for example, by analyzing the data structure of the image output by the photographing device, acquiring photographing condition data corresponding to the image, or accessing an interface for acquiring the photographing conditions from a database related to the photographing device.

[0300] Depending on the imaging device, there may be imaging conditions that cannot be acquired because they are not saved. For example, the imaging device may not have a function to acquire or save a specific imaging condition, or such a function may be disabled. In addition, for example, imaging conditions may not be saved because they are not related to the imaging device or imaging. Furthermore, for example, imaging conditions may be hidden, encrypted, or cannot be acquired without the right. However, even imaging conditions that are not saved may be acquired. For example, image analysis may be performed to identify the imaging part name and imaging area.

[0301] The region label image refers to a labeled image in which a region label is attached to each pixel. Specifically, as shown in Fig. 24, it is an image Im2420 in which an arbitrary region is divided by a group of pixel values ​​(hereinafter, region label values) that can specify an arbitrary region among a group of regions depicted in an image Im2410 acquired by an imaging device. Here, the arbitrary region that can be specified includes a region of interest (ROI: Region Of Interest) and a volume of interest (VOI: Volume Of Interest), etc.

[0302] When a coordinate group of pixels having an arbitrary region label value is identified from image Im2420, a coordinate group of pixels depicting a corresponding region such as a retinal layer can be identified in image Im2410. Specifically, for example, when the region label value indicating the ganglion cell layer constituting the retina is 1, a coordinate group having a pixel value of 1 is identified among pixels in image Im2420, and a pixel group corresponding to the coordinate group is extracted from image Im2410. This makes it possible to identify the region of the ganglion cell layer in image Im2410.

[0303] In some embodiments, a process of performing a reduction or enlargement process on the region label image is included. In this case, the image completion process method used for reducing or enlarging the region label image uses a nearest neighbor method or the like so as not to erroneously generate undefined region label values ​​or region label values ​​that should not exist at the corresponding coordinates.

[0304] Image segmentation processing is processing for identifying regions called ROI or VOI, such as organs or lesions depicted in an image, for use in image diagnosis or image analysis. For example, image segmentation processing can identify a group of regions of layers constituting the retina from an image acquired by OCT imaging of the posterior segment. If no region to be identified is depicted in the image, the number of identified regions is 0. If multiple region groups to be identified are depicted in the image, the number of identified regions may be multiple, or it may be one region surrounding the group of regions.

[0305] The identified region group is output as information that can be used in other processes. Specifically, for example, the coordinate group of the pixel group constituting each of the identified region groups can be output as a numerical data group. Also, for example, the coordinate group indicating a rectangular region, an elliptical region, a rectangular parallelepiped region, an ellipsoid region, or the like including each of the identified region groups can be output as a numerical data group. Furthermore, for example, the coordinate group indicating a straight line, a curve, a plane, or a curved surface that is on the boundary of the identified region group can be output as a numerical data group. Also, for example, a region label image indicating the identified region group can be output.

[0306] In the following, when the accuracy of the image segmentation process is expressed as high or when the image label image has high accuracy, it means that the rate at which the region is correctly identified is high. Conversely, when the accuracy of the image segmentation process is expressed as low, it means that the rate at which the region is incorrectly identified is high.

[0307] An image without region label is a type of region label image that does not contain information corresponding to ROI or VOI for use in image diagnosis or image analysis. As a specific example, we will explain a case where we want to know the region of the ganglion cell layer that constitutes the retina depicted in a medical image for use in image analysis.

[0308] Here, the region label value indicating the region of the ganglion cell layer is 1, and the region label value indicating the other parts is 0. When a region label image corresponding to a certain medical image is generated by image segmentation processing or the like, if the ganglion cell layer is not depicted in the medical image, all pixel values ​​of the region label image are 0. In this case, the region label image is an unlabeled image because there is no region with a region label value of 1 corresponding to the ROI of the ganglion cell layer to be used for image analysis in the region label image. Note that, depending on the settings and implementation form, the unlabeled image may not be an image, but may be a numerical data group indicating a coordinate group having the same information as an image.

[0309] Here, the machine learning model refers to a learning model based on a machine learning algorithm. Specific examples of machine learning algorithms include nearest neighbor methods, naive Bayes methods, decision trees, and support vector machines. In addition, deep learning, which uses a neural network to generate features and connection weighting coefficients for learning, can also be used. Any of the above algorithms that can be used can be appropriately applied to the learning model according to the embodiment. A trained model is a model in which a machine learning model according to an arbitrary machine learning algorithm is trained (learned) in advance using appropriate teacher data (learning data). However, the trained model is not limited to a model that does not undergo further learning, and can also undergo additional learning. The teacher data is composed of one or more pairs of input data and output data. The format and combination of the input data and output data of the pairs that constitute the teacher data may be suitable for a desired configuration, such as one being an image and the other being a numerical value, one being composed of a group of multiple images and the other being a character string, or both being images.

[0310] Specifically, for example, there is teacher data (hereinafter, first teacher data) configured by a pair group of an image acquired by OCT and an imaging part label corresponding to the image. The imaging part label is a unique numerical value or character string representing the part. Another example of teacher data is teacher data (hereinafter, second teacher data) configured by a pair group of an image acquired by OCT imaging of the posterior segment and a region label image of the retinal layer corresponding to the image. Still another example of teacher data is teacher data (hereinafter, third teacher data) configured by a pair group of a noisy low-quality image acquired by normal OCT imaging and a high-quality image that has been imaged multiple times by OCT and processed to improve image quality.

[0311] When input data is input to a trained model, output data according to the design of the trained model is output. The trained model outputs output data that is likely to correspond to the input data, for example, according to a trend trained using teacher data. The trained model can also output a numerical value representing the possibility that each type of output data corresponds to the input data, for example, according to a trend trained using teacher data.

[0312] Specifically, for example, when an image acquired by OCT is input to a learned model trained with the first teacher data, the learned model outputs the imaging site label of the imaging site captured in the image, or outputs the probability for each imaging site label. Also, for example, when an image depicting a retinal layer acquired by OCT imaging of the posterior eye segment is input to a learned model trained with the second teacher data, the learned model outputs a region label image for the retinal layer depicted in the image. Furthermore, for example, when a noisy low-quality image acquired by normal OCT imaging is input to a learned model trained with the third teacher data, the learned model outputs a high-quality image equivalent to an image captured multiple times by OCT and processed for high image quality.

[0313] Machine learning algorithms include deep learning techniques such as convolutional neural networks (CNNs). In deep learning techniques, the degree to which trends trained using teacher data can be reproduced in output data may differ depending on the parameter settings for the layers and nodes that make up the neural network.

[0314] For example, in a trained model of deep learning using the first teacher data, if more appropriate parameters are set, the probability of outputting a correct imaging site label may be higher. Also, in a trained model using the second teacher data, if more appropriate parameters are set, a region label image with higher accuracy may be output. Furthermore, in a trained model of deep learning using the third teacher data, if more appropriate parameters are set, a higher quality image may be output.

[0315] Specifically, the parameters in the CNN can include, for example, the filter kernel size, the number of filters, the stride value, and the dilation value set for the convolution layer, as well as the number of nodes output by the fully connected layer. The parameter group and the number of training epochs can be set to values ​​that are suitable for the usage form of the trained model based on the teacher data. For example, based on the teacher data, a parameter group and the number of epochs can be set that can output a correct imaging site label with a higher probability, output a region label image with higher accuracy, or output an image with higher image quality.

[0316] Here, an example of a method for determining such a parameter group and the number of epochs is shown. First, 70% of the pair groups constituting the teacher data are set for training, and the remaining 30% are set for evaluation, randomly. Next, the trained model is trained using the pair groups for training, and at the end of each epoch of training, a training evaluation value is calculated using the pair groups for evaluation. The training evaluation value is, for example, the average value of a group of values ​​obtained by evaluating the output when the input data constituting each pair is input to the trained model during training, and the output data corresponding to the input data, using a loss function. Finally, the parameter group and the number of epochs when the training evaluation value is the smallest are determined as the parameter group and the number of epochs of the trained model. In this way, by dividing the pair groups constituting the teacher data into those for training and those for evaluation and determining the number of epochs, it is possible to prevent the trained model from overlearning the pair groups for training.

[0317] An image segmentation engine is a module that performs image segmentation processing and outputs a region label image corresponding to an input image. Examples of input images include an OCT B-scan image and a three-dimensional tomographic image (three-dimensional OCT volume image). Examples of region label images include a region label image that indicates each layer of the retina when the input image is an OCT B-scan image, and a region label image that indicates a three-dimensional region that indicates each layer of the retina when the input image is an OCT three-dimensional tomographic image.

[0318] In the image processing method constituting the image segmentation processing method in the following embodiment, processing is performed using a trained model according to various machine learning algorithms such as deep learning. The image processing method may perform not only machine learning algorithms but also any other existing processing. The image processing includes, for example, various image filter processing, matching processing using a database of region label images corresponding to similar images, image registration processing of reference region label images, and knowledge-based image processing.

[0319] In particular, there is a configuration 2500 shown in FIG. 25 as an example of a convolutional neural network (CNN) that performs image segmentation processing on a two-dimensional image Im2510 input as an input image to generate a region label image Im2520. The CNN configuration 2500 includes a plurality of layers that process and output a group of input values. As shown in FIG. 25, the types of layers included in the configuration 2500 include a convolution layer, a downsampling layer, an upsampling layer, and a merger layer. The CNN configuration 2500 used in this embodiment is a U-net type machine learning model, similar to the CNN configuration 601 described in the first embodiment.

[0320] The convolution layer is a layer that performs convolution processing on input value groups according to parameters such as the set filter kernel size, the number of filters, the stride value, and the dilation value. The downsampling layer is a layer that performs processing to reduce the number of output value groups to be less than the number of input value groups by thinning or synthesizing the input value groups. Specifically, the processing performed in the downsampling layer includes, for example, Max Pooling processing.

[0321] The upsampling layer is a layer that performs a process to make the number of output values ​​greater than the number of input values ​​by duplicating the input values ​​or adding values ​​interpolated from the input values. A specific example of the process performed in the upsampling layer is linear interpolation. The synthesis layer is a layer that inputs value groups, such as the output values ​​of a certain layer or pixel values ​​that make up an image, from multiple sources and performs a process of synthesizing them by linking or adding them.

[0322] In addition, as parameters set for the convolutional layers included in the configuration of the CNN, for example, the filter kernel size is set to 3 pixels wide and 3 pixels high, and the number of filters is set to 64, which enables image segmentation processing with a certain degree of accuracy. However, it should be noted that if the parameter settings for the layers and nodes constituting the neural network are different, the degree to which the tendency trained from the teacher data can be reproduced in the output data may differ. In other words, in many cases, the appropriate parameters for each layer group and each node group differ depending on the embodiment, and may be changed as necessary.

[0323] In addition, depending on the embodiment, in addition to the above-mentioned method of changing parameters, the configuration of the CNN may be changed to obtain better characteristics of the CNN. Better characteristics include, for example, high accuracy of image segmentation processing, short time for image segmentation processing, and short time required for training a learned model. Examples of changes in the configuration of the CNN include incorporating a batch normalization layer or an activation layer using a rectifier linear unit after the convolution layer.

[0324] As the machine learning model used by the image segmentation engine, for example, FCN, SegNet, or the like may be used as in the first embodiment. Also, depending on a desired configuration, a machine learning model that performs object recognition on a region-by-region basis as described in the first embodiment may be used.

[0325] In addition, when a one-dimensional image, a three-dimensional image, or a four-dimensional image needs to be processed, the kernel size of the filter may correspond to one dimension, three dimensions, or four dimensions. Here, a four-dimensional image includes, for example, a three-dimensional moving image or an image in which parameters at each pixel position of a three-dimensional image are displayed in different hues.

[0326] The image segmentation process may be performed using only one image processing technique, or may be performed using a combination of two or more image processing techniques. Furthermore, multiple image segmentation techniques may be performed to generate multiple region label images.

[0327] In some embodiments, an input image is divided into small regions, and an image segmentation process is performed on each of the small regions to obtain a group of region label images of the small regions, and the group of region label images of the small regions is combined to generate a region label image. If the input image is a three-dimensional image, the small region may be a three-dimensional image smaller than the input image, a two-dimensional image, or a one-dimensional image. If the input image is a two-dimensional image, the small region may be a two-dimensional image smaller than the input image, or a one-dimensional image. In some embodiments, a plurality of region label image groups may be output.

[0328] In addition, parameters may be input to the image segmentation engine together with the input image. In this case, the input parameters may include a parameter specifying the extent of the range in which the image segmentation process is performed, such as an upper limit of the size of a lesion, and a parameter specifying an image filter size used in the image processing method. Note that, depending on the embodiment, the image segmentation engine may output other images or coordinate data groups capable of identifying regions, instead of the region label image.

[0329] When a plurality of image segmentation processing methods are implemented or when image segmentation processing is implemented on a plurality of small region groups, the processing time can be reduced by performing the image segmentation processing in parallel.

[0330] Note that when using some image processing methods, such as image processing using CNN, attention must be paid to the image size. Specifically, it should be noted that different image sizes may be required for the input image and the output region label image to address issues such as the periphery of the region label image not being segmented with sufficient accuracy.

[0331] For the sake of clarity, it will not be specified in the following examples, but when an image segmentation engine is adopted that requires different image sizes for the image input to the image segmentation engine and the image output, the image size is adjusted appropriately. Specifically, the image size is adjusted by padding or combining the surrounding shooting area of ​​the input image for an input image such as an image used as teacher data for training a learned model or an image input to the image segmentation engine. Note that the area to be padded is filled with a certain pixel value, filled with a neighboring pixel value, or mirror padded according to the characteristics of the image segmentation processing method so that the image segmentation processing can be performed effectively.

[0332] The imaging location estimation engine is a module that estimates the imaging site and imaging area of ​​an input image. The imaging location estimation engine can output the location of the imaging site and imaging area depicted in the input image, or the probability that the imaging site and imaging area are the imaging site and imaging area for each imaging site label and imaging area label of a required level of detail.

[0333] Depending on the imaging device, the imaging part and imaging area may not be saved as imaging conditions, or may not be acquired by the imaging device and saved. Even if the imaging part and imaging area are saved, the imaging part and imaging area may not be saved at the required level of detail. For example, the imaging part may only be saved as "posterior eye", and it is not clear whether it is "macular", "optic disc", "macular and optic disc", or "other". In another example, the imaging part may only be saved as "breast", and it is not clear whether it is "right breast", "left breast", or "both". Therefore, by using the imaging part estimation engine, it is possible to estimate the imaging part and imaging area of ​​the input image in these cases.

[0334] In the image and data processing method constituting the estimation method of the imaging location estimation engine, processing is performed using a trained model according to various machine learning algorithms such as deep learning. In addition to or instead of the processing using the machine learning algorithm, the image and data processing method may perform any known estimation processing such as natural language processing, matching processing using a database of similar images and similar data, knowledge base processing, etc. In addition, the teacher data for training the trained model constructed using the machine learning algorithm may be images labeled with the imaging site and imaging area. In this case, with regard to the teacher data, the image from which the imaging site and imaging area should be estimated is used as input data, and the labels of the imaging site and imaging area are used as output data.

[0335] In particular, as an example of a CNN configuration for estimating the imaging location of a two-dimensional input image Im2610, there is a configuration 2600 shown in FIG. 26. The CNN configuration 2600 includes a group of multiple convolution processing blocks 2620, each of which includes a convolution layer 2621, a batch normalization layer 2622, and an activation layer 2623 using a normalized linear function. The CNN configuration 2600 also includes a final convolution layer 2630, a fully connected layer 2640, and an output layer 2650. The fully connected layer 2640 fully connects the output value groups of the convolution processing blocks 2620. The output layer 2650 also uses a Softmax function to output the probability of each expected imaging location label for the input image Im2610 as an estimation result 2660 (Result). In such a configuration 2600, for example, if the input image Im2610 is an image of the "macular region", the highest probability is output for the imaging region label corresponding to the "macular region".

[0336] For example, the imaging site can be estimated with a certain degree of accuracy by setting the number of convolution processing blocks 2620 to 16, the parameters of the convolution layers 2621 and 2630 to a filter kernel size of 3 pixels wide and 3 pixels high, and the number of filters to 64. However, in practice, as described in the above description of the trained model, a better parameter group can be set using training data according to the usage form of the trained model. When one-dimensional images, three-dimensional images, or four-dimensional images need to be processed, the filter kernel size may be expanded to one dimension, three dimensions, or four dimensions. The estimation method may be implemented using only one image and data processing method, or may be implemented using a combination of two or more image and data processing methods.

[0337] The region label image evaluation engine is a module that evaluates whether an input region label image has been plausibly processed through image segmentation. Specifically, the region label image evaluation engine outputs a true value as an image evaluation index if the input region label image is plausible, and a false value if it is not. Examples of methods for performing this evaluation include processing using a trained model according to various machine learning algorithms such as deep learning, or knowledge-based processing. One method of knowledge-based processing, for example, uses anatomical knowledge, and evaluates region label images using known anatomical knowledge such as the regularity of the shape of the retina.

[0338] Specifically, as an example, a case will be described in which a region label image corresponding to an OCT tomographic image of the posterior segment is evaluated by knowledge-based processing. In the posterior segment, tissue groups have anatomically fixed positions. Therefore, there is a method of checking the coordinates of the pixel value group in the region label image, i.e., the region label value group, and evaluating whether the position is correctly output. In this evaluation method, for example, if there is a region label value corresponding to the crystalline lens at a coordinate close to the anterior segment in a certain range, and there is a region label value corresponding to the retinal layer group at a coordinate far from the anterior segment, it is evaluated that the image segmentation processing has been performed plausibly. On the other hand, if these region label values ​​are not in such expected positions, it is evaluated that the image segmentation processing has not been performed appropriately.

[0339] The knowledge-based evaluation method will be described in more detail using a region-labeled image Im2710 of layers constituting the retina, which corresponds to an OCT tomographic image of the posterior segment of the eye, as shown in Fig. 27. In the posterior segment of the eye, tissue groups have anatomically fixed positions, so by checking the pixel values ​​in the region-labeled image, i.e., the coordinates of the region-labeled values, it is possible to determine whether the region-labeled image is a plausible image or not.

[0340] The region label image Im2710 includes regions Seg2711, Seg2712, Seg2713, and Seg2714, which are made up of consecutive pixel groups having the same region label value. Regions Seg2711 and Seg2714 have the same region label value, but because the layers that make up the retina anatomically have a layered structure, region Seg2714 is erroneously evaluated as having been subjected to image segmentation processing based on its shape and its positional relationship with other regions. In this case, the region label image evaluation engine outputs a false value as the image evaluation index.

[0341] The knowledge-based evaluation process also includes a method for evaluating whether the region label image contains pixels having region label values ​​corresponding to regions that should definitely exist in the photographed subject, or a method for evaluating whether the region label image contains a certain number of pixels having region label values ​​corresponding to regions that should definitely exist in the photographed subject.

[0342] In addition, if the image segmentation engine implements multiple image segmentation processing techniques and generates multiple region label images, the region label image evaluation engine can also select and output one of the multiple label images that is the most likely region label image.

[0343] In addition, when each of the multiple area label image groups is a plausible area label image, there are cases where it is not possible to select one area label image to be output. In such a case where it is not possible to select one area label image, the area label evaluation engine can select one area label image, for example, with a predetermined priority. In addition, the area label evaluation engine can also weight the multiple area label images and merge them into one area label image. Furthermore, for example, the area label evaluation engine may display the multiple area label image groups on a user interface provided on an arbitrary display unit or the like, and select one according to an instruction from an examiner (user). In addition, the area label image evaluation engine may output all of the multiple plausible area images.

[0344] The region label image correction engine is a module that corrects regions that have been erroneously subjected to image segmentation processing in the input region label image. An example of a method for performing such correction is knowledge-based processing. One method of knowledge-based processing, for example, is to use anatomical knowledge.

[0345] The knowledge-based correction method for correcting the region label image will be described in more detail by using again the region label image Im2710 of the layers constituting the retina layer corresponding to the OCT tomographic image of the posterior eye shown in FIG. 27. As described above, in the region label image Im2710, the layers constituting the retina layer anatomically have a layer structure, so that it is understood that the region Seg2714 is a region that has been erroneously subjected to image segmentation processing based on its shape and its positional relationship with other regions. The region label image correction engine detects the region that has been erroneously subjected to segmentation processing, and overwrites the detected region with another region label value. For example, in the case of FIG. 27, the region Seg2714 is overwritten with a region label value that indicates that it is not one of the retina layers.

[0346] In addition, the region label image correction engine may detect or identify the erroneously segmented region using the evaluation result by the region label evaluation engine. In addition, the region label image correction engine may overwrite the label value of the detected erroneously segmented region with label information inferred from the label information of the periphery of the region. In the example of FIG. 27, when label information is added to the region surrounding the region Seg2714, the label information of the region Seg2714 can be overwritten with the label information of the region. In addition, the surrounding label information is not limited to the label information of the region completely surrounding the region to be corrected, but may be the most numerous label information among the label information of the regions adjacent to the region to be corrected.

[0347] (Configuration of image processing device according to embodiment 8) Hereinafter, a medical image processing device according to an eighth embodiment will be described with reference to Figs. 28 to 32. For the sake of simplicity, the medical image processing device will be simply referred to as an image processing device below. Fig. 28 shows an example of a schematic configuration of the image processing device according to this embodiment.

[0348] The image processing device 2800 is connected to the photographing device 2810 and the display unit 2820 via a circuit or a network. The photographing device 2810 and the display unit 2820 may be directly connected. Although these devices are separate devices in this embodiment, some or all of these devices may be integrated. These devices may be connected to any other device via a circuit or a network, or may be integrated with any other device.

[0349] The image processing device 2800 includes an acquisition unit 2801, a shooting condition acquisition unit 2802, a processing feasibility determination unit 2803, a segmentation processing unit 2804, an evaluation unit 2805, an analysis unit 2806, and an output unit 2807 (display control unit). The image processing device 2800 may be composed of a plurality of devices in which some of these components are provided.

[0350] The acquisition unit 2801 can acquire various data and images from the imaging device 2810 or other devices, and can acquire input from the examiner via an input device (not shown). As the input device, a mouse, a keyboard, a touch panel, or any other arbitrary input device may be adopted. Also, the display unit 2820 may be configured as a touch panel display.

[0351] The imaging condition acquisition unit 2802 acquires imaging conditions of the medical image (input image) acquired by the acquisition unit 2801. Specifically, the imaging condition group stored in the data structure constituting the medical image is acquired according to the data format of the medical image. If the imaging condition is not stored in the medical image, the imaging information group can be acquired from the imaging device 2810 or the image management system via the acquisition unit 2801.

[0352] A processability determination unit (determination unit) 2803 determines whether or not the medical image can be handled by the segmentation processing unit 2804 using the imaging condition group acquired by the imaging condition acquisition unit 2802. The segmentation processing unit 2804 performs image segmentation processing on the medical image that can be handled, using an image segmentation engine (segmentation engine) including a trained model, and generates a region label image (region information).

[0353] The evaluation unit 2805 uses a region label image evaluation engine (evaluation engine) to evaluate the region label image generated by the segmentation processing unit 2804, and judges whether to output the region label image based on the evaluation result. The region label image evaluation engine outputs a true value as an image evaluation index if the input region label image is likely, and a false value if not. The evaluation unit 2805 judges to output the region label image if the image evaluation index for evaluating the region label image is a true value.

[0354] The analysis unit 2806 performs image analysis processing of the input image using the region label image and the input image determined by the evaluation unit 2805 to be output. The analysis unit 2806 can calculate, for example, shape changes and layer thicknesses of tissues contained in retinal layers by the image analysis processing. Any known image analysis processing may be used as the image analysis processing. The output unit 2807 displays the region label image and the analysis results by the analysis unit 2806 on the display unit 2820. The output unit 2807 may also store the region label image and the analysis results in a storage device or an external device connected to the image processing device 2800.

[0355] Next, the segmentation processing unit 2804 will be described in detail. The segmentation processing unit 2804 uses an image segmentation engine to generate a region label image corresponding to an input image (input). In the image segmentation processing method provided in the image segmentation engine according to this embodiment, processing using a trained model is performed.

[0356] In this embodiment, training of the machine learning model uses teacher data consisting of pairs of input data, which is an image acquired under specific imaging conditions assumed to be processed, and output data, which is a region label image corresponding to the input data. The specific imaging conditions specifically include a predetermined imaging site, imaging method, imaging angle of view, image size, etc.

[0357] In this embodiment, the input data of the teacher data is an image acquired by the same model and with the same settings as the camera device 2810. Note that the input data of the teacher data may be an image acquired by a camera device having the same image quality tendency as the camera device 2810.

[0358] Moreover, the output data of the teacher data is a region label image corresponding to the input data. For example, referring to FIG. 24, the input data is a tomographic image Im2410 of a retinal layer captured by OCT. Moreover, the output data is a region label image Im2420 in which each region is divided by adding a region label value representing the type of retinal layer to the corresponding coordinate according to the type of retinal layer depicted in the tomographic image Im2410. The region label image can be prepared by a specialist referring to the tomographic image, by creating it by any image segmentation processing, or by a specialist correcting and creating the region label image created by the image segmentation processing.

[0359] The input data group of the pair group of teacher data comprehensively includes input images having various conditions that are assumed to be processed. Specifically, the various conditions are image conditions that arise due to different combinations of variations such as the pathology of the subject, the imaging environment of the imaging device, and the technical level of the photographer. By comprehensively including the image conditions included in the input data group, the machine learning model is trained so that high-precision image segmentation processing can be performed even for images under poor conditions where the accuracy is low in conventional image segmentation processing. Therefore, the segmentation processing unit 2804 can stably generate high-precision region label images for images under various conditions by using an image segmentation engine including a learned model that has been trained in this way.

[0360] Among the pairs constituting the training data, pairs that do not contribute to the image segmentation process can be removed from the training data. For example, if the region label value of a region label image that is output data constituting a pair of training data is incorrect, the region label value of the region label image obtained using a trained model trained using the training data is also likely to be incorrect. In other words, the accuracy of the image segmentation process is reduced. Therefore, by removing pairs having a region label image with an incorrect region label value as output data from the training data, it is possible to improve the accuracy of the trained model included in the image segmentation engine.

[0361] By using an image segmentation engine including such a trained model, the segmentation processing unit 2804 can output a region label image capable of identifying organs or lesions depicted in a medical image obtained by photography when the medical image is input.

[0362] Next, a series of image processing according to this embodiment will be described with reference to the flowchart of Fig. 29. Fig. 29 is a flowchart of a series of image processing according to this embodiment. First, when a series of image processing according to this embodiment is started, the process proceeds to step S2910.

[0363] In step S2910, the acquisition unit 2801 acquires an image captured by the photographing device 2810 as an input image from the photographing device 2810 connected via a circuit or a network. The acquisition unit 2801 may acquire the input image in response to a request from the photographing device 2810. Such a request may be issued, for example, when the photographing device 2810 generates an image, before or after the image generated by the photographing device 2810 is stored in the recording device of the photographing device 2810, when the stored image is displayed on the display unit 2820, when a region label image is used for image analysis processing, etc.

[0364] The acquisition unit 2801 may acquire data for generating an image from the photographing device 2810, and may acquire an image generated by the image processing device 2800 based on the data as an input image. In this case, any existing image generation method may be adopted as an image generation method for the image processing device 2800 to generate various images.

[0365] In step S2920, the photographing condition acquisition unit 2802 acquires a group of photographing conditions for the input image. Specifically, the photographing condition group stored in the data structure constituting the input image is acquired according to the data format of the input image. As described above, if the photographing conditions are not stored in the input image, the photographing condition acquisition unit 2802 can acquire a group of photographing information from the photographing device 2810 or an image management system (not shown).

[0366] In step S2930, the processability determination unit 2803 uses the acquired group of shooting conditions to determine whether or not the input image can be subjected to image segmentation processing by the image segmentation engine used by the segmentation processing unit 2804. Specifically, the processability determination unit 2803 determines whether or not the shooting part, shooting method, shooting angle of view, and image size of the input image match conditions that can be handled using the trained model of the image segmentation engine.

[0367] If the processing feasibility determination unit 2803 determines all the shooting conditions and determines that they are manageable, the process proceeds to step S2940. On the other hand, if the processing feasibility determination unit 2803 determines that the image segmentation engine cannot handle the input image based on these shooting conditions, the process proceeds to step S2970.

[0368] Depending on the settings and implementation of the image processing device 2800, step S2940 may be performed even if it is determined that the input image cannot be processed based on some of the imaging part, imaging method, imaging angle of view, and image size. For example, such processing may be performed when the image segmentation engine is assumed to be comprehensively compatible with any imaging part of the subject and is implemented to be able to handle even if the input data includes an unknown imaging part. In addition, the processability determination unit 2803 may determine whether or not at least one of the imaging part, imaging method, imaging angle of view, and image size of the input image matches a condition that can be handled by the image segmentation engine according to a desired configuration.

[0369] In step S2940, the segmentation processing unit 2804 performs image segmentation processing on the input image using the image segmentation engine, and generates a region label image from the input image. Specifically, the segmentation processing unit 2804 inputs the input image to the image segmentation engine. The image segmentation engine generates a region label image as region information capable of identifying organs and lesions depicted in the input image, based on a trained model that has been machine-learned using teacher data.

[0370] Depending on the settings and implementation form of the image processing device 2800, the segmentation processing unit 2804 may input parameters to the image segmentation engine together with the input image according to the group of shooting conditions to adjust the extent of the range of the image segmentation processing, etc. Also, the segmentation processing unit 2804 may input parameters according to the examiner's input to the image segmentation engine together with the input image to adjust the extent of the range of the image segmentation processing, etc.

[0371] In step S2950, ​​the evaluation unit 2805 uses a region label image evaluation engine to evaluate whether or not the region label image generated by the segmentation processing unit 2804 is a plausible image. In this embodiment, the evaluation unit 2805 uses a region label evaluation engine that uses a knowledge-based evaluation method to evaluate whether or not the region label image is a plausible image.

[0372] Specifically, the region label evaluation engine checks the pixel value group in the region label image, i.e., the coordinates of the region label value group, and evaluates whether the position is output at the anatomically correct position. In this case, for example, if there is a region label value corresponding to the crystalline lens at a coordinate close to the anterior segment in a certain range, and there is a region label value corresponding to the retinal layers at a coordinate far from the anterior segment, it is evaluated that the image segmentation process has been performed plausibly. On the other hand, if these region label values ​​are not at such expected positions, it is evaluated that the image segmentation process has not been performed properly. If the region label evaluation engine evaluates that the image segmentation process has been performed plausibly for the region label, it outputs a true value as the image evaluation index, and if it evaluates that the image segmentation process has not been performed plausibly, it outputs a false value.

[0373] The evaluation unit 2805 judges whether or not to output a region label image based on the image evaluation index output from the region label evaluation engine. Specifically, the evaluation unit 2805 judges to output a region label image when the image evaluation index is a true value. On the other hand, when the image evaluation index is a false value, the evaluation unit 2805 judges not to output the region label image generated by the segmentation processing unit 2804. Note that, when the evaluation unit 2805 judges not to output the region label image generated by the segmentation processing unit 2804, it can generate an image without region labels.

[0374] In step S2960, when the evaluation unit 2805 determines that a region label image is to be output, the analysis unit 2806 performs image analysis processing of the input image using the region label image and the input image. The analysis unit 2806 calculates, for example, changes in layer thickness and tissue shape depicted in the input image by the image analysis processing. Any known processing may be adopted as the method of image analysis processing. Also, when the evaluation unit 2805 determines that a region label image is not to be output or when an image without a region label is generated, the processing proceeds without performing image analysis.

[0375] In step S2970, when the evaluation unit 2805 determines that the region label image is to be output, the output unit 2807 outputs the region label image and the image analysis result and displays them on the display unit 2820. Note that the output unit 2807 may display or store the region label image and the image analysis result on the photographing device 2810 or other devices instead of displaying them on the display unit 2820. Depending on the settings and implementation form of the image processing device 2800, the output unit 2807 may process them so that they can be used by the photographing device 2810 or other devices, or convert the data format so that they can be transmitted to an image management system or the like. The output unit 2807 is not limited to a configuration that outputs both the region label image and the image analysis result, and may output only one of them.

[0376] On the other hand, if it is determined in step S2930 that the image segmentation process is impossible, the output unit 2807 outputs an image without region labels, which is a type of region labeled image, and displays it on the display unit 2820. Note that instead of outputting an image without region labels, a signal indicating that the image segmentation process is impossible may be transmitted to the photographing device 2810.

[0377] Also, when it is determined in step S2950 that the image segmentation process has not been performed appropriately, the output unit 2807 outputs an image without region labels and displays it on the display unit 2820. In this case, instead of outputting an image without region labels, the output unit 2807 may transmit a signal indicating that the image segmentation process has not been performed appropriately to the photographing device 2810. When the output process in step S2970 ends, the series of image processes ends.

[0378] As described above, the image processing device 2800 according to this embodiment includes an acquisition unit 2801 and a segmentation processing unit 2804. The acquisition unit 2801 acquires an input image that is a tomographic image of a predetermined part of a subject. The segmentation processing unit 2804 generates a region label image that is region information capable of identifying an anatomical region from the input image using an image segmentation engine including a trained model. The image segmentation engine includes a trained model in which tomographic images under various conditions and region label images are used as training data. The image segmentation engine receives a tomographic image as an input and outputs a region label image.

[0379] The image processing device 2800 further includes an evaluation unit 2805. The evaluation unit 2805 evaluates the region labeled image using a knowledge-based evaluation engine using anatomical features, and determines whether or not to output the region labeled image according to the evaluation result.

[0380] With this configuration, the image processing device 2800 according to this embodiment can generate a region label image as region information used to identify ROIs and VOIs that can be used for image diagnosis and image analysis, using a segmentation engine including a trained model. Therefore, even for an input image that is poorly conditioned for conventional segmentation processing, a highly accurate region label image can be output, and ROIs and VOIs that can be used for image diagnosis and image analysis can be provided.

[0381] Furthermore, the image processing device 2800 can prevent an inappropriate region label image from being used for image diagnosis or image analysis by evaluating, by the evaluation unit 2805, whether or not the region label image is a plausible image.

[0382] The image processing device 2800 further includes an imaging condition acquisition unit 2802 and a processing feasibility determination unit 2803. The imaging condition acquisition unit 2802 acquires imaging conditions including at least one of the imaging part, imaging method, imaging angle of view, and image size of the input image. The processing feasibility determination unit 2803 determines whether or not a region label image can be generated from the input image using an image segmentation engine. The processing feasibility determination unit 2803 makes the determination based on the imaging conditions of the input image.

[0383] With this configuration, the image processing device 2800 according to this embodiment can exclude input images that cannot be processed by the segmentation processing unit 2804 from the image segmentation processing, thereby reducing the processing load of the image processing device 2800 and the occurrence of errors.

[0384] Furthermore, the image processing device 2800 further includes an analysis unit 2806 that performs image analysis of the input image using a region label image that is region information. With this configuration, the image processing device 2800 can perform image analysis using the highly accurate region label image generated by the segmentation processing unit 2804, and can obtain highly accurate analysis results.

[0385] In this embodiment, the processability determination unit 2803 determines whether the input image can be subjected to image segmentation processing by the image segmentation engine. Thereafter, when the processability determination unit 2803 determines that the input image can be processed by the segmentation processing unit 2804, the segmentation processing unit 2804 performs image segmentation processing. On the other hand, when the photographing device 2810 photographs only under photographing conditions that allow image segmentation processing, the image acquired from the photographing device 2810 may be subjected to image segmentation processing unconditionally. In this case, as shown in FIG. 30, the processes of steps S2920 and S2930 can be omitted, and step S2940 can be performed after step S2910.

[0386] In this embodiment, the output unit 2807 (display control unit) is configured to display the generated region label image and analysis results on the display unit 2820, but the operation of the output unit 2807 is not limited to this. For example, the output unit 2807 can also output the region label image and analysis results to the imaging device 2810 or other devices connected to the image processing device 2800. For this reason, the region label image and analysis results can be displayed on the user interface of these devices, stored in any recording device, used for any image analysis, or transmitted to an image management system.

[0387] In addition, in this embodiment, the output unit 2807 is configured to display the region label image and the image analysis result on the display unit 2820. However, the output unit 2807 may display the region label image and the image analysis result on the display unit 2820 in response to an instruction from the examiner. For example, the output unit 2807 may display the region label image and the image analysis result on the display unit 2820 in response to the examiner pressing an arbitrary button on the user interface of the display unit 2820. In this case, the output unit 2807 may display the region label image in switching from the input image. In addition, the output unit 2807 may display the region label image UI3120 side by side with the input image UI3110 as in FIG. 31, or may display the input image and the semi-transparent region label image superimposed as in any of UI3210 to UI3240 in FIG. 32. Note that the method of semi-transparency may be any known method, and for example, the region label image can be made semi-transparent by setting the transparency of the region label image to a desired value.

[0388] Furthermore, the output unit 2807 may cause the display unit 2820 to display that the region label image was generated using a trained model or that the region label image is a result of image analysis performed based on the region label image generated using the trained model. Furthermore, the output unit 2807 may cause the display unit 2820 to display a display indicating what kind of teacher data the trained model was trained with. The display may include an explanation of the types of input data and output data of the teacher data, or any display related to the teacher data such as the imaging site included in the input data and output data.

[0389] In this embodiment, when the processability determination unit 2803 determines that the input image can be handled by the image segmentation engine, the process proceeds to step S2940, and the image segmentation processing unit 2804 starts image segmentation processing. In response to this, the output unit 2807 may display the determination result by the processability determination unit 2803 on the display unit 2820, and the segmentation processing unit 2804 may start image segmentation processing in response to an instruction from the examiner. At this time, the output unit 2807 may also display the input image and the imaging conditions such as the imaging part acquired for the input image, together with the determination result, on the display unit 2820. In this case, the image segmentation processing is performed after the examiner determines whether the determination result is correct or not, so that the image segmentation processing based on erroneous determination can be reduced.

[0390] In relation to this, the output unit 2807 may display the input image and the imaging conditions such as the imaging part acquired for the input image on the display unit 2820 without making a determination by the processing feasibility determination unit 2803. In this case, the segmentation processing unit 2804 can start the image segmentation process in response to an instruction from the examiner.

[0391] Furthermore, in this embodiment, the evaluation unit 2805 uses a region label image evaluation engine that employs a knowledge-based evaluation method to evaluate the region label image generated by the segmentation processing unit 2804. In contrast, the evaluation unit 2805 may evaluate whether or not the region label image is a plausible image using a region label image evaluation engine including a learned model that has been trained using the region label image and an image evaluation index according to a predetermined evaluation method as teacher data.

[0392] In this case, the training data of the machine learning model included in the region label image evaluation engine uses region label images and fake images that seem to be region label images as input data, and the image evaluation index for each image as output data. The image evaluation index is a true value when the input data is an appropriate region label image, and a false value when the input data is a fake image. Note that, as a method of generating fake images, a method of using an arbitrary generator of region label images with inappropriate conditions set, a method of intentionally overwriting an appropriate region label image with an inappropriate region label to generate it, etc. may be adopted.

[0393] Even if the evaluation unit 2805 uses a region label image evaluation engine including a trained model that has undergone such training to evaluate whether a region label image is a plausible image, it is possible to prevent an inappropriate region label image from being used for image diagnosis or image analysis.

[0394] When the processing feasibility determination unit 2803 determines that the image segmentation process is impossible, the image without region labels may be generated by the output unit 2807 or may be generated by the processing feasibility determination unit 2803. When the evaluation unit 2805 determines that the image segmentation has not been performed appropriately, the output unit 2807 may display on the display unit 2820 in step S2970 that the image segmentation has not been performed appropriately.

[0395] Example 9 Next, an image processing device according to a ninth embodiment will be described with reference to Figs. 28 and 33. In the eighth embodiment, the segmentation processing unit 2804 includes one image segmentation engine. In contrast, in the present embodiment, the segmentation processing unit generates a plurality of region label images for an input image using a plurality of image segmentation engines including trained models each of which has been machine-learned using different teacher data.

[0396] Unless otherwise specified, the configuration and processing of the image processing device according to this embodiment are the same as the image processing device 2800 according to the eighth embodiment. Therefore, the image processing device according to this embodiment will be described below, focusing on the differences from the image processing device according to the eighth embodiment. Since the configuration of the image processing device according to this embodiment is the same as the configuration of the image processing device according to the eighth embodiment, the same reference numerals are used for the configuration shown in FIG. 28, and the description will be omitted.

[0397] The segmentation processing unit 2804 according to this embodiment performs image segmentation processing on an input image using two or more image segmentation engines each including a trained model that has been machine-learned using different training data.

[0398] Here, a method for creating a teacher data group according to this embodiment will be described. Specifically, first, a pair group of images of various imaging sites as input data and area label images as output data is prepared. Next, the pair groups are grouped for each imaging site to create a teacher data group. For example, a teacher data group is created such that a first teacher data group is composed of a pair group acquired by imaging a first imaging site, and a second teacher data group is composed of a pair group acquired by imaging a second imaging site.

[0399] Then, machine learning is performed on machine learning models included in separate image segmentation engines using each training data. For example, a first image segmentation engine including a trained model trained with the first training data is prepared. In addition, a second image segmentation engine including a trained model trained with the second training data is prepared, and thus a group of image segmentation engines is prepared.

[0400] Such image segmentation engines use different teacher data for training the learned models included in each of them. Therefore, the degree to which such image segmentation engines can perform image segmentation processing on input images varies depending on the shooting conditions of the images input to the image segmentation engine. Specifically, the first image segmentation engine has high accuracy of image segmentation processing for input images acquired by shooting the first shooting site, and low accuracy of image segmentation processing for images acquired by shooting the second shooting site. Similarly, the second image segmentation engine has high accuracy of image segmentation processing for input images acquired by shooting the second shooting site, and low accuracy of image segmentation processing for images acquired by shooting the first shooting site.

[0401] Since each of the teacher data is composed of a pair group grouped by the imaging site, the image quality tendency of the image group constituting the pair group is similar. Therefore, the image segmentation engine can perform image segmentation processing with higher accuracy than the image segmentation engine according to the eighth embodiment, if the imaging site corresponds. Note that the imaging condition for grouping the pairs of the teacher data is not limited to the imaging site, and may be the imaging angle of view, the image resolution, or a combination of two or more of these.

[0402] A series of image processing according to this embodiment will be described with reference to Fig. 33. Fig. 33 is a flowchart of a series of image processing according to this embodiment. Note that the processes of steps S3310 and S3320 are similar to steps S2910 and S2920 according to the eighth embodiment, and therefore the description will be omitted. Note that, when performing image segmentation processing unconditionally on an input image, after the process of step S3320, the process of step S3330 may be omitted and the process may proceed to step S3340.

[0403] When the shooting conditions of the input image are acquired in step S3320, the process proceeds to step S3330. In step S3330, the processability determination unit 2803 uses the shooting conditions acquired in step S3320 to determine whether any of the image segmentation engines described above can handle the input image.

[0404] If the processing possibility determination unit 2803 determines that none of the image segmentation engines can handle the input image, the process proceeds to step S3380. On the other hand, if the processing possibility determination unit 2803 determines that any of the image segmentation engines can handle the input image, the process proceeds to step S3340. Note that, depending on the settings and implementation form of the image processing device 2800, step S3340 may be performed, as in the eighth embodiment, even if the image segmentation engine determines that some shooting conditions cannot be handled.

[0405] In step S3340, the segmentation processing unit 2804 selects an image segmentation engine to perform processing based on the imaging conditions of the input image acquired in step S3320 and the information on the teacher data of the image segmentation engine group. Specifically, for example, for an imaging site in the imaging condition group acquired in step S3320, an image segmentation engine that has information on teacher data related to the imaging site or surrounding imaging sites and has high accuracy of image segmentation processing is selected. In the above example, when the imaging site is the first imaging site, the segmentation processing unit 2804 selects the first image segmentation engine.

[0406] In step S3350, the segmentation processing unit 2804 generates a region label image by performing image segmentation processing on the input image using the image segmentation engine selected in step S3340. Steps S3360 and S3370 are similar to steps S2950 and S2960 in the eighth embodiment, and therefore will not be described.

[0407] After the image analysis is performed in step S3370, the process proceeds to step S3380. In step S3380, when the evaluation unit 2805 determines to output a region label image, the output unit 2807 outputs the region label image and the analysis result, and causes the display unit 2820 to display them. When displaying the region label image on the display unit 2820, the output unit 2807 may display that the region label image was generated using the image segmentation engine selected by the segmentation processing unit 2804. The output unit 2807 may output only either the region label image or the analysis result.

[0408] On the other hand, if it is determined in step S3330 that the image segmentation process is impossible, the output unit 2807 outputs an image without region labels, which is a type of region labeled image, and displays it on the display unit 2820. Note that instead of generating an image without region labels, a signal indicating that the image segmentation process is impossible may be transmitted to the photographing device 2810.

[0409] Also, when it is determined in step S3360 that the image segmentation process has not been performed appropriately, the output unit 2807 outputs an image without region labels, which is a type of region labeled image, and displays it on the display unit 2820. In this case, instead of generating an image without region labels, the output unit 2807 may transmit a signal indicating that the image segmentation process has not been performed appropriately to the imaging device 2810. When the output process in step S3380 ends, the series of image processes ends.

[0410] As described above, the segmentation processing unit 2804 according to this embodiment generates a region label image using at least one of a plurality of image segmentation engines including trained models trained using different training data. In this embodiment, each of the plurality of image segmentation engines includes trained models trained using different training data for at least one of the imaging site, imaging angle of view, and image resolution. The segmentation processing unit 2804 generates a region label image using an image segmentation engine corresponding to at least one of the imaging site, imaging angle of view, and image resolution of the input image.

[0411] With this configuration, the image processing device 2800 according to this embodiment can perform more accurate image segmentation processing in accordance with the shooting conditions.

[0412] In this embodiment, the segmentation processing unit 2804 selects an image segmentation engine to be used for image segmentation processing based on the photographing conditions of the input image, but the selection process of the image segmentation engine is not limited to this. For example, the output unit 2807 may display the photographing conditions of the acquired input image and the image segmentation engine group on the user interface of the display unit 2820. Furthermore, the segmentation processing unit 2804 may select an image segmentation engine to be used for image segmentation processing in response to an instruction from the examiner.

[0413] The output unit 2807 may display information on the teacher data used in training each image segmentation engine together with the image segmentation engine group on the display unit 2820. The display format of the information on the teacher data used in training the image segmentation engine may be arbitrary, and for example, the image segmentation engine group may be displayed using a name related to the teacher data used in training.

[0414] Furthermore, the output unit 2807 may display the image segmentation engine selected by the segmentation processing unit 2804 on the user interface of the display unit 2820 and accept an instruction from the examiner. In this case, the segmentation processing unit 2804 may determine whether or not to finally select the image segmentation engine as the image segmentation engine to be used for image segmentation processing, in response to the instruction from the examiner.

[0415] Note that, similarly to the eighth embodiment, the output unit 2807 may output the generated region label image and the evaluation result to the photographing device 2810 or another device connected to the image processing device 2800. Depending on the settings and implementation form of the image processing device 2800, the output unit 2807 may process these so that they can be used by the photographing device 2810 or other devices, or convert the data format so that they can be transmitted to an image management system or the like.

[0416] Example 10 Next, an image processing device according to a tenth embodiment will be described with reference to Figs. 28 and 33. In the eighth and ninth embodiments, the imaging condition acquisition unit 2802 acquires an imaging condition group from the data structure of an input image, etc. In contrast, in the present embodiment, the imaging condition acquisition unit estimates an imaging part or imaging area of ​​an input image based on the input image by using an imaging location estimation engine.

[0417] Unless otherwise specified, the configuration and processing of the image processing device according to this embodiment are the same as the image processing device 2800 according to the ninth embodiment. Therefore, the image processing device according to this embodiment will be described below, focusing on the differences from the image processing device according to the ninth embodiment. Since the configuration of the image processing device according to this embodiment is the same as the configurations of the image processing devices according to the eighth and ninth embodiments, the configuration shown in FIG. 28 is indicated by the same reference numerals, and description thereof will be omitted.

[0418] The photographing condition acquisition unit 2802 according to this embodiment uses a photographing location estimation engine (estimation engine) to estimate and acquire the photographing part or photographing area depicted in the input image acquired by the acquisition unit 2801. In the photographing location estimation method provided in the photographing location estimation engine according to this embodiment, estimation processing is performed using a machine learning algorithm.

[0419] In this embodiment, training of a learned model related to a photographing location estimation method using a machine learning algorithm uses teacher data consisting of a pair group of input data, which is an image, and output data, which is a photographing location label corresponding to the input data. Here, the input data refers to an image having a specific photographing condition assumed as a processing target (input image). The input data is preferably an image acquired from a photographing device having the same image quality tendency as the photographing device 2810, and more preferably the same model with the same settings as the photographing device 2810. The type of the photographing location label, which is the output data, may be a photographing location at least partially included in the input data. The type of the photographing location label, which is the output data, may be, for example, "macular area", "optic disc", "macular area and optic disc", and "others".

[0420] The imaging location estimation engine according to the present embodiment includes a trained model that has been trained using such teacher data, and can output the location of the imaging part or imaging area depicted in the input image. In addition, the imaging location estimation engine can output the probability that the imaging part or imaging area is the imaging part or imaging area for each imaging part label or imaging area label of a required level of detail.

[0421] By using the photographing location estimation engine, the photographing condition acquisition unit 2802 can estimate the photographing part and photographing area of ​​the input image based on the input image and acquire them as the photographing conditions for the input image. Note that, when the photographing location estimation engine outputs the probability of the photographing part or photographing area for each photographing part label or photographing area label, the photographing condition acquisition unit 2802 acquires the photographing part or photographing area with the highest probability as the photographing condition of the input image.

[0422] Next, a series of image processing steps according to this embodiment will be described with reference to the flowchart in Fig. 33, similarly to the ninth embodiment. Note that the processing steps S3310 and steps S3330 to S3380 according to this embodiment are similar to those in the ninth embodiment, and therefore will not be described. Note that when performing image segmentation processing unconditionally on an input image, after the processing step S3320, the processing step S3330 may be omitted and the processing may proceed to step S3340.

[0423] When the input image is acquired in step S3310, the process proceeds to step S3320. In step S3320, the shooting condition acquisition unit 2802 acquires a group of shooting conditions for the input image acquired in step S3310.

[0424] Specifically, the imaging condition group stored in the data structure constituting the input image is acquired according to the data format of the input image. Also, if the imaging condition group does not include information on the imaging part or imaging area, the imaging condition acquisition unit 2802 inputs the input image to an imaging location estimation engine and estimates which imaging part and imaging area the input image was acquired by imaging. Specifically, the imaging condition acquisition unit 2802 inputs the input image to an imaging location estimation engine, evaluates the output probability for each of the imaging part label group or imaging area label group, and sets and acquires the imaging part or imaging area with the highest probability as the imaging condition of the input image.

[0425] In addition, when the input image does not store the imaging conditions other than the imaging part and the imaging area, the imaging condition acquisition unit 2802 can acquire the imaging information group from the imaging device 2810 or the image management system (not shown). The subsequent processing is similar to the series of image processing according to the ninth embodiment, and therefore the description will be omitted.

[0426] As described above, the photographing condition acquisition unit 2802 according to this embodiment functions as an estimation unit that estimates at least one of the photographing part and the photographing area from the input image by using the photographing part estimation engine including the trained model. The photographing condition acquisition unit 2802 estimates the photographing part and the photographing area of ​​the input image by inputting the input image to the photographing part estimation engine including the trained model in which images labeled with the photographing part and the photographing area are used as learning data.

[0427] In this way, the image processing device 2800 according to this embodiment can acquire the imaging conditions for the imaging part or imaging area of ​​the input image based on the input image.

[0428] In this embodiment, the photographing condition acquisition unit 2802 estimates the photographing part and the photographing area of ​​the input image using the photographing location estimation engine when the photographing condition group does not include information on the photographing part and the photographing area. However, the situation in which the photographing part and the photographing area are estimated using the photographing location estimation engine is not limited to this. The photographing condition acquisition unit 2802 may estimate the photographing part and the photographing area using the photographing location estimation engine even when the information on the photographing part and the photographing area included in the data structure of the input image is insufficient as information at a required level of detail.

[0429] Furthermore, regardless of whether the data structure of the input image includes information on the photographing part and the photographing area, the photographing condition acquisition unit 2802 may estimate the photographing part and the photographing area of ​​the input image using the photographing location estimation engine. In this case, the output unit 2807 may display the estimation result output from the photographing location estimation engine and the information on the photographing part and the photographing area included in the data structure of the input image on the display unit 2820, and the photographing condition acquisition unit 2802 may determine these photographing conditions according to the examiner's instructions.

[0430] Example 11 Next, an image processing device according to an eleventh embodiment will be described with reference to Figs. 28, 29, 34, and 35. In this embodiment, a segmentation processing unit enlarges or reduces an input image so that the input image has an image size that the image segmentation engine can handle. In addition, the segmentation processing unit reduces or enlarges an output image from the image segmentation engine so that the output image has the image size of the input image, thereby generating a region label image.

[0431] Unless otherwise specified, the configuration and processing of the image processing device according to this embodiment are the same as the image processing device 2800 according to the eighth embodiment. Therefore, the image processing device according to this embodiment will be described below, focusing on the differences from the image processing device according to the eighth embodiment. Since the configuration of the image processing device according to this embodiment is the same as the configuration of the image processing device according to the eighth embodiment, the same reference numerals are used for the configuration shown in FIG. 28, and the description will be omitted.

[0432] The segmentation processing unit 2804 according to this embodiment includes an image segmentation engine similar to the image segmentation engine according to the eighth embodiment. However, in this embodiment, training data used for learning a machine learning model included in the image segmentation engine is different from the training data in the eighth embodiment. Specifically, in this embodiment, a group of pairs of input data and output data, which are constituted by a group of images obtained by enlarging or reducing an image of the input data and an image of the output data to a certain image size, is used as training data.

[0433] Here, with reference to Fig. 34, the teacher data of the trained model included in the image segmentation engine according to the present embodiment will be described. As shown in Fig. 34, for example, consider a case where an input image Im3410 and a region label image Im3420 are smaller than a certain image size set for the teacher data. In this case, the input image Im3410 and the region label image Im3420 are each enlarged so as to have the certain image size set for the teacher data. Then, the enlarged image Im3411 and the enlarged region label image Im3421 are paired, and the pair is used as one of the teacher data.

[0434] As in the eighth embodiment, an image having a specific imaging condition assumed as a processing target (input image) is used as input data for the teacher data, and the specific imaging condition is a predetermined imaging part, imaging method, and imaging angle of view. In other words, unlike the eighth embodiment, the specific imaging condition in this embodiment does not include an image size.

[0435] The segmentation processing unit 2804 according to this embodiment performs image segmentation processing on the input image using an image segmentation engine trained with such teacher data to generate a region label image. At this time, the segmentation processing unit 2804 generates a deformed image by enlarging or reducing the input image so that it has a certain image size set for the teacher data, and inputs the deformed image to the image segmentation engine.

[0436] Furthermore, the segmentation processing unit 2804 reduces or enlarges the output image from the image segmentation engine so that it becomes the image size of the input image, and generates a region label image. Therefore, even if the input image has an image size that cannot be handled in the eighth embodiment, the segmentation processing unit 2804 according to this embodiment can perform image segmentation processing using the image segmentation engine to generate a region label image.

[0437] Next, a series of image processing according to this embodiment will be described with reference to Figs. 29 and 35. Fig. 35 is a flowchart of the segmentation processing according to this embodiment. Note that the processing of steps S2910, S2920, and S2950 to S2970 according to this embodiment is similar to those in the eighth embodiment, and therefore the description will be omitted. Note that, when performing image segmentation processing on an input image unconditionally with respect to shooting conditions other than image size, after the processing of step S2920, the processing of step S2930 may be omitted and the processing may proceed to step S2940.

[0438] In step S2920, similarly to the eighth embodiment, when the imaging condition acquisition unit 2802 acquires the imaging condition group of the input image, the process proceeds to step S2930. In step S2930, the processability determination unit 2803 uses the acquired imaging condition group to determine whether the input image can be handled by the image segmentation engine. Specifically, the processability determination unit 2803 determines whether the imaging conditions of the input image are imaging body part, imaging method, and imaging angle of view that the image segmentation engine can handle. Unlike the eighth embodiment, the processability determination unit 2803 does not determine the image size.

[0439] If the processing feasibility determination unit 2803 determines the imaging part, imaging method, and imaging angle of view, and determines that the input image is processable, the process proceeds to step S2940. On the other hand, if the processing feasibility determination unit 2803 determines that the image segmentation engine cannot process the input image based on these imaging conditions, the process proceeds to step S2970. Note that, depending on the settings and implementation form of the image processing device 2800, the image segmentation process in step S2940 may be performed even if it is determined that the input image is not processable based on some of the imaging part, imaging method, and imaging angle of view.

[0440] When the process proceeds to step S2940, the image segmentation process according to this embodiment shown in Fig. 35 starts. In the image segmentation process according to this embodiment, first, in step S3510, the segmentation processing unit 2804 enlarges or reduces the input image to a certain image size set for the training data, and generates a deformed image.

[0441] Next, in step S3520, the segmentation processing unit 2804 inputs the generated deformed image to an image segmentation engine, and obtains a first region label image that has been subjected to image segmentation processing.

[0442] Then, in step S3530, the segmentation processing unit 2804 reduces or enlarges the first region label image to the image size of the input image to generate a final region label image. When the segmentation processing unit 2804 generates the final region label image in step S3530, the image segmentation processing according to this embodiment ends, and the process proceeds to step S2950. The process after step S2950 is the same as the process after step S2950 in the eighth embodiment, and therefore the description thereof will be omitted.

[0443] As described above, the segmentation processing unit 2804 according to this embodiment adjusts the image size of the input image to an image size that the image segmentation engine can handle and inputs the image to the image segmentation engine. In addition, the segmentation processing unit 2804 generates a region label image by adjusting the output image from the image segmentation engine to the original image size of the input image. As a result, the image processing device 2800 according to this embodiment can perform image segmentation processing on an input image of an image size that was not supported in the eighth embodiment, and generate a region label image including information on ROI and VOI that can be used for image diagnosis and image analysis.

[0444] Example 12 Next, an image processing device according to a twelfth embodiment will be described with reference to Figs. 28, 29, 36 and 37. In this embodiment, a segmentation processing unit generates a region label image by image segmentation processing using an image segmentation engine with a certain resolution as a reference.

[0445] Unless otherwise specified, the configuration and processing of the image processing device according to this embodiment are the same as the image processing device 2800 according to the eighth embodiment. Therefore, the image processing device according to this embodiment will be described below, focusing on the differences from the image processing device according to the eighth embodiment. Since the configuration of the image processing device according to this embodiment is the same as the configuration of the image processing device according to the eighth embodiment, the same reference numerals are used for the configuration shown in FIG. 28, and the description will be omitted.

[0446] The segmentation processing unit 2804 according to this embodiment includes an image segmentation engine similar to that of the eighth embodiment. However, in this embodiment, training data used for learning a machine learning model included in the image segmentation engine is different from the training data of the eighth embodiment. Specifically, the image group constituting a pair group of input data and output data of the training data is enlarged or reduced to an image size such that the resolution of the image group becomes a certain resolution, and then padding is performed to obtain a sufficiently large certain image size. Here, the resolution of the image group refers to, for example, the spatial resolution of the imaging device or the resolution for the imaging area.

[0447] Here, referring to FIG. 36, the teacher data of the image segmentation engine according to the present embodiment will be described. As shown in FIG. 36, for example, consider the case where there is an image Im3610 and an area label image Im3620 having a resolution lower than a certain resolution set for the teacher data. In this case, the image Im3610 and the area label image Im3620 are each enlarged so as to have the certain resolution set for the teacher data. Furthermore, padding is performed on each of the enlarged image Im3610 and the area label image Im3620 so as to have a certain image size set for the teacher data. Then, the image Im3611 and the area label image Im3621 that have been enlarged and padded are paired, and the pair is used as one of the teacher data.

[0448] The certain image size set for the training data is the maximum possible image size when an image assumed to be processed (input image) is enlarged or reduced to a certain resolution. If the certain image size is not large enough, when an image input to the image segmentation engine is enlarged, the image size may become too large for the trained model to handle.

[0449] In addition, the area where padding is performed is filled with a certain pixel value, filled with a neighboring pixel value, or mirror padded according to the characteristics of the trained model so that image segmentation processing can be performed effectively. As in the eighth embodiment, an image having a specific shooting condition assumed to be a processing target is used as input data, and the specific shooting condition is a predetermined shooting part, shooting method, and shooting angle of view. In other words, unlike the eighth embodiment, the specific shooting condition in this embodiment does not include an image size.

[0450] The segmentation processing unit 2804 according to this embodiment performs image segmentation processing on the input image using an image segmentation engine including a trained model trained with such teacher data to generate a region label image. At this time, the segmentation processing unit 2804 generates a deformed image by enlarging or reducing the input image so that it has a certain resolution set for the teacher data. In addition, the segmentation processing unit 2804 pads the deformed image so that it has a certain image size set for the teacher data to generate a padded image, and inputs the padded image to the image segmentation engine.

[0451] The segmentation processing unit 2804 also trims the first region label image output from the image segmentation engine by the area where the padding was performed to generate a second region label image. After that, the segmentation processing unit 2804 reduces or enlarges the generated second region label image to the image size of the input image to generate a final region label image.

[0452] Therefore, the segmentation processing unit 2804 according to this embodiment can perform image segmentation processing by the image segmentation engine to generate a region label image even for an input image with an image size that cannot be handled in the eighth embodiment.

[0453] Next, a series of image processing according to this embodiment will be described with reference to Figs. 29 and 37. Fig. 37 is a flowchart of image segmentation processing according to this embodiment. Note that the processing of steps S2910, S2920, and S2950 to S2970 according to this embodiment is similar to those in the eighth embodiment, and therefore description thereof will be omitted. Note that, when image segmentation processing is performed unconditionally on the input image with respect to shooting conditions other than the image size, after the processing of step S2920, the processing of step S2930 may be omitted and the processing may proceed to step S2940.

[0454] In step S2920, as in the eighth embodiment, when the imaging condition acquisition unit 2802 acquires the imaging condition group of the input image, the process proceeds to step S2930. In step S2930, the processability determination unit 2803 uses the acquired imaging condition group to determine whether the input image can be handled by the image segmentation engine. Specifically, the processability determination unit 2803 determines whether the imaging conditions of the input image are imaging body part, imaging method, and imaging angle of view that the image segmentation engine can handle. Unlike the eighth embodiment, the processability determination unit 2803 does not determine the image size.

[0455] If the processing feasibility determination unit 2803 determines the imaging part, imaging method, and imaging angle of view, and determines that the input image is processable, the process proceeds to step S2940. On the other hand, if the processing feasibility determination unit 2803 determines that the image segmentation engine cannot process the input image based on these imaging conditions, the process proceeds to step S2970. Note that, depending on the settings and implementation form of the image processing device 2800, the image segmentation process in step S2940 may be performed even if it is determined that the input image is not processable based on some of the imaging part, imaging method, and imaging angle of view.

[0456] When the process proceeds to step S2940, the image segmentation process according to this embodiment shown in Fig. 37 starts. In the image segmentation process according to this embodiment, first, in step S3710, the segmentation processing unit 2804 enlarges or reduces the input image to a certain resolution set for the training data, and generates a deformed image.

[0457] Next, in step S3720, the segmentation processing unit 2804 performs padding on the generated deformed image so that the image size is the image size set for the training data, and generates a padded image. At this time, the segmentation processing unit 2804 performs padding on the area to be padded with a fixed pixel value, with a neighboring pixel value, or with mirror padding in accordance with the characteristics of the trained model so that image segmentation processing can be performed effectively.

[0458] In step S3730, the segmentation processing unit 2804 inputs the padding image to the image segmentation engine and obtains a first region label image that has been subjected to image segmentation processing.

[0459] Next, in step S3740, the segmentation processing unit 2804 performs trimming on the first region label image by the area padded in step S3720, and generates a second region label image.

[0460] Then, in step S3750, the segmentation processing unit 2804 reduces or enlarges the second region label image to the image size of the input image to generate a final region label image. When the segmentation processing unit 2804 generates the final region label image in step S3750, the image segmentation processing according to this embodiment ends, and the process proceeds to step S2950. The process after step S2950 is the same as the process after step S2950 in the eighth embodiment, and therefore the description thereof will be omitted.

[0461] As described above, the segmentation processing unit 2804 according to this embodiment adjusts the image size of the input image so that the resolution of the input image becomes a predetermined resolution. In addition, the segmentation processing unit 2804 generates a padded image for the input image whose image size has been adjusted so that the image size can be handled by the image segmentation engine, and inputs the padded image to the image segmentation engine. After that, the segmentation processing unit 2804 trims the output image from the image segmentation engine by the amount of the padded area. Then, the segmentation processing unit 2804 generates a region label image by adjusting the image size of the trimmed image to the original image size of the input image.

[0462] As a result, the segmentation processing unit 2804 of this embodiment can perform image segmentation processing by the image segmentation engine to generate a region label image even for an input image of an image size that could not be handled by the embodiment 8. Furthermore, by using an image segmentation engine trained with teacher data based on the resolution, it may be possible to perform image segmentation processing of an input image more efficiently than the image segmentation engine according to the embodiment 10 that processes images of the same image size.

[0463] Example 13 Next, an image processing device according to a thirteenth embodiment will be described with reference to Figs. 28, 29, and 38 to 40. In this embodiment, a segmentation processing unit performs image segmentation processing on an input image for each region of a certain image size, thereby generating a region label image.

[0464] Unless otherwise specified, the configuration and processing of the image processing device according to this embodiment are the same as those of the image processing device 2800 according to the eighth embodiment. Therefore, the image processing device according to this embodiment will be described below, focusing on the differences from the image processing device according to the eighth embodiment. Since the configuration of the image processing device according to this embodiment is the same as that of the image processing device according to the eighth embodiment, the configuration shown in FIG. 28 is indicated by the same reference numerals, and description thereof will be omitted.

[0465] The segmentation processing unit 2804 according to this embodiment includes an image segmentation engine similar to that of the embodiment 8. However, in this embodiment, training data used for learning a machine learning model included in the image segmentation engine is different from the training data in the embodiment 8. Specifically, a pair group of input data which is an input image and output data which is a region label image corresponding to the input image, which constitutes the training data, is constituted by rectangular region images of a certain image size in which the positional relationship between the input image and the region label image corresponds.

[0466] Here, the teacher data of the image segmentation engine according to the present embodiment will be described with reference to Fig. 38. As shown in Fig. 38, for one of the pairs constituting the teacher data, for example, consider a case where there is an image Im3810 which is an input image and an area label image Im3820 which is a corresponding area label image. In this case, in the eighth embodiment, the input data of the teacher data is the image Im3810, and the output data is the area label image Im3820.

[0467] In contrast to this, in this embodiment, rectangular area image R3811 of image Im3810 is used as input data, and rectangular area image R3821, which is the same (corresponding) shooting area as rectangular area image R3811 in area label image Im3820, is used as output data. Then, rectangular area image R3811, which is the input data, and rectangular area image R3821, which is the output data, form a pair of teacher data (hereinafter, a first rectangular area image pair).

[0468] Here, the rectangular region image R3811 and the rectangular region image R3821 are images of a fixed image size. The image Im3810 and the region label image Im3820 may be aligned by any method. The corresponding positional relationship between the rectangular region image R3811 and the rectangular region image R3821 may be specified by any method such as template matching. Depending on the design of the machine learning model included in the image segmentation engine, the image sizes and the number of dimensions of the input data and the output data may differ. For example, this is the case when the input data is part of a B-scan image (two-dimensional image) and the output data is part of an A-scan image (one-dimensional).

[0469] The above-mentioned certain image size can be determined, for example, from a common divisor of the pixel counts of the corresponding dimensions for a group of image sizes of images assumed to be processed (input images). In this case, the positional relationship of the rectangular region images output by the image segmentation engine may be prevented from overlapping.

[0470] Specifically, for example, consider a case where the image assumed to be processed is a two-dimensional image, and a first image size of the image size group is 500 pixels wide and 500 pixels high, and a second image size is 100 pixels wide and 100 pixels high. Here, a fixed image size for the rectangular area images R3811 and R3821 is selected from the common divisor of each side. In this case, for example, the fixed image size is selected from 100 pixels wide and 100 pixels high, 50 pixels wide and 50 pixels high, or 25 pixels wide and 25 pixels high. If the image assumed to be processed is three-dimensional, the number of pixels is determined for the width, height, and depth.

[0471] Note that multiple rectangular areas can be set for one pair of an image corresponding to the input data and an area label image corresponding to the output data. For this reason, for example, rectangular area image R3812 in image Im3810 is set as input data, and rectangular area image R3822, which is the same shooting area as rectangular area image R3812 in area label image Im3820, is set as output data. Then, a pair of teacher data is formed by rectangular area image R3812, which is the input data, and rectangular area image R3822, which is the output data. This makes it possible to create a rectangular area image pair other than the first rectangular area image pair.

[0472] In addition, the pair group constituting the training data can be enriched by creating many pairs of rectangular area images while changing the image of the rectangular area to an image with different coordinates. Then, an image segmentation engine trained using the pair group constituting the training data can be expected to perform efficient image segmentation processing. However, pairs that do not contribute to the image segmentation processing of the trained model can be prevented from being added to the training data.

[0473] As the input data and output data of the training data, an image depicting one layer or a region to which one label is attached can be used as the training data. Also, as the input data and output data of the training data, an image of a region depicting a plurality of layers, for example, two layers, more preferably three or more layers, can be used. Similarly, an image of a region depicting a plurality of regions to which labels are separated in a region label image can be used. In these cases, it is expected that the image segmentation process can be performed more appropriately using a trained model based on the positional relationship of the trained layer or region, compared to the case where an image depicting one layer or a region to which one label is attached is used as the training data.

[0474] Furthermore, for example, consider a case where the structures and positions of the objects depicted in the pair of rectangular area images created from the input image and rectangular area images created from the area label image are significantly different. In this case, an image segmentation engine that has learned using such training data may output an area label image with low accuracy. Therefore, such pairs can be removed from the training data.

[0475] As in the eighth embodiment, the input data of the teacher data uses an image having a specific imaging condition assumed to be a processing target, and the specific imaging condition is a predetermined imaging part, imaging method, and imaging angle of view. That is, unlike the eighth embodiment, the specific imaging condition in this embodiment does not include an image size.

[0476] The segmentation processing unit 2804 according to this embodiment uses an image segmentation engine that has been trained using such teacher data to perform image segmentation processing on an input image to generate a region label image. At this time, the segmentation processing unit 2804 divides the input image into a group of rectangular region images that are continuous without gaps and have a certain image size set for the teacher data. The segmentation processing unit 2804 uses the image segmentation engine to perform image segmentation processing on each of the divided rectangular region image groups to generate a group of divided region label images. After that, the segmentation processing unit 2804 arranges and combines the generated group of divided region label images according to the positional relationship of the input image to generate a final region label image.

[0477] In this way, the segmentation processing unit 2804 of this embodiment performs image segmentation processing on the input image in units of rectangular regions, and combines the images that have been subjected to the image segmentation processing. This makes it possible to perform image segmentation processing on images of image sizes that were not supported in the eighth embodiment, and generate region label images.

[0478] Next, a series of image processing according to this embodiment will be described with reference to Figs. 29, 39, and 40. Fig. 39 is a flowchart of image segmentation processing according to this embodiment. Note that the processing of steps S2910, S2920, and S2950 to S2970 according to this embodiment is similar to those in the eighth embodiment, and therefore description thereof will be omitted. Note that, when image segmentation processing is performed unconditionally on the input image with respect to shooting conditions other than the image size, after the processing of step S2920, the processing of step S2930 may be omitted and the processing may proceed to step S2940.

[0479] In step S2920, as in the eighth embodiment, when the imaging condition acquisition unit 2802 acquires the imaging condition group of the input image, the process proceeds to step S2930. In step S2930, the processability determination unit 2803 uses the acquired imaging condition group to determine whether the input image can be handled by the image segmentation engine. Specifically, the processability determination unit 2803 determines whether the imaging conditions of the input image are imaging body part, imaging method, and imaging angle of view that the image segmentation engine can handle. Unlike the eighth embodiment, the processability determination unit 2803 does not determine the image size.

[0480] If the processing feasibility determination unit 2803 determines the imaging part, imaging method, and imaging angle of view, and determines that the input image is processable, the process proceeds to step S2940. On the other hand, if the processing feasibility determination unit 2803 determines that the image segmentation engine cannot process the input image based on these imaging conditions, the process proceeds to step S2970. Note that, depending on the settings and implementation form of the image processing device 2800, the image segmentation process in step S2940 may be performed even if it is determined that the input image is not processable based on some of the imaging part, imaging method, and imaging angle of view.

[0481] When the process proceeds to step S2940, the image segmentation process according to this embodiment shown in FIG. 39 is started. In the image segmentation process according to this embodiment, first, in step S3910, as shown in FIG. 40, an input image is divided into a group of rectangular area images of a certain image size set for the teacher data, which are continuous without gaps. Here, FIG. 40 shows an example in which an input image Im4010 is divided into a group of rectangular area images R4011 to R4026 of a certain image size. Note that the image size and the number of dimensions of the input image and the output image may differ depending on the design of the machine learning model included in the image segmentation engine. In that case, the division positions of the input image described above are adjusted by overlapping or separating so that there is no loss in the combined area label image generated in step S3920.

[0482] Next, in step S3920, the segmentation processing unit 2804 performs image segmentation processing on each of the rectangular region images R4011 to R4026 using an image segmentation engine, and generates a group of segmented region label images.

[0483] Then, in step S3930, the segmentation processing unit 2804 combines the generated divided region label images in the same positional relationship as the rectangular region images R4011 to R4026 generated by dividing the input image, thereby enabling the segmentation processing unit 2804 to generate region label images.

[0484] When the segmentation processing unit 2804 generates the region label image in step S3930, the image segmentation processing according to this embodiment ends, and the process proceeds to step S2950. The process after step S2950 is the same as the process after step S2950 in the eighth embodiment, and therefore the description thereof will be omitted.

[0485] As described above, the segmentation processing unit 2804 according to this embodiment divides an input image into a plurality of rectangular area images R4011-R4026 of a predetermined image size. After that, the segmentation processing unit 2804 inputs the divided rectangular area images R4011-R4026 to an image segmentation engine to generate a plurality of divided area label images, and integrates the plurality of divided area label images to generate an area label image. Note that if rectangular area groups overlap in positional relationship during integration, pixel value groups of the rectangular area groups are integrated or overwritten.

[0486] As a result, the segmentation processing unit 2804 of this embodiment can perform image segmentation processing using an image segmentation engine to generate a region label image even for an input image of an image size that could not be handled in the eighth embodiment. Also, if the training data is created fro...

Claims

1. An acquisition unit for acquiring a tomographic image of the subject's eye; A trained model obtained by training using training data including a tomographic image of a subject's eye and an image including positional information related to at least one layer in the tomographic image, the trained model using the acquired tomographic image as input data of the trained model, and outputting an image including positional information related to at least one layer in the acquired tomographic image as output data of the trained model, thereby acquiring positional information related to at least one layer in the acquired tomographic image; a generating unit that generates a front image corresponding to at least a part of a depth range in a three-dimensional tomographic image of the subject's eye, the depth range being determined using the acquired position information related to at least one layer; A medical image processing device comprising:

2. The medical image processing apparatus according to claim 1 , wherein the generating unit generates a motion contrast front image corresponding to the determined depth range by using three-dimensional motion contrast data corresponding to the three-dimensional tomographic image.

3. 3. The medical image processing device according to claim 1, wherein the three-dimensional tomographic image is a plurality of two-dimensional tomographic images including the acquired tomographic image, the three-dimensional tomographic image being composed of a plurality of two-dimensional tomographic images at different positions of the subject's eye.

4. An acquisition unit for acquiring a tomographic image of the subject's eye; A trained model obtained by training using training data including a tomographic image of a subject's eye and an image including positional information related to at least one layer in the tomographic image, the trained model using the acquired tomographic image as input data of the trained model, and outputting an image including positional information related to at least one layer in the acquired tomographic image as output data of the trained model, thereby acquiring positional information related to at least one layer in the acquired tomographic image; a generating unit that generates an image analysis result regarding the acquired tomographic image by using position information regarding the acquired at least one layer; A medical image processing device comprising:

5. 5. The medical image processing device according to claim 4, wherein the generation unit uses position information relating to the acquired at least one layer to generate, as the image analysis result, at least one of a thickness graph showing a thickness of the at least one layer in the acquired tomographic image and a thickness map showing the thickness as a color map.

6. The medical image processing apparatus according to claim 1 , wherein the processing unit corrects the acquired position information related to at least one layer in response to an instruction from an operator.

7. An acquisition unit for acquiring a tomographic image of the subject's eye; A trained model obtained by training using training data including a tomographic image of a subject's eye and an image including positional information related to at least one layer in the tomographic image, the trained model using the acquired tomographic image as input data of the trained model, and outputting an image including positional information related to at least one layer in the acquired tomographic image as output data of the trained model, thereby acquiring positional information related to at least one layer in the acquired tomographic image; Equipped with The processing unit corrects the acquired position information relating to at least one layer in response to an instruction from an operator.

8. The medical image processing device according to claim 1 , wherein the processing unit acquires position information regarding another layer that cannot be obtained from the position information regarding the at least one acquired layer, using the position information regarding the at least one acquired layer.

9. The medical image processing device according to claim 1 , wherein the processing unit acquires position information regarding other layers that cannot be obtained from the position information regarding the at least one layer acquired, without using a trained model obtained by machine learning.

10. 10. The medical image processing device according to claim 8 or 9, wherein the processing unit acquires position information regarding a plurality of layers in the acquired tomographic image as position information regarding at least one layer in the acquired tomographic image, and acquires position information regarding at least one layer between the plurality of layers as position information regarding the other layer, by using the acquired tomographic image as input data for the learned model.

11. The medical image processing device according to claim 10, wherein the processing unit acquires a plurality of layer boundaries from the boundary between the internal limiting membrane and nerve fiber layer of the test eye to the sclera as positional information regarding the plurality of layers by using the acquired tomographic image as input data for the trained model, and acquires at least one layer boundary between the plurality of layer boundaries as positional information regarding at least one layer between the plurality of layers.

12. The medical image processing device according to claim 8 , wherein the processing unit acquires position information regarding at least one layer in the acquired tomographic image and position information regarding the other layers, using the same or different computing devices.

13. 13. The medical image processing device according to claim 1, further comprising a display control unit that causes a two-dimensional tomographic image corresponding to a position determined in response to an instruction from an examiner to be displayed on a display unit with position information regarding the acquired at least one layer superimposed thereon.

14. A display control unit that displays an image of the subject's eye on a display unit, The acquisition unit generates an image having a higher image quality than the image of the test eye by using the image of the test eye as input data of a trained model different from a trained model for acquiring positional information related to at least one layer in the acquired tomographic image, The medical image processing apparatus according to claim 1 , wherein the display control unit causes the generated image to be displayed on the display unit.

15. The medical image processing apparatus according to claim 1 , further comprising a display control unit that causes a display unit to display a message indicating that the processing unit was unable to properly acquire position information regarding at least one layer in the acquired tomographic image.

16. The medical image processing device according to claim 1 , further comprising a display control unit that causes a display unit to display that the acquired position information regarding the at least one layer has been acquired using a trained model.

17. A medical image processing device as described in any one of claims 1 to 16, wherein an image including positional information regarding at least one layer in a tomographic image included in the learning data is an image labeled for each pixel.

18. The acquisition unit further acquires a front image of the fundus of the subject's eye, The processing unit acquires information regarding the region of the optic disc in the acquired front fundus image by using the acquired front fundus image as input data for a trained model obtained by learning using training data including a front fundus image of the test eye and information regarding the region of the optic disc in the front fundus image. A medical image processing device according to any one of claims 1 to 17.

19. The medical image processing device according to claim 18, wherein the processing unit acquires position information regarding at least one layer of the optic disc in the acquired tomographic image by using the position information regarding the acquired at least one layer and the information regarding the acquired region of the optic disc.

20. The medical image processing device according to claim 18 , wherein the processing unit calculates an optic disc cup / optic disc ratio using the acquired information on the region of the optic disc.

21. The trained model includes a plurality of trained models, The processing unit includes: Selecting a trained model to be used for acquiring position information regarding at least one layer in the acquired tomographic image based on the imaging conditions of the acquired tomographic image and the training data of the plurality of trained models; The medical image processing device according to any one of claims 1 to 20, wherein the acquired tomographic image is used as input data for the selected trained model, and an image including position information regarding at least one layer in the acquired tomographic image is output as output data for the selected trained model, thereby acquiring position information regarding at least one layer in the acquired tomographic image.

22. The trained model includes a plurality of trained models, 21. The medical image processing apparatus according to claim 1, wherein the processing unit selects, in response to an instruction from an operator, one of a plurality of acquisition results obtained by using the acquired tomographic image as input data for the plurality of trained models as an output.

23. The processing unit includes: By using the acquired tomographic image as input data of a trained model different from a trained model for acquiring position information related to at least one layer in the acquired tomographic image, a tomographic image having higher image quality than the acquired tomographic image is generated; 23. The medical image processing apparatus according to claim 1, wherein the generated tomographic image is used as input data for a trained model for acquiring position information regarding at least one layer in the acquired tomographic image, thereby acquiring position information regarding at least one layer in the acquired tomographic image.

24. The acquisition unit acquires a three-dimensional tomographic image of the subject's eye, 24. The medical image processing device according to claim 1, wherein the processing unit divides the acquired three-dimensional tomographic image into a plurality of two-dimensional images, and uses each of the divided two-dimensional images as input data for a trained model for acquiring positional information regarding at least one layer in the acquired tomographic image.

25. A medical image processing device as described in any one of claims 1 to 24, wherein the processing unit outputs an image including position information regarding at least one layer in the acquired tomographic image as output data of the trained model, and acquires position information regarding at least one layer in the acquired tomographic image using the output data of the trained model.

26. An imaging device for imaging the subject's eye; A medical image processing apparatus according to any one of claims 1 to 25, which is communicably connected to the imaging apparatus; A medical image processing system comprising:

27. acquiring a tomographic image of the subject's eye; A trained model obtained by training using training data including a tomographic image of a subject's eye and an image including positional information related to at least one layer in the tomographic image, the trained model using the acquired tomographic image as input data of the trained model, and outputting an image including positional information related to at least one layer in the acquired tomographic image as output data of the trained model, thereby acquiring positional information related to at least one layer in the acquired tomographic image; generating a front image corresponding to at least a part of a depth range in a three-dimensional tomographic image of the subject's eye, the depth range being determined using the acquired position information related to at least one layer; A medical image processing method comprising:

28. acquiring a tomographic image of the subject's eye; A trained model obtained by training using training data including a tomographic image of a subject's eye and an image including positional information related to at least one layer in the tomographic image, the trained model using the acquired tomographic image as input data of the trained model, and outputting an image including positional information related to at least one layer in the acquired tomographic image as output data of the trained model, thereby acquiring positional information related to at least one layer in the acquired tomographic image; generating an image analysis result for the acquired tomographic image using position information for the acquired at least one layer; A medical image processing method comprising:

29. acquiring a tomographic image of the subject's eye; A trained model obtained by training using training data including a tomographic image of a subject's eye and an image including positional information related to at least one layer in the tomographic image, the trained model using the acquired tomographic image as input data of the trained model, and outputting an image including positional information related to at least one layer in the acquired tomographic image as output data of the trained model, thereby acquiring positional information related to at least one layer in the acquired tomographic image; modifying the acquired position information relating to at least one layer in response to an instruction from an operator; A medical image processing method comprising:

30. A program which, when executed by a processor, causes the processor to perform each step of the medical image processing method according to any one of claims 27 to 29.

Citation Information

Patent Citations

  • Fundus observation apparatus, fundus image display device, and fundus observation program

    JP2008073099A

  • Automatic nipple inversion ratio measurement system

    JP2011520503A

  • Image processor, program, and storage medium

    JP2012019958A

  • Method for determining the depth-resolved physical and / or optical properties of a scattering medium.

    JP2014516646A

  • Ophthalmologic apparatus, display control method, and program

    JP2017047111A