Medical image processing device, method, computer readable medium and learning model

Abstract

The present invention relates to a medical image processing device, a method, a computer-readable medium, and a learning model. The medical image processing device includes: an acquisition unit that acquires a tomographic image of an eye to be inspected; and a first processing unit that performs a first detection process for detecting at least one of a plurality of retinal layers in the acquired tomographic image by using a learning model obtained by training data indicating at least one of a plurality of retinal layers in the tomographic image of the eye to be inspected.

Description

Technical Field

[0001] The present invention relates to a medical image processing device, a medical image processing method, a computer-readable medium and a learning model. Background Art

[0002] A tomographic imaging device for an eye, such as an apparatus using optical coherence tomography (OCT) (OCT apparatus), can three-dimensionally observe the state of the inside of the retinal layer. The tomographic imaging device has attracted attention in recent years because the tomographic imaging device can be used to make a diagnosis more accurately.

[0003] As a form of OCT, for example, there is TD-OCT (Time Domain OCT) which combines a broadband light source and a Michelson interferometer. This is configured to measure interference light with backscattered light of a signal arm by scanning the delay of a reference arm, and obtain information on depth resolution. However, in such TD-OCT, it is difficult to quickly obtain an image.

[0004] Therefore, as a method for obtaining images more quickly, SD-OCT (spectral domain OCT) is known, which has a technique of using a broadband light source and obtaining an interference pattern by a spectroscope. In addition, SS-OCT (swept source OCT) is known, which has a technique of using a fast wavelength swept light source as a light source and measuring spectral interference by a single-channel photodetector.

[0005] In the case of obtaining a tomographic image captured by OCT, when the thickness of the nerve fiber layer can be measured, the progression of a disease such as glaucoma and the recovery state after treatment can be quantitatively diagnosed. Patent document 1 discloses a technology for quantitatively measuring the thickness of the layers by detecting the boundaries of each retinal layer from a tomographic image using a computer and measuring the thickness of each layer.

[0006] [Citation List]

[0007] [Patent Document]

[0008] Patent Document 1: Japanese Patent Application Laid-Open No. 2008-73099 Summary of the invention

[0009] [Technical issues]

[0010] However, the conventional technology has the following problems. Since there is disappearance of layers, hemorrhage, and occurrence of vitiligo and new blood vessels in diseased eyes, the shape of the retina becomes irregular. Therefore, in the conventional image processing method that determines the result of image feature extraction by utilizing the regularity of the shape of the retina and performing boundary detection of the retinal layer, there is a limitation that false detection occurs when the boundary detection of the retinal layer is automatically performed.

[0011] Therefore, an object of the present invention is to provide a medical image processing device, a medical image processing method, a computer-readable medium storing a program, and a learning model that can detect the boundary of the retinal layer regardless of the disease, location, etc.

[0012] [Solution to the problem]

[0013] According to one embodiment of the present invention, a medical image processing device includes: an acquisition unit, which is configured to obtain a tomographic image of an eye to be inspected; and a first processing unit, which is configured to perform a first detection process for detecting at least one of a plurality of retinal layers in the obtained tomographic image by using a learning model, wherein the learning model is obtained by training data indicating at least one of a plurality of retinal layers in the tomographic image of the eye to be inspected.

[0014] In addition, a medical image processing method according to another embodiment of the present invention includes: obtaining a tomographic image of an eye to be inspected; and performing a first detection process for detecting at least one of a plurality of retinal layers in the obtained tomographic image by using a learning model, wherein the learning model is obtained by training data indicating at least one of a plurality of retinal layers in the tomographic image of the eye to be inspected.

[0015] Further features of the present invention will become apparent from the following description of exemplary examples with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] [ Figure 1 ]

[0017] Figure 1 An example of a schematic configuration of an image processing system according to Example 1 is shown.

[0018] [ Figure 2A ]

[0019] Figure 2A is a diagram used to describe the eye.

[0020] [ Figure 2B ]

[0021] Figure 2B A diagram used to describe a tomographic image.

[0022] [ Figure 2C ]

[0023] Figure 2C A diagram used to describe fundus images.

[0024] [ Figure 3 ]

[0025] Figure 3is a flowchart of a series of processes according to Example 1.

[0026] [ Figure 4A ]

[0027] Figure 4A is a diagram for describing examples of learning images.

[0028] [ Figure 4B ]

[0029] Figure 4B is a diagram for describing examples of learning images.

[0030] [ Figure 5A ]

[0031] Figure 5A is a diagram for describing an example of the size of a learning image.

[0032] [ Figure 5B ]

[0033] Figure 5B is a diagram for describing an example of the size of a learning image.

[0034] [ Figure 5C ]

[0035] Figure 5C is a diagram for describing an example of the size of a learning image.

[0036] [ Figure 6 ]

[0037] Figure 6 is a diagram for describing an example of a machine learning model according to Example 1.

[0038] [ Figure 7 ]

[0039] Figure 7 An example of a display screen is shown.

[0040] [ Figure 8 ]

[0041] Figure 8 An example of a schematic configuration of an image processing system according to Example 2 is shown.

[0042] [ Fig.9A ]

[0043] Fig.9A is a flowchart of a series of processes according to Example 2.

[0044] [ Fig. 9B ]

[0045] Fig. 9B is a flowchart of a boundary detection process according to Example 2.

[0046] [ Fig. 10A ]

[0047] Fig. 10A is a diagram used to describe the detection of retinal areas.

[0048] [ Fig. 10B ]

[0049] Fig. 10B is a diagram used to describe the detection of retinal areas.

[0050] [ Fig.11A ]

[0051] Fig.11A is a diagram for describing an example of the size of a learning image.

[0052] [ Fig. 11B ]

[0053] Fig. 11B is a diagram for describing an example of the size of a learning image.

[0054] [ Fig. 11C ]

[0055] Fig. 11C is a diagram for describing an example of the size of a learning image.

[0056] [ Fig.12 ]

[0057] Fig.12 is a diagram for describing an example of a machine learning model according to Example 2.

[0058] [ Fig.13A ]

[0059] Fig.13A is a diagram for describing retinal layer detection according to Example 2.

[0060] [ Fig. 13B ]

[0061] Fig. 13B is a diagram for describing retinal layer detection according to Example 2.

[0062] [ Fig. 13C ]

[0063] Fig. 13C is a diagram for describing retinal layer detection according to Example 2.

[0064] [ Fig.13D ]

[0065] Fig.13D is a diagram for describing retinal layer detection according to Example 2.

[0066] [ Fig.14A ]

[0067] Fig.14A is a diagram for describing examples of input and output images in a learning model.

[0068] [ Fig. 14B ]

[0069] Fig. 14B is a diagram for describing examples of input and output images in a learning model.

[0070] [ Fig. 14C ]

[0071] Fig. 14C is a diagram for describing examples of input and output images in a learning model.

[0072] [ Fig.14D ]

[0073] Fig.14D is a diagram for describing examples of input and output images in a learning model.

[0074] [ Fig.15 ]

[0075] Fig.15 is an example of a schematic configuration of an image processing system according to Example 4.

[0076] [ Fig.16A ]

[0077] Fig.16A is a flowchart of a series of processes according to Example 4.

[0078] [ Fig. 16B ]

[0079] Fig. 16B is a flowchart of a series of processes according to Example 4.

[0080] [ Fig.17 ]

[0081] Fig.17 An example of a schematic configuration of an image processing system according to Example 5 is shown.

[0082] [ Fig.18A ]

[0083] Fig.18A is a flowchart of a series of processes according to Example 5.

[0084] [ Fig.18B ]

[0085] Fig.18B is a flowchart of a boundary detection process according to Example 5.

[0086] [ Fig.19A ]

[0087] Fig.19A is a diagram for describing compensation processing of the retinal area.

[0088] [ Fig.19B ]

[0089] Fig.19B is a diagram for describing compensation processing of the retinal area.

[0090] [ Fig.19C ]

[0091] Fig.19C is a diagram for describing compensation processing of the retinal area.

[0092] [ Fig.19D ]

[0093] Fig.19D is a diagram for describing compensation processing of the retinal area.

[0094] [ Fig. 20 ]

[0095] Fig. 20 is a diagram for describing an example of a learning image according to Example 6.

[0096] [ Fig.21A ]

[0097] Fig.21A Examples of multiple OCTA En-Face images are shown.

[0098] [ Fig.21B ]

[0099] Fig.21B Examples of a plurality of brightness tomographic images are shown.

[0100] [ Fig.22A ]

[0101] Fig.22A An example of a user interface according to Example 7 is shown.

[0102] [ Fig. 22B ]

[0103] Fig. 22B An example of a user interface according to Example 7 is shown.

[0104] [ Fig.23 ]

[0105] Fig.23 An example of a user interface according to Example 7 is shown.

[0106] [ Fig.24 ]

[0107] Fig.24 An example of an area label image related to the description of a term is shown.

[0108] [ Fig.25 ]

[0109] Fig.25 An example of the construction of a neural network is shown in relation to the description of terms.

[0110] [ Fig.26 ]

[0111] Fig.26 An example of the construction of a neural network is shown in relation to the description of terms.

[0112] [ Fig. 27 ]

[0113] Fig. 27 An example of an area label image related to the description of a term is shown.

[0114] [ Fig.28 ]

[0115] Fig.28 An example of a configuration of an image processing apparatus according to Example 8 is shown.

[0116] [ Fig.29 ]

[0117] Fig.29 is a flowchart illustrating an example of a processing flow of an image processing apparatus according to Example 8.

[0118] [ Fig.30 ]

[0119] Fig.30 is a flowchart illustrating an example of a processing flow of an image processing apparatus according to Example 8.

[0120] [ Fig.31 ]

[0121] Fig.31 is a diagram illustrating an example of a user interface included in an imaging apparatus according to Example 8.

[0122] [ Fig.32 ]

[0123] Fig.32 is a diagram illustrating an example of a user interface included in an imaging apparatus according to Example 8.

[0124] [ Fig.33 ]

[0125] Fig.33 : is a flowchart showing an example of a processing flow of an image processing apparatus according to Example 9.

[0126] [ Fig.34 ]

[0127] Fig.34 Image processing according to Example 11 is shown.

[0128] [ Fig.35 ]

[0129] Fig.35 is a flowchart illustrating an example of a processing flow of an image processing apparatus according to Example 11.

[0130] [ Fig.36 ]

[0131] Fig.36 Image processing according to Example 12 is shown.

[0132] [ Fig.37 ]

[0133] Fig.37 is a flowchart illustrating an example of a processing flow of an image processing apparatus according to Example 13.

[0134] [ Fig.38 ]

[0135] Fig.38 Image processing according to Example 13 is shown.

[0136] [ Fig.39 ]

[0137] Fig.39 is a flowchart illustrating an example of a processing flow of an image processing apparatus according to Example 13.

[0138] [ Fig.40 ]

[0139] Fig.40 Image processing according to Example 13 is shown.

[0140] [ Fig.41 ]

[0141] Fig.41 is a flowchart illustrating an example of a processing flow of an image processing apparatus according to Example 14.

[0142] [ Fig.42 ]

[0143] Fig.42 is a diagram showing an example of a user interface included in an imaging device according to Example 15.

[0144] [ Fig.43]

[0145] Fig.43 An example of the configuration of an image processing device according to Example 18 is shown.

[0146] [ Fig.44 ]

[0147] Fig.44 An example of the configuration of an image processing device according to Example 19 is shown.

[0148] [ Fig.45 ]

[0149] Fig.45 is a flowchart showing an example of a processing flow of an image processing device according to Example 19.

[0150] [ Fig.46A ]

[0151] Fig.46A An example of the construction of a neural network used as a machine learning model according to Modification 9 is shown.

[0152] [ Fig.46B ]

[0153] Fig.46B An example of the construction of a neural network used as a machine learning model according to Modification 9 is shown.

[0154] [ Fig.47A ]

[0155] Fig.47A An example of the construction of a neural network used as a machine learning model according to Modification 9 is shown.

[0156] [ Fig.47B ]

[0157] Fig.47B An example of the construction of a neural network used as a machine learning model according to Modification 9 is shown. DETAILED DESCRIPTION

[0158] Hereinafter, exemplary examples for implementing the present invention will be described in detail with reference to the accompanying drawings.

[0159] However, the dimensions, materials, shapes, and relative positions of the components described in the following examples are arbitrary and may be changed according to the configuration of the device to which the present invention is applied or according to various conditions. In addition, the same reference numerals are used throughout the drawings to represent the same or functionally similar elements in the drawings.

[0160] (Example 1)

[0161] In the following, reference is made to Figures 1 to 7, an image processing system including an image processing device using a tomographic image of an eye according to Example 1 of the present invention will be described. In this example, all target retinal layers are detected by using a learning model related to a machine learning model. Note that, hereinafter, a machine learning model refers to a learning model having a machine learning algorithm such as deep learning. In addition, a learning model is a model that has been trained (learned) in advance on a machine learning model having an arbitrary machine learning algorithm by using appropriate training data. However, this does not mean that the learning model is no longer learned, and the learning model should also be capable of incremental learning. Note that, hereinafter, training data refers to training data, and includes a pair of input data and ground truth. In addition, the correct answer data refers to the ground truth of the training data.

[0162] Figure 1 An example of a schematic configuration of an image processing system 1 including an image processing apparatus 20 (medical image processing apparatus) according to the present example is shown. Figure 1 As shown, the image processing system 1 is provided with an OCT apparatus 10 as an example of a tomographic imaging apparatus, an image processing apparatus 20 , a fundus image imaging apparatus 30 , an external storage apparatus 40 , a display unit 50 , and an input unit 60 .

[0163] The OCT apparatus 10 is an example of a tomographic imaging apparatus that is an apparatus for capturing a tomographic image of an eye to be inspected. Any kind of OCT apparatus may be used as the OCT apparatus, for example, SD-OCT or SS-OCT may be used.

[0164] The image processing device 20 is connected to the OCT device 10, the fundus image camera 30, the external storage device 40, the display unit 50, and the input unit 60 via an interface, and can control them. The image processing device 20 can generate various images, such as a tomographic image and an En-Face image (frontal image) of the eye to be inspected, based on various signals obtained from the OCT device 10, the fundus image camera 30, and the external storage device 40. In addition, the image processing device 20 can perform image processing on these images. Note that the image processing device 20 may include a general-purpose computer, or may include a dedicated computer of the image processing system 1.

[0165] The fundus image pickup device 30 is a device for picking up a fundus image of the eye to be inspected, and for example, a fundus camera or SLO (Scanning Laser Ophthalmoscope) can be used as the device. Note that the device configurations of the OCT device 10 and the fundus image pickup device 30 may be integrated or separated.

[0166] The external storage device 40 associates and stores information related to the eye to be inspected (the patient's name, age, sex, etc.) with various camera image data, shooting parameters, image analysis parameters, and parameters set by the operator. The external storage device 40 may include any storage device and may include a storage medium such as an optical disk or a memory.

[0167] The display unit 50 includes any display, and can display information related to the eye to be inspected and various images according to the control of the image processing device 20 .

[0168] The input unit 60 is, for example, a mouse, a keyboard, or a touch operation screen, and the operator can input instructions for the image processing device 20, the OCT device 10, and the fundus image pickup device 30 to the image processing device 20 via the input unit 60. Note that when the input unit 60 is a touch operation screen, the input unit 60 can be constructed integrally with the display unit 50.

[0169] Please note that although these components are Figure 1 Although shown as separate bodies, some or all of these components may be constructed as an integral unit.

[0170] Next, the OCT apparatus 10 will be described. The OCT apparatus 10 is provided with a light source 11, a galvanometer mirror 12, a focusing lens stage 13, a coherence gate stage 14, a detector 15, and an internal fixation lamp 16. Note that since the OCT apparatus 10 is a known apparatus, a detailed description will be omitted, and here, the capturing of tomographic images according to instructions from the image processing apparatus 20 will be described.

[0171] When an instruction for imaging is transmitted from the image processing device 20, the light source 11 emits light. The light from the light source 11 is divided into measurement light and reference light by a spectrometer (not shown). The OCT device 10 can generate an interference signal including tomographic information of the subject (eye to be inspected) by irradiating the subject with the measurement light and detecting the return light from the subject and the interference light with the reference light.

[0172] The galvanometer mirror 12 is used to scan the measurement light in the fundus of the eye to be inspected, and the imaging range of the fundus by OCT imaging can be specified by the galvanometer mirror 12 through the scanning range of the measurement light. The image processing device 20 can specify the imaging range of the fundus and the number of scanning lines in the plane direction (scanning speed in the plane direction) by controlling the driving range and speed of the galvanometer mirror 12. Figure 1In order to simplify the description, the galvanometer mirror 12 is shown as one unit, but in fact, the galvanometer mirror 12 includes two mirrors, namely, a mirror for X scanning and a mirror for Y scanning, and can scan a desired range in the fundus with the measurement light. Note that the configuration of the scanning unit for scanning the measurement light is not limited to the galvanometer mirror, and other arbitrary deflection mirrors can be used. In addition, a deflection mirror capable of scanning the measurement light in two-dimensional directions using one mirror (e.g., a MEMS mirror) can be used as the scanning unit.

[0173] The focusing lens stage 13 is provided with a focusing lens not shown. By moving the focusing lens stage 13, the focusing lens can be moved along the optical axis of the measurement light. Therefore, the measurement light can be focused to the retinal layer of the fundus via the anterior segment of the eye to be inspected by the focusing lens. The measurement light irradiated to the fundus is reflected and scattered by each retinal layer and returns as return light on the optical path.

[0174] The coherence gate stage 14 is used to adjust the optical path length of the reference light or the measuring light to correspond to the difference in the axial length of the eye to be inspected, etc. In the present example, the coherence gate stage 14 includes a stage provided with a reflector, and can make the optical path length of the reference light correspond to the optical path length of the measuring light by moving along the optical axis direction in the optical path of the reference light. Here, the coherence gate represents a position where the optical distances of the measuring light and the reference light are equal in OCT. The coherence gate stage 14 can be controlled by the image processing device 20. By controlling the position of the coherence gate by the coherence gate stage 14, the image processing device 20 can control the imaging range in the depth direction of the eye to be inspected, and can control the imaging on one side of the retinal layer or the imaging on the side of a portion deeper than the retinal layer, etc.

[0175] The detector 15 detects the return light of the measurement light from the subject's eye and the interference light with the reference light appearing in the interference unit not shown, and generates an interference signal. The image processing device 20 obtains the interference signal from the detector 15, and can generate a tomographic image of the subject's eye by performing Fourier transform on the interference signal, etc.

[0176] The internal fixation lamp 16 is provided with a display unit 161 and a lens 162. In this example, a plurality of light emitting diodes (LDs) arranged in a matrix are used as an example of the display unit 161. The light emitting position of the light emitting diode is changed according to the part to be imaged by the control of the image processing device 20. The light from the display unit 161 is guided to the eye to be inspected via the lens 162. The light emitted from the display unit 161 has a wavelength of, for example, 520 nm, and is displayed in a desired mode by the control of the image processing device 20.

[0177] Note that the OCT apparatus 10 may be provided with a driving control unit for the OCT apparatus 10 that controls driving of each component based on control of the image processing apparatus 20 .

[0178] Next, refer to FIG. 2A to FIG. 2C , the structure and image of the eye obtained by the image processing system 1 will be described. Figure 2A It is a diagram of an eyeball. Figure 2A There are shown the cornea C, the lens CL, the vitreous body V, the macula region M (the central part of the macula represents the fovea), and the optic nerve head D. In this example, a case will be described where the posterior pole of the retina including the vitreous body V, the macula region M, and the optic nerve head D is mainly imaged. Note that although not described below, the OCT apparatus 10 can also image the anterior segment of the eye such as the cornea and the lens.

[0179] Figure 2B An example of a tomographic image obtained by imaging the retina using the OCT device 10 is shown. Figure 2B In the figure, AS represents an image unit obtained by a single A scan. Here, A scan refers to obtaining tomographic information in the depth direction at a point of the eye to be inspected through the above series of operations of the OCT device 10. In addition, performing A scan multiple times in any lateral direction (main scanning direction) to obtain two-dimensional tomographic information in the lateral direction and depth direction of the eye to be inspected is called B scan. By collecting multiple A scan images obtained by A scan, a B scan image can be formed. In the following, the B scan image is referred to as a tomographic image.

[0180] Figure 2B The blood vessel Ve, the vitreous body V, the macular region M, and the optic nerve head D are presented. In addition, the boundary line L1 represents the boundary between the inner limiting membrane (ILM) and the nerve fiber layer (NFL), the boundary line L2 represents the boundary between the nerve fiber layer and the ganglion cell layer (GCL), and the boundary line L3 represents the photoreceptor inner segment-outer segment junction (ISOS). In addition, the boundary line L4 represents the retinal pigment epithelium (RPE), the boundary line L5 represents the Bruch's membrane (BM), and the boundary line L6 represents the choroid. In the tomographic image, it is assumed that the horizontal axis (the main scanning direction of OCT) is the x-axis, and the vertical axis (depth direction) is the z-axis.

[0181] Figure 2C An example of a fundus image obtained by imaging the fundus of the eye to be inspected using the fundus image capturing device 30 is shown. Figure 2C The macular area M and the optic nerve head D are presented, and the blood vessels of the retina are indicated by thick curves. In the fundus image, it is assumed that the horizontal axis (main scanning direction of OCT) is the x-axis, and the vertical axis (sub-scanning direction of OCT) is the y-axis.

[0182] Next, the image processing device 20 will be described. The image processing device 20 is provided with an obtaining unit 21, an image processing unit 22, a drive control unit 23, a memory 24, and a display control unit 25.

[0183] The acquisition unit 21 can obtain data of the interference signal of the eye to be inspected from the OCT device 10. Note that the data of the interference signal obtained by the acquisition unit 21 can be an analog signal or a digital signal. When the acquisition unit 21 obtains an analog signal, the analog signal can be converted into a digital signal by the image processing device 20. In addition, the acquisition unit 21 can obtain various images generated by the image processing unit 22, such as tomographic data, tomographic images, and En-Face images. Here, the tomographic data is data including information related to tomography of the object, and refers to data including interference signals based on OCT and data that has undergone fast Fourier transform (FFT: Fast Fourier Transform) or arbitrary signal processing.

[0184] In addition, the acquisition unit 21 acquires the imaging conditions of the tomographic image to be image-processed (for example, information related to the imaging date and time, the imaging part name, the imaging area, the imaging angle of view, the imaging system, the resolution and grayscale of the image, the pixel size of the image, the image filter, and the data format of the image). Note that the imaging conditions are not limited to the enumerated imaging conditions. In addition, the imaging conditions do not need to include all of the enumerated imaging conditions, and may include some of them.

[0185] In addition, the obtaining unit 21 can obtain data including fundus information obtained by the fundus image pickup device 30. Furthermore, the obtaining unit 21 can obtain information for identifying the subject's eye such as a subject identification number from the input unit 60, etc. The obtaining unit 21 can cause the memory 24 to store the obtained various data and images.

[0186] The image processing unit 22 can generate a tomographic image, an En-Face image, etc. from the data obtained by the obtaining unit 21 and the data stored in the memory 24, and can perform image processing on the generated or obtained images. Therefore, the image processing unit 22 can be used as an example of a generating unit that generates an En-Face image and a motion contrast frontal image described later. The image processing unit 22 is provided with a tomographic image generating unit 221 and a processing unit 222 (first processing unit).

[0187] The tomographic image generation unit 221 can generate tomographic data by performing processing such as Fourier transform on the interference signal obtained by the obtaining unit 21, and can generate a tomographic image based on the tomographic data. Note that any known method can be adopted as a tomographic image generation method, and detailed description will be omitted.

[0188] The processing unit 222 may include a learning model related to a machine learning model having a machine learning algorithm such as deep learning. The specific machine learning model will be described later. The processing unit 222 performs detection processing for detecting the retinal layer of the subject's eye in the tomographic image by using the learning model, and detects each retinal layer.

[0189] The drive control unit 23 can control the driving of each component of the OCT device 10 and the fundus image pickup device 30 connected to the image processing device 20. The memory 24 can store various images, data, etc., for example, tomographic data obtained by the acquisition unit 21, and tomographic images generated and processed by the image processing unit 22. In addition, the memory 24 can also store programs for realizing the functions of each component of the image processing device 20 when executed by the processor.

[0190] The display control unit 25 can control display of various information obtained by the obtaining unit 21 , the tomographic image generated and processed by the image processing unit 22 , information input by the operator, and the like in the display unit 50 .

[0191] Each component of the image processing device 20 except the memory 24 may be composed of a software module executed by a processor such as a CPU (Central Processing Unit) or an MPU (Micro Processing Unit). Note that the processor may be, for example, a GPU (Graphics Processing Unit) or an FPGA (Field Programmable Gate Array). In addition, each component may be composed of a circuit such as an ASIC that implements a specific function. The memory 24 may be composed of any storage medium such as an optical disk and a memory, for example.

[0192] Next, refer to Figure 3 , a series of processing according to this example will be described. Figure 3 301 is a flowchart of a series of processes according to this example. When a series of processes according to this example is started, the process proceeds to step S301.

[0193] In step S301, the obtaining unit 21 obtains a subject identification number, which is an example of information for identifying the subject's eye, from the outside of the image processing device 20, such as the input unit 60. The obtaining unit 21 obtains information related to the subject's eye held by the external storage device 40 based on the subject identification number, and stores the information in the memory 24.

[0194] In step S302, the drive control unit 23 performs imaging by controlling the OCT device 10 to scan the eye to be inspected, and the acquisition unit 21 obtains an interference signal including tomographic information of the eye to be inspected from the OCT device 10. The eye to be inspected is scanned by controlling the OCT device 10 by the drive control unit 23 according to a scan start instruction by the operator, and operating the light source 11, the galvanometer mirror 12, and the like.

[0195] The galvanometer mirror 12 includes an X scanner for the horizontal direction and a Y scanner for the vertical direction. Therefore, the drive control unit 23 can scan the measurement light in each of the horizontal direction (X) and the vertical direction (Y) in the device coordinate system by changing the orientations of each of these scanners. Note that the drive control unit 23 can scan the measurement light in a combined direction of the horizontal direction and the vertical direction by simultaneously changing the orientations of these scanners. Therefore, the drive control unit 23 can scan the measurement light in an arbitrary direction on the fundus plane.

[0196] When performing imaging, the drive control unit 23 adjusts various imaging parameters. Specifically, the drive control unit 23 sets at least the position of the pattern displayed by the internal fixation lamp 16, the scanning range and scanning pattern of the galvanometer mirror 12, the coherence gate position, and the focus.

[0197] The drive control unit 23 controls the light emitting diode of the display unit 161 to control the position of the pattern displayed by the internal fixation lamp 16, thereby imaging the center of the macular region and the optic nerve head of the eye to be inspected. In addition, the drive control unit 23 sets a scanning mode for imaging a three-dimensional volume as a scanning mode of the galvanometer mirror 12, such as a raster scan, a radial scan, and a cross scan. Note that no matter which scanning mode is selected, imaging is repeated multiple times on one line (the number of repetitions is two or more times). In this example, a case will be described where the scanning mode is a cross scan and imaging is repeated 150 times at the same position. After the adjustment of these imaging parameters is completed, according to the operator's imaging start instruction, the drive control unit 23 controls the OCT device 10 to image the eye to be inspected. Note that the number of repetitions according to this example is an example and can be set to any number of times according to the desired configuration.

[0198] Although detailed description is omitted in the present disclosure, the OCT apparatus 10 can track the subject's eye to capture images of the same position for addition averaging. Therefore, the OCT apparatus 10 can scan the subject's eye by reducing the influence of involuntary eye movement.

[0199] In step S303, the tomographic image generation unit 221 generates a tomographic image based on the interference signal obtained by the obtaining unit 21. The tomographic image generation unit 221 can generate a tomographic image by performing a general reconstruction process on each interference signal.

[0200] First, the tomographic image generation unit 221 performs fixed pattern noise removal from the interference signal. Fixed pattern noise removal is performed by averaging multiple signals obtained from the A scan to extract fixed pattern noise and subtracting the fixed pattern noise from the input interference signal. Then, the tomographic image generation unit 221 performs a desired window function processing to optimize the depth resolution and dynamic range, which will be in a trade-off relationship when the interference signal is Fourier transformed in a finite interval. The tomographic image generation unit 221 generates tomographic data by performing fast Fourier transform (FFT) processing on the interference signal that has been subjected to the window function processing.

[0201] The tomographic image generation unit 221 derives each pixel value of the tomographic image based on the generated tomographic data, and generates the tomographic image. Note that the generation method of the tomographic image is not limited to this, and can be performed by a known arbitrary method.

[0202] In step S304, the processing unit 222 of the image processing unit 22 performs detection processing on the retinal layer. Figure 4A and Figure 4B , the processing of the processing unit 222 will be described.

[0203] The processing unit 222 detects the boundary of the retinal layer in the plurality of tomographic images obtained by using the OCT apparatus 10. The processing unit 222 detects each retinal layer by using a learning model related to a machine learning model that has been machine-learned in advance.

[0204] Here, refer to FIG. 4A to FIG. 6 , a machine learning algorithm according to this example will be described. The training data for the machine learning model according to this example includes one or more input data and ground truth values ​​in pairs. Specifically, a tomographic image 401 obtained by OCT is listed as input data, and a boundary image 402 in which the boundaries of the retinal layers are specified for the tomographic image is listed as the ground truth value. In this example, an image showing a boundary 403 between the ILM and the NFL, a boundary 404 between the NFL and the GCL, an ISOS 405, an RPE 406, and a BM 407 is used as the boundary image 402. Note that although not shown, an image in which the boundary between the outer plexiform layer (OPL) and the outer nuclear layer (ONL), the boundary between the inner plexiform layer (IPL) and the inner nuclear layer (INL), the boundary between the INL and the OPL, the boundary between the GCL and the IPL, etc. are shown as other boundaries can be used.

[0205] Note that the boundary image 402 used as the ground truth may be an image showing a boundary in a tomographic image of a doctor or the like, or may be an image in which a boundary is detected by a rule-based boundary detection process. However, when machine learning is performed by using a boundary image for which boundary detection is not properly performed as the ground truth of training data, an image obtained by using a learning model learned using the training data may also be a boundary image for which boundary detection is not properly performed. Therefore, the possibility of generating an inappropriate boundary image by using the learning model can be reduced by removing a pair including such a boundary image from the training data. Here, rule-based processing refers to processing that utilizes known regularities, and rule-based boundary detection refers to boundary detection processing that utilizes known regularities such as the regularity of the retinal shape.

[0206] In addition, despite Figure 4A and Figure 4B An example of a specific XZ cross section in the XY surface of the retina is shown in , however, the cross section is not limited thereto. Although not shown, tomographic images and boundary images of multiple arbitrary XZ cross sections in the XY surface can be learned in advance so as to correspond to cross sections imaged by various different scanning modes such as raster scanning and radial scanning. For example, when using data such as tomographic images of the retina in three dimensions by raster scanning, volume data aligned between multiple adjacent tomographic images can be used for training data. In this case, a pair of images of arbitrary angles can be generated from one volume data (three-dimensional tomographic image) and one three-dimensional boundary data (three-dimensional boundary image) corresponding thereto. In addition, the machine learning model can be learned by using images actually imaged in various scanning modes as training data.

[0207] Next, the images during learning will be described. Images constituting a pair of tomographic images 401 and boundary images 402 are created using square area images having a constant image size corresponding to the positional relationship, and these images constitute training data for the machine learning model. FIG. 5A to FIG. 5C Describes the creation of an image.

[0208] First, a case where one of the pairs constituting the training data is a tomographic image 401 and a boundary image 402 will be described. In this case, Figure 5A As shown, a pair is formed by using a square area image 501 which is the entire tomographic image 401 as input data and a square area image 502 which is the entire boundary image 402 as ground truth. Note that although Figure 5A In the example shown, the pairs of input data and ground truth are composed of respective whole images, but the pairs are not limited to this.

[0209] For example, Figure 5BAs shown, the pair can be constructed by using a square area image 511 of the tomographic image 401 as input data and a square area image 513 as the corresponding imaging area in the boundary image 402 as the ground truth. The square areas of the square area images 511 and 513 are based on an A-scan unit. The A-scan unit can be one A-scan unit or several A-scan units.

[0210] Please note that although Figure 5B Based on the A-scan unit, it is possible to provide portions outside the square area above and below the image without setting the entire depth direction to the area. In other words, the size of the square area in the horizontal direction can be set to several A-scans, and the size of the square area in the depth direction can be set to be smaller than the size in the depth direction of the image.

[0211] In addition, if Figure 5C As shown, the pair can be constructed by using a square area image 521 of the tomographic image 401 as input data and a square area image 523 which is the corresponding imaging area in the boundary image 402 as the ground truth.

[0212] Note that during learning, the size of the square area can be made constant by standardizing the scan range (viewing angle) and scan density (number of A scans) to align the image size. FIG. 5A to FIG. 5C Each square region image shown in is an example of the square region size during learning.

[0213] exist Figure 5A In the example shown, the number of square regions can be set to one, and Figure 5B and Figure 5C In the example shown, it can be set to multiple. For example, in Figure 5B In the example shown in the example of , the pair can also be constructed by using the square area image 512 of the tomographic image 401 as input data and the square area image 514 as the corresponding imaging area in the boundary image 402 as the ground truth. Figure 5C In the example shown in the example of , the pair can also be constructed by using the square area image 522 of the tomographic image 401 as input data and the square area image 524 as the corresponding imaging area in the boundary image 402 as the ground truth. In this way, pairs of mutually different square area images can be created from a pair of a single tomographic image and a single boundary image. Note that the pairs constituting the training data can be enriched by creating a large number of pairs of square area images while changing the positions of the areas to different coordinates in the original tomographic image and the boundary image.

[0214] Despite Figure 5B and Figure 5C The square areas are discretely shown in the example shown, but, in reality, the original tomographic image and the boundary image can be divided into continuous square area images with a constant image size without gaps. In addition, the original tomographic image and the boundary image can be divided into square area images corresponding to each other at random positions. In this way, by selecting smaller area images as square areas (or strip areas) for input data and ground truth pairs, a large amount of pair data can be generated from the tomographic image 401 and the boundary image 402 that constitute the original pair. Therefore, the time spent on training the machine learning model can be reduced. On the other hand, in the learning model of the completed machine learning model, there is a tendency for the time to be spent on image segmentation processing to become longer. Here, image segmentation processing refers to the process of identifying and distinguishing regions and boundaries in an image.

[0215] Next, we will refer to Figure 6 A convolutional neural network (CNN) that performs image segmentation processing on an input tomographic image is described as an example of a machine learning model according to the present example. Figure 6 An example of a construction 601 of a machine learning model in the processing unit 222 is shown. Note that, for example, FCN (Fully Convolutional Network) or SegNet can also be used as a machine learning model according to this example. In addition, a machine learning model for object recognition per region can be used according to a desired construction. For example, RCNN (Regional CNN), fastRCNN, or fasterRCNN can be used as a machine learning model for object recognition. In addition, YOLO (Only Look Once) or SSD (Single Shot MultiBox Detector) can also be used as a machine learning model for object recognition per region.

[0216] Figure 6 The machine learning model shown in includes a plurality of layers responsible for processing and outputting the processing of input values. Note that as types of layers included in the structure 601 of the machine learning model, there are convolution layers, downsampling layers, upsampling layers, and merging layers.

[0217] A convolutional layer is a layer that performs convolution on the input values ​​according to parameters such as the kernel size of the filter, the number of filters, the value of the stride, and the dilation value. Note that the dimension of the kernel size of the filter can also change depending on the dimensionality of the input image.

[0218] The down-sampling layer is a layer that performs processing to make the number of output values ​​smaller than the number of input values ​​by thinning or combining input values. Specifically, for example, as such processing, there is a maximum pooling process.

[0219] The up-sampling layer is a layer that performs a process of making the number of output values ​​exceed the number of input values ​​by duplicating input values ​​or adding values ​​interpolated from input values. Specifically, for example, as such a process, there is a linear interpolation process.

[0220] The merging layer is a layer to which values ​​such as an output value of a certain layer and pixel values ​​constituting an image are input from a plurality of sources, and the values ​​are combined by concatenating or adding the values.

[0221] Note that by setting the kernel size of the filter to 3 pixels in width and 3 pixels in height, and setting the number of filters to 64, as included in Figure 6 The parameters of the convolution layer in the configuration 601 shown are set to achieve image segmentation processing with a certain accuracy. However, caution is required because when the parameters of the layers and nodes constituting the neural network are set differently, the degree of tendency of training from the training data that can be reproduced in the output data may be different. In other words, in many cases, since the appropriate parameters are different depending on the mode at the time of implementation, the parameters can be changed to preferred values ​​as needed.

[0222] In addition, CNN can obtain better characteristics not only by changing the parameters as described above, but also by changing the structure of CNN. For example, the better characteristics are high accuracy of image segmentation processing, short time of image segmentation processing, and short time spent on machine learning model training.

[0223] Note that the construction 601 of the CNN used in this example is a U-net type machine learning model, which includes the function of an encoder and the function of a decoder, the encoder including multiple layers including multiple downsampling layers, and the decoder including multiple layers including multiple upsampling layers. The U-net type machine learning model is constructed (for example, by using skip connections) so that geometric information (spatial information) that becomes blurred in the multiple layers constructed as the encoder can be used in the layers of the same dimension (layers corresponding to each other) in the multiple layers constructed as the decoder.

[0224] Although not shown, as an example of a structural change of a CNN, for example, a batch normalization layer and an activation layer using a normalized linear function (Rectifier Linear Unit) may be incorporated after the convolution layer.

[0225] When data is input into the learning model of such a machine learning model, data according to the design of the machine learning model is output. For example, output data with a high probability corresponding to the input data is output according to the trend trained by using the training data.

[0226] In the learning model of the processing unit 222 according to the present example, when the tomographic image 401 is input, the boundary image 402 is output according to the trend trained by using the training data. The processing unit 222 can detect the retinal layer and its boundary in the tomographic image 401 based on the boundary image 402.

[0227] Please note that if Figure 5B and Figure 5C As shown in , when the regions of the image are divided and learned, the processing unit 222 obtains a square region image, which is a boundary image corresponding to each square region, by using the learning model. Therefore, the processing unit 222 can detect the retinal layer in each square region. In this case, the processing unit 222 can generate a boundary image 402 corresponding to the input tomographic image 401 by arranging each square region image, which is a boundary image obtained by using the learning model, in a positional relationship similar to that of each square region image, and combining each square region image. Also in this case, the processing unit 222 can detect the retinal layer and its boundary in the tomographic image 401 based on the generated boundary image 402.

[0228] In step S304, when the processing unit 222 performs the detection processing of the retinal layer, the processing proceeds to step S305. In step S305, the display control unit 25 displays the boundary and the tomographic image detected by the processing unit 222 on the display unit 50. Here, Figure 7 An example of a screen displayed on the display unit 50 is shown.

[0229] exist Figure 7 , and the display screen 700 shows an SLO image 701 of the retina, a thickness map 702 superimposed on the SLO image 701, an En-Face image 703, a tomographic image 711, and a thickness graph 712. The boundaries 715 and 716 of the retina are superimposed on the tomographic image 711.

[0230] Note that in this example, the range of the retina is from the boundary L1 between the inner limiting membrane and the nerve fiber layer to the retinal pigment epithelium L4, and the boundaries 715 and 716 correspond to the boundary L1 and the retinal pigment epithelium L4, respectively. The range of the retina is not limited to this, and for example, it may be the range from the boundary L1 between the inner limiting membrane and the nerve fiber layer to the choroid L6, and in this case, the boundaries 715 and 716 may correspond to the boundary L1 and the choroid L6, respectively.

[0231] The retinal thickness graph 712 is a graph showing the thickness of the retina derived from the boundaries 715 and 716. In addition, the thickness graph 702 shows the thickness of the retina derived from the boundaries 715 and 716 in a color map. Figure 7 Although color information corresponding to the thickness map 702 is not shown for description, in practice, the thickness map 702 may display the thickness of the retina corresponding to each coordinate in the SLO image 701 according to the corresponding color map.

[0232] The En-Face image 703 is a frontal image generated by projecting data within a range between boundaries 715 and 716 in the XY direction. The frontal image is generated by projecting or integrating data corresponding to a depth range of at least a portion of volume data (three-dimensional tomographic image) obtained by using optical interference on a two-dimensional plane, the depth range being defined based on two reference planes. The En-Face image 703 according to the present example is a frontal image generated by projecting data corresponding to a depth range (depth range between boundaries 715 and 716) determined based on the retinal layer of the detected volume data on a two-dimensional plane. Note that as a technique for projecting data corresponding to a depth range defined based on two reference planes on a two-dimensional plane, for example, a technique for using a representative value of data within a depth range as a pixel value on a two-dimensional plane may be used. Here, the representative value may include a value such as an average value, a median value, or a maximum value of pixel values ​​within a range in the depth direction in an area surrounded by two reference planes.

[0233] In addition, the depth range of the En-Face image 703 displayed on the display screen 700 is not limited to the depth range between the boundaries 715 and 716. The depth range of the En-Face image 703 may be, for example, a range including a predetermined number of pixels in the deeper direction or the shallower direction based on one of the two layer boundaries 715 and 716 related to the detected retinal layer. In addition, the depth range of the En-Face image 703 may be, for example, a range changed (offset) from the range between the two layer boundaries 715 and 716 related to the detected retinal layer according to an instruction of the operator.

[0234] Note that the frontal image displayed on the display screen 700 is not limited to the En-Face image (luminance En-Face image) based on the luminance value. The frontal image displayed on the display screen 700 may be, for example, a motion contrast frontal image generated by projecting or integrating data corresponding to the above-mentioned depth range of motion contrast data between a plurality of volume data on a two-dimensional plane. Here, the motion contrast data is data indicating changes in a plurality of volume data obtained by controlling the measurement light to be scanned multiple times in the same area (same position) of the eye to be inspected. At this time, the volume data is composed of a plurality of tomographic images obtained at different positions. Then, by obtaining data indicating changes between a plurality of tomographic images obtained at substantially the same position in each of the different positions, the motion contrast data can be obtained as the volume data. Note that the motion contrast frontal image is also referred to as an OCTA frontal image (OCTA En-Face image) related to OCT angiography (OCTA) for measuring blood flow motion, and the motion contrast data is also referred to as OCTA data. The motion contrast data can be derived, for example, as a decorrelation value, a variance value, or a value obtained by dividing a maximum value by a minimum value (maximum value / minimum value) between interference signals corresponding to the two tomographic images, and can be derived by any known method. At this time, the two tomographic images can be obtained, for example, by controlling so that the measurement light is scanned multiple times in the same area (same position) of the eye to be inspected.

[0235] In addition, three-dimensional OCTA data (OCT volume data) used when generating an OCTA frontal image can be generated by using at least a portion of the interference signal common to the volume data including the tomographic image for detecting the retinal layer. In this case, the volume data (three-dimensional tomographic image) and the three-dimensional OCTA data can correspond to each other. Therefore, for example, by using the three-dimensional motion contrast data corresponding to the volume data, a motion contrast frontal image corresponding to the depth range determined based on the detected retinal layer can be generated.

[0236] Here, the display of the thickness map 702, the En-Face image 703, the thickness graph 712, and the boundaries 715 and 716 are examples of the contents that the image processing device 20 can generate based on the boundaries and retinal layers detected by the processing unit 222. Note that the generation method for generating these contents can adopt any known method.

[0237] Note that in addition to these, the display screen 700 of the display unit 50 may be provided with a patient tab, an imaging tab, a report tab, a setting tab, etc. In this case, Figure 7The contents displayed on the display screen 700 of the report will be displayed on the report tab. In addition, the patient information display part, the examination classification tab, the examination list, etc. can also be displayed on the display screen 700. Thumbnails of fundus images, tomographic images, and OCTA images can be displayed in the examination list.

[0238] Next, in step S306, the obtaining unit 21 obtains from the outside an instruction whether to end a series of processes related to the imaging of tomographic images by the image processing system 1. The operator can input the instruction using the input unit 60. When the obtaining unit 21 obtains an instruction to end the process, the image processing system 1 ends a series of processes according to the present example. On the other hand, when the obtaining unit 21 obtains an instruction not to end the process, the process returns to step S302, and imaging is continued.

[0239] As described above, the image processing device 20 according to the present example includes the acquisition unit 21 and the processing unit 222 (first processing unit). The acquisition unit 21 acquires a tomographic image of the eye to be inspected. The processing unit 222 performs a first detection process for detecting at least one retinal layer among a plurality of retinal layers of the eye to be inspected in the tomographic image by using a learning model.

[0240] When image segmentation processing is performed by using a learning model, boundary detection can also be appropriately performed based on, for example, a learning trend of layer structure changes caused by pathological changes in a diseased eye. Therefore, in the image processing device 20 according to this example, by performing image segmentation processing using a learning model, boundary detection can be performed regardless of the disease, part, etc., and the accuracy of boundary detection can be improved.

[0241] In addition, the image processing device 20 further includes an image processing unit 22 that generates a frontal image corresponding to a depth range of at least a portion of the three-dimensional tomographic image of the eye to be inspected, the depth range being determined based on the detected at least one retinal layer. The image processing unit 22 can generate a motion contrast frontal image corresponding to the determined depth range by using three-dimensional motion contrast data corresponding to the three-dimensional tomographic image.

[0242] Note that although a configuration has been described in this example in which the image segmentation process is performed by using one learning model, the image segmentation process may be performed by using a plurality of learning models.

[0243] As described above, since the learning model generates output data according to the learning trend using the training data as described above, the reproducibility of the learning trend with respect to the output data can be improved by learning using training data having similar characteristic trends. Therefore, for example, by performing image segmentation processing on the tomographic image of the corresponding imaging part using a plurality of learning models learned for each imaging part, a more accurate boundary image can be generated. In this case, the image processing system can detect the retinal layer more accurately. In addition, in this case, since the learning model can be additionally added, it is also possible to expect the upgraded version to gradually improve the performance.

[0244] In addition, the processing unit 222 can generate the final output of the processing unit 222 by using a plurality of learning models learned for each region in the tomographic image (e.g., a region near the vitreous body, a retinal region, and a region near the sclera) and combining the outputs of the respective learning models. In this case, since a more accurate boundary image can be generated for each region, the retinal layer can be detected more accurately.

[0245] In addition, in the present example, although a machine learning model that performs image segmentation processing has been described as a machine learning model, for example, a machine learning model that estimates an imaging site of a tomographic image may be used.

[0246] Generally, the construction of the machine learning model is not limited to the construction of outputting an image corresponding to an image as input data. For example, the machine learning model can be constructed so that for the input data, the kind of output data trained by using the training data can be output, or the probability can be output as a numerical value of each class. Therefore, the format and combination of the input data and the ground truth constituting the training data can be made suitable for the utilization form, such as one is an image and the other is a numerical value, one includes multiple images and the other is a string, or both are images.

[0247] As an example of training data for a machine learning model for estimating an imaging site, specifically, training data including pairs of tomographic images obtained by OCT and imaging site labels corresponding to the tomographic images can be listed. Here, the imaging site label is a unique numerical value or character string representing a site. When a tomographic image obtained by using OCT is input into a learning model trained by using such training data, the imaging site label of the site currently being imaged is output in the image, or, depending on the design, the probability of each imaging site label is output.

[0248] The processing unit 222 can estimate the imaging site of the tomographic image by further using the learning model that estimates such an imaging site, and perform image segmentation processing according to the estimated imaging site or the imaging site with the highest probability using the learning model. With such a configuration, even when the obtaining unit 21 cannot obtain imaging conditions for the imaging site of the tomographic image, the retinal layer can be detected more accurately by estimating the imaging site from the tomographic image and performing image segmentation processing corresponding to the imaging site.

[0249] (Example 2)

[0250] In Example 1, image segmentation processing for detecting all target retinal layers from a tomographic image is performed by using a learning model. In contrast, in Example 2, boundary detection according to rule-based image features is performed based on the detection result of the retinal area by the learning model.

[0251] Conventionally, although detection of the opening end of Bruch's membrane is usually performed when detecting Cup (optic disccupping) and Disc (optic nerve head) using OCT images in the optic nerve head, its detection is sometimes difficult in cases of peripapillary chorioretinal atrophy.

[0252] In addition, in conventional rule-based image segmentation processing, the robustness to individual differences and pathological changes of the subject's eye is low, and the retinal area may be erroneously detected initially. In this case, the subsequent inner retinal layer boundary detection cannot be performed appropriately.

[0253] In contrast, the accuracy of boundary detection can be improved by using a machine learning model for image segmentation processing. However, when recognizing the imaging site and detecting the boundary of the retinal layer by using a machine learning model with a machine learning algorithm (such as deep learning), it is generally difficult to collect the number of cases of normal images and lesion images with correct answers because this is the field of medical imaging. In addition, it takes time to create the correct answer data for learning.

[0254] Therefore, in this example, the retinal area is detected by using the learning model, and the boundary detection by the image feature is used together for the detected retinal area. Therefore, since the false detection of the retinal area is suppressed, the detection accuracy of the inner retinal layer boundary is improved, and in the process of machine learning, only the correct answer data for the retinal layer may be created during learning, and other data may be created otherwise, so learning can be performed efficiently.

[0255] In the following, reference is made to Figures 8 to 13D, an image processing system 8 according to the present example will be described. Hereinafter, with respect to the image processing of the image processing system according to the present example, differences from the image processing according to Example 1 will be mainly described. Note that the configuration and processing of the image processing system according to the present example that are similar to those of the image processing system 1 according to Example 1 will be shown by using the same reference numerals, and the description will be omitted.

[0256] Figure 8 An example of a schematic configuration of an image processing system 8 according to the present example is shown. In the image processing system 8, in the image processing unit 82 of the image processing apparatus 80, a first processing unit 822 and a second processing unit 823 are provided instead of the processing unit 222.

[0257] The first processing unit 822 includes a learning model for a machine learning model having a machine learning algorithm such as deep learning, and detects a retinal area in the tomographic image by using the learning model. The second processing unit 823 performs boundary detection of the retinal layer by determining a result of image feature extraction in a rule-based manner for the retinal area detected by the first processing unit 822.

[0258] Next, refer to Fig.9A and Fig. 9B , a series of processing according to this example will be described. Fig.9A is a flowchart of a series of processes according to this example, and Fig. 9B 304 is a flowchart of the boundary detection processing in this example. Note that since the processing other than the boundary detection processing is similar to that in Example 1, the description will be omitted. When the tomographic image is generated in step S303, the process proceeds to step S904.

[0259] When the boundary detection process in step S904 is started, the process proceeds to step S941. In step S941, the first processing unit 822 detects the retinal area in the tomographic image by using the learning model as the first boundary detection process.

[0260] Here, refer to FIG. 10A to FIG. 12 , a machine learning model according to this example will be described. The training data for the machine learning model according to this example includes one or more input data and ground truth values ​​in pairs. As an example of training data, training data, etc. can be listed, which includes pairs of Fig. 10A The tomographic image 1001 and Fig. 10B The label image 1002 shown in the figure is a label image 1002 in which a label is given to an arbitrary layer in the tomographic image 1001 .

[0261] Here, the label image is an image in which each pixel is labeled (an image obtained by annotation), and in this example, the label image is an image in which a label is assigned to each pixel, and the label is related to the image that appears (is being photographed) in the pixel. In the label image 1002, as label examples, a label 1003 of the side shallower than the retina (vitreous body side), a label 1004 of the inner retinal layer, and a label 1005 of the side deeper than the retina (choroid side) are given. The first processing unit 822 in this example detects the inner retinal layer based on such a label image. Please note that in this example, the range of the retina (the range of the inner retinal layer) is from the boundary L1 between the inner limiting membrane and the nerve fiber layer to the retinal pigment epithelium L4, but is not limited to this. For example, the range of the retina can be defined as the range from the boundary L1 between the inner limiting membrane and the nerve fiber layer to the photoreceptor inner segment-outer segment junction L3, the range from the boundary L1 between the inner limiting membrane and the nerve fiber layer to the Bruch's membrane L5, or the range from the boundary L1 between the inner limiting membrane and the nerve fiber layer to the choroid L6.

[0262] In addition, despite the Fig. 10A and Fig. 10B An example of an XZ cross section in the XY surface of the retina is shown in , however, the cross section is not limited thereto. Although not shown, a plurality of arbitrary XZ cross sections on the XY surface can be learned in advance, and the learning model can be made to process cross sections imaged by various different scanning modes such as raster scanning and radial scanning. For example, when data such as a tomographic image of the retina in three dimensions by raster scanning is used, volume data aligned between a plurality of adjacent tomographic images can be used for training data. In this case, a pair of images of arbitrary angles can be generated from a volume data and a three-dimensional label data (three-dimensional label image) corresponding thereto. In addition, the machine learning model can be learned by using images actually imaged in various scanning modes as training images.

[0263] Next, the images during learning will be described. Images constituting pairs of tomographic images 1001 and label images 1002 are created using square area images having a constant image size corresponding to the positional relationship, and these images constitute training data for the machine learning model. FIG. 11A to FIG. 11C Describes the creation of an image.

[0264] First, a case where one of the pairs constituting the training data is a tomographic image 1001 and a label image 1002 will be described. In this case, Fig.11A As shown, a pair is formed by using a square area image 1101 as the entire tomographic image 1001 as input data and a square area image 1102 as the entire label image 1002 as the ground truth. Note that although Fig.11AIn the example shown, the pairs of input data and ground truth are composed of respective whole images, but the pairs are not limited to this.

[0265] For example, Fig. 11B As shown, the pair can be constructed by using a square area image 1111 of the tomographic image 1001 as input data and a square area image 1113 of the corresponding imaging area in the label image 1002 as the ground truth. The square area images 1111 and 1113 are based on an A-scan unit. The A-scan unit can be one A-scan unit or several A-scan units.

[0266] Please note that although Fig. 11B Based on the A-scan unit, it is possible to provide portions outside the square area above and below the image without setting the entire depth direction to the area. In other words, the size of the square area in the horizontal direction can be set to several A-scans, and the size of the square area in the depth direction can be set to be smaller than the size in the depth direction of the image.

[0267] In addition, if Fig. 11C As shown, the pair can be constructed by using a square area image 1121 of the tomographic image 1001 as input data and a square area image 1123 of the corresponding imaging area in the label image 1002 as the ground truth. In this case, it is assumed that the size of the square area is a size that includes a plurality of labels in one square area.

[0268] Note that during learning, the size of the square area can be made constant by standardizing the scan range (viewing angle) and scan density (number of A scans) to align the image size. FIG. 11A to FIG. 11C Each square region image shown in is an example of the square region size during learning.

[0269] exist Fig.11A In the example shown, the number of square regions can be set to one, and Fig. 11B and Fig. 11C In the example shown, it can be set to multiple. For example, in Fig. 11B In the example shown in , the pair can also be constructed by using the square area image 1112 of the tomographic image 1001 as input data and the square area image 1114 of the corresponding imaging area in the label image 1002 as the ground truth. Fig. 11CIn the example shown in , the pair can also be constructed by using the square area image 1122 of the tomographic image 1001 as input data and the square area image 1124 of the corresponding imaging area in the label image 1002 as the ground truth. In this way, pairs of mutually different square area images can be created from a pair of a single tomographic image and a single label image. Note that the pairs constituting the training data can be enriched by creating a large number of pairs of square area images while changing the positions of the areas to different coordinates in the original tomographic image and the label image.

[0270] Despite Fig. 11B and Fig. 11C The square areas are discretely shown in the example shown, but, in reality, the original tomographic image and the label image can be divided into continuous square area images with a constant image size without gaps. In addition, the original tomographic image and the label image can be divided into square area images corresponding to each other at random positions. In this way, by selecting smaller area images as square areas (or strip areas) for input data and ground truth pairs, a large amount of pair data can be generated from the tomographic image 1001 and the label image 1002 that constitute the original pair. Therefore, the time spent on training the machine learning model can be reduced. On the other hand, in the learning model of the completed machine learning model, there is a tendency for the time to be spent on image segmentation processing to become longer.

[0271] Next, we will refer to Fig.12 The construction of a convolutional neural network (CNN) that performs image segmentation processing on an input tomographic image is described as an example of a machine learning model according to the present example. Fig.12 An example of the construction 1201 of the machine learning model in the first processing unit 822 is shown. Note that, for example, similar to Example 1, FCN or SegNet can also be used as the machine learning model according to this example. In addition, a machine learning model for object recognition per region as described in Example 1 can be used according to a desired construction.

[0272] and Figure 6 Similar to the example of the machine learning model according to Example 1 shown, Fig.12 The machine learning model shown includes a plurality of layers responsible for processing and outputting the processing of input values. As the types of layers included in the structure 1201 of the machine learning model, there are convolution layers, downsampling layers, upsampling layers, and merging layers. Note that since the structures of these layers and the variant examples of the structure of the CNN are similar to those in the machine learning model according to Example 1, the detailed description will be omitted. Note that similar to the structure 601 of the CNN described in Example 1, the structure 1201 of the CNN used in this example is a U-net type machine learning model.

[0273] In the learning model of the first processing unit 822 according to this example, when the tomographic image 1001 is input, the label image 1002 is output according to the trend trained by using the training data. The first processing unit 822 can detect the retinal area in the tomographic image 1001 based on the label image 1002.

[0274] Please note that if Fig. 11B and Fig. 11C As shown in , when the area of ​​the image is divided and learned, the first processing unit 822 obtains a square area image by using the learning model, which is a label image corresponding to each square area. Therefore, the first processing unit 822 can detect the retinal layer in each square area. In this case, the first processing unit 822 arranges each square area image as a label image obtained by using the learning model in a positional relationship similar to that of each square area image, and combines each square area image. Therefore, the first processing unit 822 can generate a label image 1002 corresponding to the input tomographic image 1001. In this case as well, the first processing unit 822 can detect the retinal area in the tomographic image 1001 based on the generated label image 1002.

[0275] In step S941, when the first processing unit 822 detects the retinal area, the process proceeds to step S942. In step S942, the second processing unit 823 detects the retinal area based on Fig. 10A The retinal area detected by the first processing unit 822 in the tomographic image 1001 shown is subjected to rule-based image segmentation processing to detect the remaining boundary in the inner retinal layer as the second detection processing.

[0276] Reference FIG. 13A to FIG. 13D , the second boundary detection processing performed by the second processing unit 823 will be described. Fig.13A A tomographic image 1001 is shown, which is an example of a tomographic image used as an input. Fig. 13B The label image 1002 is output by the first processing unit 822, and is an image to which the label 1004 of the retina area and other corresponding labels 1003, 1005 are assigned. The second processing unit 823 according to the present example uses the range of the retina area indicated by the label 1004 in the label image 1002 as the target area for layer detection.

[0277] The second processing unit 823 may detect the object boundary by detecting the contour in the retina area indicated by the label 1004 in the label image 1002 . Fig. 13C FIG. 1 shows an edge enhancement processed image 1303 that has been processed by the second processing unit 823. The processing by the second processing unit 823 will be described below. Fig. 13C and Fig.13D As shown, since the retinal layer is interrupted, the second processing unit 823 does not perform boundary detection on the optic nerve head.

[0278] The second processing unit 823 performs noise removal and edge enhancement processing on the region corresponding to the label 1004 in the tomographic image 1001 to be processed. The second processing unit 823 applies, for example, a median filter or a Gaussian filter as the noise removal processing. In addition, the second processing unit 823 applies a Sobel filter or a Hessian filter as the edge enhancement processing.

[0279] Here, the edge enhancement processing of a two-dimensional tomographic image using a two-dimensional Hessian filter will be described. The Hessian filter can emphasize the secondary local structure of the two-dimensional shadow distribution based on the relationship between the two eigenvalues ​​(λ1, λ2) of the Hessian matrix. Therefore, in this example, the two-dimensional line structure is emphasized by utilizing the relationship between the eigenvalues ​​and the eigenvectors (e1, e2) of the Hessian matrix. Since the line structure in the two-dimensional tomographic image of the eye to be examined corresponds to the structure of the retinal layer, the structure of the retinal layer can be emphasized by applying the Hessian filter.

[0280] Note that in order to detect retinal layers with different thicknesses, the resolution of smoothing by the Gaussian function when calculating the Hessian matrix can be changed. In addition, when a two-dimensional Hessian filter is applied, the two-dimensional Hessian filter can be applied after the data is transformed to adjust the XZ physical size of the image. In the case of ordinary OCT, the physical dimensions in the XY direction and the Z direction are different. Therefore, the filter is applied by adjusting the physical dimensions of the retinal layer for each pixel. Note that since the physical dimensions in the XY direction and the Z direction can be grasped from the design / construction of the OCT device 10, the data of the tomographic image can be transformed based on the physical dimensions. In addition, when the physical dimensions are not standardized, they can also be roughly corresponded by changing the smoothing resolution using a Gaussian function.

[0281] In the above, although the processing using two-dimensional tomographic images has been described, the object to which the Hessian filter is applied is not limited to this. When the data structure when the tomographic image is photographed is a three-dimensional tomographic image by raster scanning, a three-dimensional Hessian filter can also be applied. In this case, after the image processing unit 82 performs alignment processing in the XZ direction between adjacent tomographic images, the second processing unit 823 can emphasize the secondary local structure of the three-dimensional shadow distribution based on the relationship between the three eigenvalues ​​(λ1, λ2, λ3) of the Hessian matrix. Therefore, by emphasizing the three-dimensional layer structure by utilizing the relationship between the eigenvalues ​​of the Hessian matrix and the eigenvectors (e1, e2, e3), the edge can also be emphasized three-dimensionally.

[0282] In the edge enhancement processing image 1303, the portion where the edge is emphasized is displayed as a white line 1304. Note that the region in the tomographic image 1001 that does not correspond to the label 1004 can be regarded as a region where the edge is not detected. In addition, although the configuration in which the edge enhancement processing is performed by using the Hessian filter has been described here, the processing method of the edge enhancement processing is not limited thereto and can be performed by an existing arbitrary method.

[0283] Fig.13D A boundary image 1305 is shown, which shows the boundary of the retinal layer detected by the second processing unit 823 by using the label image 1002 and the edge enhancement processed image 1303. In the boundary image 1305, a black line 1306 indicates an example of the boundary. The process of detecting the boundary of the retinal layer from the label image 1002 and the edge enhancement processed image 1303 by the second processing unit 823 will be described below.

[0284] The second processing unit 823 detects edge-emphasized boundaries from the edge-enhancement processed image 1303. In this example, since the first processing unit 822 has already detected the boundary between the ILM and the NFL and the RPE, the second processing unit 823 then detects the ISOS and the boundary between the NFL and the GCL. Note that, although not shown, the boundary between the outer plexiform layer (OPL) and the outer nuclear layer (ONL), the boundary between the inner plexiform layer (IPL) and the inner nuclear layer (INL), the boundary between the INL and the OPL, and the boundary between the GCL and the IPL, etc., may be detected as other boundaries.

[0285] As a method for detecting the boundary, in each A scan, multiple positions with strong edge brightness are detected as boundary candidates, and based on the continuity of the boundary candidates in adjacent A scans, the points (positions with strong edge brightness) are connected as lines. In addition, when the points are connected as lines, the second processing unit 823 can remove outliers by evaluating the smoothness of the lines. More specifically, for example, the positions of the connection points in the Z direction are compared, and when the difference between the positions in the Z direction is greater than a predetermined threshold, the newly connected points can be determined as outliers, and the outliers can be excluded from the connection process. In addition, when the outliers are removed, the boundary candidates in the A scans adjacent to the A scan positions of the excluded points can be connected as lines. Please note that the method for removing outliers is not limited to this, and can be performed by any existing method.

[0286] The second processing unit 823 determines the corresponding boundary for each line formed by connecting the points based on the vertical distance and positional relationship of the boundary of the retinal layer in the Z direction. Note that when the boundary is not detected as a result of removing outliers in each A scan, the boundary can be derived by interpolating from the peripheral boundary. In addition, the boundary candidate can be searched from the peripheral boundary in the horizontal direction (X or Y direction) by relying on the edge, and the boundary can be determined again based on the boundary candidate searched from the peripheral boundary.

[0287] Then, the second processing unit 823 performs processing to correct the shape of the boundary to be smooth on the detected boundary. For example, the shape of the boundary can be smoothed by using image features and shape features together with a dynamic contour model (such as Snakes and Level Set method). In addition, considering that the coordinate value of the boundary shape is time series data with a signal, the shape can be smoothed by smoothing processing such as Savitzky-Golay filter, simple moving average, weighted moving average and exponential moving average.

[0288] Through this processing, the second processing unit 823 can detect the retinal layer in the retinal area detected by the first processing unit 822. Note that the above-mentioned detection processing of the retinal layer by the second processing unit 823 is an example, and the retinal layer can also be detected by using an existing arbitrary segmentation processing. When the second processing unit 823 detects the retinal layer, the process proceeds to step S305. Since the subsequent processing is similar to that of Example 1, its description will be omitted.

[0289] As described above, the image processing device 80 according to the present example includes the obtaining unit 21, the first processing unit 822, and the second processing unit 823. The obtaining unit 21 obtains a tomographic image of the eye to be inspected. The first processing unit 822 performs a first detection process for detecting at least one of the multiple retinal layers of the eye to be inspected in the tomographic image by using a learning model. The second processing unit 823 performs a second detection process for detecting at least one of the multiple retinal layers in the tomographic image without using a learning model.

[0290] Specifically, the second processing unit 823 detects at least one retinal layer other than at least one retinal layer detected by the first detection processing through the second detection processing. In particular, in this example, the first detection processing is a processing for detecting a layer from the boundary between the inner limiting membrane and the nerve fiber layer of the eye to the photoreceptor inner segment-outer segment junction, the retinal pigment epithelium, and one of the Bruch's membrane. In addition, the second detection processing is a processing performed after the first detection processing, which detects at least one retinal layer included in the layers detected by the first detection processing, that is, at least one retinal layer between the detected layers.

[0291] In the image processing device 80 according to this example, boundary detection can be performed regardless of a disease, a part, etc. In addition, by using boundary detection of image features together with the region output by the machine learning model, the accuracy of boundary detection can be improved. In addition, in the process of machine learning, since only correct answer data for the retinal layer may be created during learning, other data may be created otherwise, learning can also be performed efficiently.

[0292] In addition, as the number of boundaries detected by the machine learning model increases, the possibility of erroneous detection in the output label image and the boundary image increases. On the other hand, in the detection of the retinal area using the machine learning model according to the present example, since there are few boundaries to be detected, erroneous detection in the output label image and the boundary image can be suppressed.

[0293] Note that also in this example, similar to Example 1, the first processing unit 822 can be configured to detect the retinal area by using multiple machine learning models. In this case, the detection accuracy of the retinal area can be improved. In addition, since additional learning models can be added, it is also possible to expect the upgraded version to gradually improve the performance.

[0294] (Variation of Example 2)

[0295] In the above-mentioned Example 2, such an example has been shown that before the second processing unit 823 detects the boundary of the retina based on the rule base, the first processing unit 822 performs the detection of the retinal area by using the learning model as an initial stage. However, the order of the processing of the first processing unit 822 and the processing of the second processing unit 823 is not limited to this. For example, when the detection of the retinal area by the first processing unit 822 is very time-consuming, the second processing unit 823 may first perform the detection processing of the retinal area based on the rule base.

[0296] In this case, the second processing unit 823 first detects the boundary between the ILM and the NFL, RPE, or ISOS by using a method similar to the method of the second processing unit 823 in Example 2. This is because these boundaries are located at positions with high luminance values ​​and are boundaries located in the superficial part and deep part of the retina. When detecting the boundary between the ILM and the NFL, RPE, or ISOS, since the features are more easily visible than other boundaries, these boundaries can be detected based on an image with a lot of blur that has been subjected to several noise processing. In this case, since only general features can be detected, erroneous detection of other boundaries can be prevented. In addition, the retinal area can be limited by performing binarization processing on the tomographic image with a dynamic threshold, and the boundary between the ILM and the NFL, RPE, or ISOS can be detected therefrom. Note that the second processing unit 823 can detect the boundaries between the ILM and the NFL and the BM.

[0297] However, when the retinal area as the first detection target is detected by such a rule base as described above, the robustness to individual differences and pathological changes of the subject's eye is low, and the retinal area may be detected erroneously at first. In this case, the subsequent detection of the inner retinal layer boundary may not be performed appropriately.

[0298] As a countermeasure, in the present modification, the image processing unit 82 compares parameters for checking for erroneous detection of the retinal area (e.g., discontinuity of the retinal area boundary and distribution of boundary coordinates in the local curvature or the local area) with predetermined threshold values. When these parameters exceed the predetermined threshold values, the image processing unit 82 determines that the detection of the retinal area by the second processing unit 823 is an erroneous detection. Then, the first processing unit 822 is configured to perform detection of the retinal area when the detection of the retinal area by the second processing unit 823 is determined to be an erroneous detection by the image processing unit 82.

[0299] According to this variation, even when detection of the retinal area by the first processing unit 822 is very time-consuming, robustness to individual differences and pathological changes in the eye being examined can be ensured while reducing a large amount of processing waiting time for an inspector (operator) who performs a large number of inspections.

[0300] In addition, in the present modification, although a configuration has been described in which the processing is performed by the second processing unit 823 before the processing is performed by the first processing unit 822, these processes may be started at the same time. In this case, when the detection of the retinal area by the second processing unit 823 is determined to be an erroneous detection by the image processing unit 82, the second processing unit 823 performs the boundary detection of the inner retinal layer after waiting for the detection of the retinal area by the first processing unit 822. Note that when the detection of the retinal area by the second processing unit 823 is appropriately performed, the processing of the first processing unit 822 may be interrupted, or the processing result of the first processing unit 822 may be discarded.

[0301] In addition, when the first processing unit 822 and the second processing unit 823 detect the retinal area (the same retinal layer), the display control unit 25 may cause the display unit 50 to display the processing results of the detection processing performed by the first processing unit 822 and the second processing unit 823. In this case, according to the operator's instruction on the processing results displayed on the display unit 50, the second processing unit 823 may perform boundary detection of the inner retinal layer on any one of the processing results of the detection processing performed by the first processing unit 822 and the second processing unit 823. In this case, the detection processing of the retinal area by the second processing unit 823 may be defined as the second detection processing, and the subsequent detection processing of the boundary of the inner retinal layer by the second processing unit 823 may be defined as the third detection processing.

[0302] (Example 3)

[0303] In Example 2, an example has been shown in which a retinal area is detected by using a learning model, and a boundary of an inner retinal layer is detected for the detected retinal area. On the other hand, in this example, as an area detected by using a learning model, not only the retinal area but also a characteristic area of ​​the imaging part of an image as input data is detected.

[0304] Hereinafter, with respect to the image processing of the image processing system according to the present example, the difference from the image processing according to Example 2 will be mainly described. Note that since the configuration and processing process of the image processing system according to the present example are similar to the configuration and processing process of the image processing system 8 according to Example 2, the configuration and processing process of the image processing system according to the present example will be shown by using the same reference numerals, and the description will be omitted.

[0305] Reference FIG. 14A to FIG. 14D , the area detected by the first processing unit 822 according to the present example by using the learning model will be described. FIG. 14A to FIG. 14D An example of an image captured for each part of the eye to be inspected and a label image of a processing result processed by the first processing unit 822 is shown.

[0306] Fig.14A A tomographic image 1401 in the case of imaging the macular region and a label image 1402 in the macular region obtained by using a learning model are shown. The label image 1402 shows a vitreous body label 1403, a label 1404 ranging from the ILM to the ISOS, a label 1405 ranging from the ISOS to the RPE, a choroid label 1406, and a sclera label 1407.

[0307] As for the macular area, in order to identify, for example, the thickness of the entire retina caused by hemorrhage, neovascularization, etc., photoreceptor defects related to vision, or choroidal thinning caused by pathological myopia, labels are set for each area that is prone to morphological changes, and learning is performed in advance through a machine learning model. For the training data, it is assumed that a tomographic image of the macular area is input data, and a label image is a ground truth, in which, for example, a vitreous label, a label ranging from ILM to ISOS, a label ranging from ISOS to RPE, a choroid label, and a sclera label are given. Therefore, the first processing unit 822 can obtain a label image showing the labels of each area that is prone to the above-mentioned morphological changes by inputting the tomographic image of the macular area into the learning model, and can detect the area in units of these labels.

[0308] Fig. 14B A tomographic image 1411 in the case of photographing the optic nerve head and a label image 1412 in the optic nerve head obtained by using a learning model are shown. A vitreous label 1413, a label 1414 of the range from the ILM to the boundary between the NFL and the GCL, and a label 1415 of the range from the boundary between the NFL and the GCL to the boundary between the GCL and the IPL are shown in the label image 1412. In addition, a label 1416 of the range from the boundary between the GCL and the IPL to the ISOS, and a label 1417 of the range from the ISOS to the RPE are shown in the label image 1412. In addition, a label 1418 of the range deeper than the RPE and a label 1419 of the range of the lamina cribrosa are shown in the label image 1412. As for the training data in this case, it is assumed that a tomographic image of the optic nerve head is the input data and a label image is the ground truth, in which, for example, a label of the vitreous body, a label of the range from the ILM to the boundary between the NFL and GCL, a label of the range from the boundary between the NFL and GCL to the boundary between the GCL and IPL, a label of the range from the boundary between the GCL and IPL to the ISOS, a label of the range from the ISOS to the RPE, a label of the range deeper than the RPE, and a label of the range of the lamina cribrosa are given.

[0309] Fig. 14CThe front image 1421 is an image viewed from the front direction on the XY surface, and is an image captured by using the fundus image capturing device 30 when the optic nerve head is captured. Fig. 14C The label image 1422 is a label image obtained by using the learning model for the frontal image 1421 of the optic nerve head. In the label image 1422, a label 1423 for the periphery of the optic nerve head, a Disc label 1424, and a Cup label 1425 are shown.

[0310] In the optic nerve head, due to glaucoma, the disappearance of ganglion cells, morphological changes at the RPE (RPE-tip) end or the Bruch's membrane opening (BMO) end, cribriform plate, Cup and Disc, etc. are easy to appear. Therefore, a label is set for each of these areas, and learning is performed in advance through a machine learning model. For the training data, it is assumed that the front image of the optic nerve head is the input data, and the label image is the ground truth, in which, for example, labels around the optic nerve head, Disc labels and Cup labels are given. Therefore, the first processing unit 822 can obtain a label image showing the labels of each area where the above-mentioned morphological changes are prone to occur by inputting the image of the optic nerve head into the learning model, and can detect the area in units of these labels.

[0311] Fig.14D A tomographic image 1431 in the case of photographing the anterior segment of the eye, and a label image 1432 in the photographing of the anterior segment of the eye obtained by using a learning model are shown. A corneal label 1433, an anterior chamber label 1434, an iris label 1435, and a lens label 1436 are shown in the label image 1432. As for the tomographic image 1431 of the anterior segment of the eye, unlike the image of the posterior segment of the eye, the main areas as described above are pre-learned by the machine learning model. As for the training data, it is assumed that the tomographic image of the anterior segment of the eye is input data, and the label image is the ground truth, in which, for example, a corneal label, anterior chamber label, iris label, and lens label are given. Therefore, by inputting the image of the anterior segment of the eye into the learning model, the first processing unit 822 can obtain a label image in which labels of each area where the above-mentioned morphological changes are prone to occur are shown, and can detect areas in units of these labels.

[0312] According to this example, the second processing unit 823 is based on the first processing unit 822 Fig.14A , Fig. 14B and Fig.14D The remaining boundary is detected by using the region detected in the tomographic images 1401, 1411, and 1431 shown in the figure. In addition, the second processing unit 823 may measure the thickness of the detected boundary, the layer region sandwiched by the detected boundary, or the region detected by the first processing unit 822.

[0313] In addition, the second processing unit 823 can Fig. 14C Each region classified by a label on the front image of is measured, and the height, width, area, and Cup / Disc ratio of each region can be calculated. Note that any existing method can be used for these measurements and the calculation of the ratio.

[0314] In this way, the second processing unit 823 according to this example can perform an image processing algorithm corresponding to the area detected by the first processing unit 822, and can change the rules when applying the image processing algorithm to each area. The rules here include, for example, the type of boundary to be detected. For example, similar to Example 1, the second processing unit 823 can Fig.14A The tomographic image 1401 of the macular region is shown to be subjected to additional boundary detection within the range of ILM to ISOS shown with a label 1404. Note that the boundary detection method may be similar to the boundary detection method in Example 1. In addition, the second processing unit 823 may apply an image processing algorithm and rule to detect the boundary between the ILM and the NFL, the boundary between the OPL and the ONL, the boundary between the IPL and the INL, the boundary between the INL and the OPL, and the boundary between the GCL and the IPL for the tomographic image 1401 of the macular region. In this way, in the second processing unit 823, arbitrary image processing algorithms and rules may be set for each region according to a desired configuration.

[0315] As described above, in the image processing device 80 according to the present example, the first processing unit 822 detects a predetermined boundary for each imaging part for the input image. Therefore, in the present example, since the area detected by using the learning model is not limited to the retinal area, but is a characteristic area related to the imaging part, variations such as diseases can be resolved. Note that when the input image is a tomographic image of the retina, the first processing unit 822 can detect the predetermined boundary of each imaging part as a first detection process for detecting at least one retinal layer. In addition, when the input image is an image other than a tomographic image of the retina, the first processing unit 822 can detect the predetermined boundary of each imaging part as a process different from the first detection process. Note that, similar to Example 1, the respective outputs of a plurality of learning models that have been learned for each area of ​​the tomographic image can be combined to generate the final output of the first processing unit 822.

[0316] In addition, in the image processing device 80 according to the present example, predetermined shape features such as the height (thickness), width, area, and Cup / Disc ratio of each region related to the eye to be inspected can be measured based on the result of the first detection process or the second detection process.

[0317] (Example 4)

[0318] In Example 3, an example has been shown in which the area detected by using the learning model is not limited to the retinal area, but a characteristic area in the imaging site is detected. On the other hand, in Example 4, the performance of the processing using the learning model is selected according to the imaging conditions under which the image is captured, and the area detected by using the learning model is further narrowed down.

[0319] In the following, reference is made to Figures 15 to 16B , with respect to the image processing of the image processing system 150 according to the present example, the difference from the image processing according to Example 2 will be mainly described. Note that the configuration and processing of the image processing system according to the present example that are similar to the configuration and processing of the image processing system 8 according to Example 2 will be shown by using the same reference numerals, and the description will be omitted.

[0320] Fig.15 An example of a schematic configuration of an image processing system 150 according to the present example is shown. The image processing system 150 according to the present example is provided with a selection unit 1524 in addition to the tomographic image generation unit 221, the first processing unit 822, and the second processing unit 823 in the image processing unit 1520 of the image processing apparatus 152.

[0321] The selection unit 1524 selects image processing to be performed on the tomographic image based on the imaging conditions obtained by the acquisition unit 21 and the learning content (training data) related to the learning model of the first processing unit 822. Specifically, based on the imaging conditions and the learning of the learning model, it is selected whether to detect the retinal layer only by the first processing unit 822, or to detect the retinal area by the first processing unit 822 and the retinal layer by the second processing unit 823, or to detect the retinal layer only by the second processing unit 823. In addition, when the first processing unit 822 includes a plurality of learning models, the selection unit 1524 can select the learning model used for the detection processing of the first processing unit 822 based on the imaging conditions and the learning content related to the learning model of the first processing unit 822.

[0322] Next, refer to Fig.16A and Fig. 16B , a series of processing according to this example will be described. Fig.16A is a flowchart of a series of processes according to this example, and Fig. 16B 1604. It is a flowchart of the boundary detection processing according to this example. Note that since the processing other than the boundary detection processing is similar to the processing in Example 2, the description will be omitted. When the tomographic image is generated in step S303, the processing proceeds to step S1604. When the processing proceeds to step S1604, the boundary detection processing starts, and the processing proceeds to step S1641.

[0323] In step S1641, the acquisition unit 21 acquires the imaging conditions of the generated tomographic image, and the image processing unit 1520 acquires, from the acquired imaging conditions, information required for selecting the processing of the first processing unit 822 and the second processing unit 823. For example, these conditions may include an imaging part, an imaging system, an imaging area, an imaging angle of view, and an image resolution.

[0324] In step S1642, the selection unit 1524 selects whether to perform the processing of the first processing unit 822 based on the imaging condition obtained in step S1641. Here, as an example, the following case is considered: for the learning model of the first processing unit 822, only two models are prepared, one for the optic nerve head and one for the macula area, and the two models have been learned by using the training data of each imaging site for the optic nerve head and the macula area. In addition, in this case, the first processing unit 822 will be described as being unable to correspond to a wide-angle image (an image of a range in which both the optic nerve head and the macula area are imaged).

[0325] In this example, when the selection unit 1524 determines that the input image is an image in which the optic nerve head or the macula region is imaged alone based on information such as the imaging part name and the imaging angle of view regarding the imaging conditions, the selection unit 1524 selects processing by the first processing unit 822. Therefore, the process proceeds to step S1643. On the other hand, when the selection unit 1524 determines that the input image is an image obtained by imaging other than the optic nerve head and the macula region, or a wide-angle image including both the optic nerve head and the macula region, the selection unit 1524 selects not processing by the first processing unit 822. Therefore, the process proceeds to step S1645.

[0326] In step S1643, the selection unit 1524 selects an appropriate learning model used by the first processing unit 822 based on the imaging condition obtained in step S1641. In the above example, when the selection unit 1524 determines that the input image is an image obtained by imaging the optic nerve head based on, for example, the imaging part name and the information about the imaging angle of view, the selection unit 1524 selects the model for the optic nerve head. Similarly, when the selection unit 1524 determines that the input image is an image obtained by imaging the macula region, the selection unit 1524 selects the model for the macula region.

[0327] Note that here, although an example has been shown in which the learning model has learned only the image obtained by photographing the optic nerve head and the macula region, the learning content of the learning model is not limited to this. For example, a learning model that has been learned for other parts, or a learning model that has been learned by using a wide-angle image including the optic nerve head and the macula region may be used.

[0328] In addition, when learning models are prepared separately according to the imaging system instead of the imaging part, selection of processing and selection of learning model can be performed according to the imaging system. As an example of the imaging system, there are imaging systems of SD-OCT and SS-OCT, and due to the difference between these two imaging systems, image quality, imaging range, invasion depth in the depth direction, etc. are different. Therefore, selection of appropriate processing and selection of learning model can be performed for images of these different imaging systems. Note that when learning is performed together regardless of the imaging system at the time of learning, the processing may not be changed according to the imaging system. In addition, when there is only one learning model, since selection of the learning model in step S1643 is not necessary, the processing can be skipped.

[0329] In step S1644, the first processing unit 822 performs a first boundary detection process by using the learning model selected in step S1643. Note that the processes described in Examples 1 to 3 may be used for this process. For example, when the model for the macula region has been learned for the image segmentation process of each retinal layer in the macula region, similarly to Example 1, as the first detection process, the first processing unit 822 may detect all boundaries to be detected. In addition, for example, when the model for the optic nerve head has been learned for the process of detecting the retinal region in the optic nerve head, similarly to Example 2, as the first detection process, the first processing unit 822 may detect the retinal region. Similarly, when the model for the macula region has been learned for the process of detecting the characteristic region of the macula region as in Example 3, as the first detection process, the first processing unit 822 may detect the characteristic region as in Example 3. Note that since the specific detection method is similar to the detection method in Examples 1 to 3, the description will be omitted.

[0330] In step S1645, the selection unit 1524 selects whether to perform processing by the second processing unit 823 based on the imaging condition obtained in step S1641. When the selection unit 1524 selects to perform processing by the second processing unit 823, the process proceeds to step S1646. On the other hand, when the selection unit 1524 selects not to perform processing by the second processing unit 823, the process in step S1604 ends, and the process proceeds to step S305.

[0331] Here, an example of selection processing performed by the selection unit 1524 in step S1645 will be described. The case of processing by the second processing unit 823 is, for example, the case described in Examples 2 and 3 where the second processing unit 823 detects a boundary based on the area detected by the first processing unit 822.

[0332] In addition, when an image that is not learned by the first processing unit 822 is input, it is also processed by the second processing unit 823. In this case, in step S1642, since the processing by the first processing unit 822 is skipped, the second processing unit 823 performs boundary detection by rule-based image segmentation processing without using a learning model.

[0333] On the other hand, the case where the processing of the second processing unit 823 is not performed is, for example, a case where the first processing unit 822 can detect all target boundaries by using a learning model. In this case, since the processing is completed only by the first processing unit 822, the processing performed by the second processing unit 823 can be skipped.

[0334] However, even when the first processing unit 822 can detect all target boundaries by using the learning model, when the measurement of layer thickness, etc. is performed based on the detected boundaries by the second processing unit 823, it is possible to select the second processing unit 823 to perform the processing. On the other hand, the measurement of layer thickness, etc. based on the detected boundaries may not be limited to being performed by the second processing unit 823, and may be performed in the image processing unit 1520. Therefore, even when the measurement of layer thickness, etc. is performed, it is possible not to select the second processing unit 823 to perform the processing.

[0335] In step S1646, the selection unit 1524 selects image processing required for the processing of the second processing unit 823 and a rule when applying the image processing based on the imaging conditions obtained in step S1641. Fig.14A , Fig. 14B and Fig.14D When the tomographic image shown in FIG. 1 is viewed, the selection unit 1524 selects a process and a rule for detecting a remaining boundary based on the area detected by the first processing unit 822 .

[0336] Specifically, for example, when the input image is a tomographic image 1401 obtained by imaging the macula region shown in FIG. 14 , the selection unit 1524 selects image processing and rules that can correctly perform layer identification of the macula region. Fig. 14B In the case of the tomographic image 1411 obtained by photographing the optic nerve head shown in , the selection unit 1524 selects image processing and rules for abnormal processing of the optic nerve head in consideration of the influence of the Bruch's membrane opening (BMO), the lamina cribrosa, the Cup, the Disc, etc. For example, as for the tomographic image 1431, in the case of Fig.14D When the anterior segment of the eye is imaged as shown, the selection unit 1524 selects image processing and rules that can further perform layer identification on the cornea portion.

[0337] Furthermore, in step S1647, when measuring the thickness of the detected boundary or the layer region sandwiched by the boundary, the selection unit 1524 may select image processing required for such an image measurement function in addition to detection of the layer boundary or detection alone.

[0338] In step S1647, the second processing unit 823 performs detection of boundaries and / or measurement of the detected boundaries and regions. Note that similar to what is described in Examples 2 and 3, descriptions of the process of detecting boundaries in the region based on the region detected by the first processing unit 822 and the process of measuring boundaries, etc., will be omitted.

[0339] Here, an example of processing performed by the second processing unit 823 when an image that has not been learned by the learning model of the first processing unit 822 is input will be described. In this case, since there are no candidates for the retinal area detected from the input image, the second processing unit 823 first detects the boundary between the ILM and the NFL, RPE, or ISOS as described in the variation of Example 2.

[0340] After detecting the boundary between the ILM and the NFL, RPE, or ISOS, the second processing unit 823 detects the remaining boundary based on the area between these boundaries. Since the detection processing is similar to the detection processing in Examples 2 and 3, the description will be omitted. When the second processing unit 823 performs these processes, the process enters step S305. Note that similar to the variation of Example 2, the second processing unit 823 may first detect the boundary between the ILM and the NFL and the BM. In addition, since the subsequent processing is similar to the processing in Examples 1 to 3, the description will be omitted.

[0341] As described above, in the image processing device 152 according to the present example, the obtaining unit 21 obtains the imaging conditions related to the tomographic image of the subject's eye. The image processing device 152 further includes a selecting unit 1524 that selects a process based on the imaging conditions. The selecting unit 1524 selects at least one of the first detection process and the second detection process based on the imaging conditions.

[0342] Therefore, in the image processing device 152 according to the present example, it is determined whether boundary detection can be performed by the first processing unit 822 through the learning model, and whether it is necessary to perform the processing of boundary detection by the second processing unit 823 through image features, based on the imaging conditions. Therefore, even when the boundary detection performed by the learning model corresponds only to a specific image, the processing can be appropriately performed according to the input image. Therefore, even when the learning model does not correspond to various image patterns, the boundary detection processing can be reliably performed. Therefore, by using at least one machine learning model in each maturity stage created by machine learning together with the following image processing method, the accuracy of boundary detection can be improved, which determines the image feature extraction result in a rule-based manner to perform boundary detection of the retinal layer.

[0343] In addition, the first processing unit 822 includes multiple learning models on which machine learning has been performed using different training data. In addition, the first processing unit 822 performs the first detection processing by using a learning model among the multiple learning models on which machine learning has been performed using training data corresponding to the imaging conditions.

[0344] According to this example, the retinal layer can be detected by using an appropriate learning model based on the imaging conditions, and more appropriate processing can be performed according to the input image. In addition, since the learning model can be additionally added, it is also possible to expect an upgraded version to gradually improve performance. In addition, according to the selection unit 1524, the image processing and selection of the rule used in the second processing unit 823 can be performed based on the imaging conditions, and more appropriate processing can be performed according to the input image.

[0345] Note that in this example, although the processing performed by the first processing unit 822 and the second processing unit 823 are selected respectively in step S1642 and step S1645, the process of selecting the processing is not limited thereto. For example, the selection unit 1524 may be configured to select the processing performed by only the first processing unit 822, the processing performed by only the second processing unit 823, or the processing performed by both the first processing unit 822 and the second processing unit 823 in one step.

[0346] (Variation of Example 4)

[0347] In Example 4, it has been shown that appropriate processing can be performed in various cases, for example, when the first processing unit 822 and the second processing unit 823 share boundary detection, etc. on both sides, or when boundary detection, etc. is completed by only one of them. In contrast, the first processing unit 822 and the second processing unit 823 can perform the same processing in parallel, for example, processing to detect the same boundary.

[0348] As such an example, for example, consider the following case: the first processing unit 822 can detect all target boundaries by using a learning model, and the second processing unit 823 can detect all target boundaries including the retinal area as the first detection target. In this case, the first processing unit 822 and the second processing unit 823 output the results of detecting each boundary respectively.

[0349] Since these detection results are the results of processing by the learning model and the results of image processing by the rule base, respectively, the two results may be different. Therefore, in this modification, the display control unit 25 can display the two results side by side on the display unit 50, can switch the display of the two results, and can display the two results in an overlapping manner. In addition, it can be determined in the image processing unit 1520 whether the two results match, and the display control unit 25 can display the unmatched part on the display unit 50 in an emphasized manner. In this case, the reliability of the layer detection can be indicated to the operator. In addition, the display control unit 25 can cause the display unit 50 to display the unmatched part, and can allow a more satisfactory result to be selected according to the operator's instructions.

[0350] (Example 5)

[0351] In Examples 2 to 4, an example has been shown in which a retinal area is detected by using a learning model and the boundary of the inner retinal layer on the rule base is detected for the detected retinal area. In contrast, in Example 5, the area detected by using a learning model is corrected based on medical features.

[0352] In the following, reference is made to Figures 17 to 19D , with respect to the image processing of the image processing system 170 according to the present example, the difference from the image processing according to Example 2 will be mainly described. Note that the configuration and processing of the image processing system according to the present example that are similar to the configuration and processing of the image processing system 8 according to Example 2 will be shown by using the same reference numerals, and the description will be omitted.

[0353] Fig.17 An example of a schematic configuration of the image processing system 170 according to the present example is indicated. The image processing unit 1720 of the image processing apparatus 172 in the image processing system 170 according to the present example is provided with a correction unit 1724 in addition to the tomographic image generation unit 221, the first processing unit 822, and the second processing unit 823.

[0354] The correction unit 1724 corrects the label area based on the medical features of the eye of the label image obtained by the first processing unit 822 using the learning model. Therefore, the image processing unit 1720 can more appropriately detect the retinal area and the feature area.

[0355] Next, refer to Fig.18A and Fig.18B , a series of processing according to this example will be described. Fig.18A is a flowchart of a series of processes according to this example, and Fig.18B 80 is a flowchart of boundary detection according to this example. Note that since the processing other than the boundary detection processing is similar to that in Example 2, the description will be omitted. When the tomographic image is generated in step S303, the processing proceeds to step S1804. When the processing proceeds to step S1804, and the processing by the first processing unit 822 is performed in step S941, the processing proceeds to step S1841.

[0356] In step S1841, the correction unit 1724 corrects the retinal area detected in step S941 by the first processing unit 822. More specifically, the labeled area is corrected based on the medical features of the eye of the label image obtained by the first processing unit 822 by using the learning model.

[0357] Here, refer to FIG. 19A to FIG. 19D , the correction processing performed by the correction unit 1724 according to the present example will be described. Fig.19A An example of a tomographic image 1901 used as an input to the first processing unit 822 is shown. Fig.19B An example of a label image 1902 obtained by using a learning model by the first processing unit 822 with a tomographic image 1901 as an input is shown. In the label image 1902, a label 1904 of the inner retinal layer, a label 1903 of the side shallower than the retina (vitreous body side), and a label 1905 of the side deeper than the retina (choroid side) are shown.

[0358] Note that labeling is performed based on the setting of labels when the learning model is learned. Therefore, the type of label is not limited to this, and multiple labels can be set on the retinal layer as shown in Example 3. In this case, the correction processing according to this example can be applied.

[0359] In this example, the first processing unit 822 performs image segmentation processing pixel by pixel by using a learning model. Fig.19B As shown in the labels 1903' and 1904', some erroneous detections may be performed. The correction unit 1724 corrects these erroneous detections based on the medical characteristics of the eye.

[0360] The first processing unit 822 performs labeling processing on each detected label, and pixels with the same label among adjacent pixels are integrated into one area. There are three types of labels given in this example, namely, label 1904 of the inner retinal layer, label 1903 on the side shallower than the retina (vitreous side), and label 1905 on the side deeper than the retina (choroid side). In addition, since the image obtained by photographing the tomographic image of the retina is the target, the order in which these labels appear is label 1903, label 1904, and label 1905, starting from the top of the image. Note that in the case of the EDI (Enhanced Depth Imaging) mode in which the choroid side is photographed in an emphasized manner, since the retina is photographed in reverse, the order in which the labels appear is label 1905, label 1904, and label 1903, starting from the top of the image.

[0361] As described above, since the image input to the first processing unit 822 is a tomographic image of the retina, the positional relationship between the labels can be estimated from the conditions at the time of imaging and the imaging site based on medical characteristics. Therefore, the correction unit 1724 specifies the detection result for each label area, and corrects the area considered to be an erroneous detection to the area estimated based on the medical characteristics.

[0362] Specifically, the correction unit 1724 specifies the label area starting from the label area with a larger area, and determines the label area with an area smaller than the threshold and the label area having a spatial distance from the already specified area as an error detection. Then, the correction unit 1724 resets the label information determined as an error detection. An example in this case is Fig.19C Shown in. Fig.19C The area 1910 shown in indicates an area whose label information has been reset by the correction unit 1724 for the areas indicated by the label 1903 ′ and the label 1904 ′ (ie, areas regarded as erroneously detected).

[0363] Correction unit 1724 assigns label information estimated from surrounding label information to region 1910 where label information has been reset. Fig.19C In the illustrated example, the label 1903 is assigned to the area 1910 surrounded by the label 1903 , and the label 1905 is assigned to the area 1910 surrounded by the label 1905 .

[0364] Through these processes of the correction unit 1724, as Fig.19D As shown, the final label image 1920 is output. Therefore, the image processing unit 1720 can detect the retinal area more appropriately.

[0365] When the correction processing is performed by the correction unit 1724, the process proceeds to step S942. In step S942, the second processing unit 823 performs second boundary detection processing based on the corrected retinal area as in Example 2. Since the subsequent processing is similar to that of Example 2, its description will be omitted.

[0366] As described above, the image processing device 172 according to the present example further includes the correction unit 1724 that corrects the structure of the retinal layer detected by the first processing unit 822 based on the medical features in the retinal layer.

[0367] Therefore, in the image processing device 172 according to the present example, for the region detected by using the learning model, correction of the region can be performed by using the medical features. Therefore, even when the image is detected pixel by pixel, false detection can be reduced.

[0368] Note that in this example, although the correction processing performed by the correction unit 1724 is added to the processing according to Example 2, the correction processing may be added to the processing according to Examples 3 and 4.

[0369] (Example 6)

[0370] In Examples 1 to 5, the boundaries of the inner retinal layer and the retinal area are detected for the captured tomographic image by using the learning model. In contrast, in this example, a high-quality image in which the image quality of the tomographic image is improved by using another learning model is generated, and the boundary detection and area detection using the learning model according to Example 1 or 2 or the like are performed on the high-quality image. Note that the improvement in image quality in this example includes noise reduction, conversion of the imaging target to easy-to-observe color and grayscale, improvement in resolution or spatial resolution, and enlargement of the image size while suppressing a decrease in resolution.

[0371] Hereinafter, with respect to the image processing of the image processing system according to the present example, the difference from the image processing according to Example 2 will be mainly described. Note that since the configuration and processing process of the image processing system according to the present example are similar to the configuration and processing process of the image processing system 8 according to Example 2, the configuration and processing process of the image processing system according to the present example will be shown by using the same reference numerals, and the description will be omitted.

[0372] In this example, the first processing unit 822 performs processing to improve the image quality of the input image by using a learning model related to an image quality improvement model, which is a machine learning model different from a machine learning model for detecting a retinal area. The image quality improvement model is a learning model that outputs an image in which the image quality of the input image is improved by training a machine learning model having an arbitrary machine learning algorithm in advance using appropriate training data.

[0373] Here, in Fig. 20 An example of training data for the image quality improvement model according to this example is shown in FIG. Fig. 20 , a tomographic image 2001 shows an example of a tomographic image obtained by imaging of OCT, and a tomographic image 2002 shows a tomographic image obtained by performing image quality improvement processing on the tomographic image 2001. The tomographic image 2001 shows an example of input data, the tomographic image 2002 shows an example of a ground truth, and the training data includes a pair including these images.

[0374] Note that as an image quality improvement process, it is possible to align tomographic images obtained by taking multiple shots of the same position in space, and perform an addition averaging process on these aligned tomographic images. Note that the image quality improvement process is not limited to the addition averaging process, and may be, for example, a process using a smoothing filter, a maximum a posteriori estimation process (MAP estimation process), or a grayscale conversion process. In addition, an image that has been subjected to an image quality improvement process may be, for example, an image that has been subjected to a filtering process such as noise removal and edge emphasis, or an image in which the contrast has been adjusted so that a low-brightness image becomes a high-brightness image. In addition, since the ground truth of the training data for the image quality improvement model may be a high-quality image, the ground truth of the training data for the image quality improvement model may be a tomographic image captured by using an OCT device having a high performance when imaging a tomographic image as input data, or a tomographic image captured with a high load setting.

[0375] The first processing unit 822 inputs the tomographic image obtained by the OCT imaging into the image quality improvement model trained by using such training data, and obtains the tomographic image with improved image quality. Note that the first processing unit 822 can obtain the volumetric tomographic image with improved image quality by inputting the tomographic image having a volume obtained by three-dimensionally scanning the retina with raster scanning into the image quality improvement model.

[0376] As in Examples 2 to 5, the first processing unit 822 detects a retinal area or a feature area by using a learning model, by using a high-quality image obtained by using an image quality improvement model as an input.

[0377] In addition, the second processing unit 823 may detect a retinal layer based on the high-quality image obtained by the first processing unit 822 and the detected retinal area or feature area.

[0378] As described above, in the image processing device 80 according to the present example, the first processing unit 822 performs the first detection processing on the tomographic image whose image quality is improved by using the learning model.

[0379] Therefore, the image processing device 80 according to the present example can improve the image quality of the input image by using the learning model of the machine learning model, and can detect the retinal layer of the image whose image quality has been improved. Therefore, the retinal layer can be detected by using the image on which the image quality improvement such as noise reduction has been performed, and false detection can be reduced.

[0380] In addition, in this example, although the process of improving the image quality of the tomographic image as the input image is added to the process according to Example 2, the image quality improvement process may be added to the processes according to Example 1 and Examples 3 to 5.

[0381] In addition, in this example, the image quality improvement model that performs image quality improvement is a machine learning model different from the machine learning model that performs detection processing. However, the machine learning model that performs detection processing can be made to learn the image quality improvement processing, and the machine learning model can be constructed to perform both image quality improvement and detection processing.

[0382] Note that in this example, the first processing unit 822 generates a high-quality image obtained by improving the image quality of the tomographic image by using a learning model (image quality improvement model) related to the image quality improvement process. However, the component that generates a high-quality image by using the image quality improvement model is not limited to the first processing unit 822. For example, a third processing unit (image quality improvement unit) different from the first processing unit 822 may be provided, and the third processing unit may generate a high-quality image by using the image quality improvement model. Therefore, the first processing unit 822 or the third processing unit may be used as an example of a generation unit that generates a tomographic image having improved image quality compared to the tomographic image by using a learning model for image quality improvement. Note that the third processing unit and the image quality improvement model may include a software module executed by a processor such as a CPU, an MPU, a GPU, and an FPGA, or may include a circuit that implements a specific function such as an ASIC.

[0383] (Example 7)

[0384] Next, refer to FIG. 21A to FIG. 23 , the image processing device 80 according to Example 7 will be described. In Example 6, the first processing unit 822 performs the first detection processing on the tomographic image whose image quality is improved by using the image quality improvement model, and detects the retinal area or the characteristic area. In this regard, the first processing unit 822 can perform the image quality improvement processing on other images by using the image quality improvement model, and the display control unit 25 can cause the display unit 50 to display various images whose image quality is improved. For example, the first processing unit 822 can perform the image quality improvement processing on the luminance En-Face image, the OCTA frontal image, etc. generated based on the information (e.g., the boundary image) related to the retinal layer detected by the first detection processing and the second detection processing. In addition, the display control unit 25 can cause the display unit 50 to display at least one of the tomographic image, the luminance En-Face image, and the OCTA frontal image on which the first processing unit 822 has performed the image quality improvement processing. Note that the image whose image quality is improved and displayed may be an SLO image, a fundus image obtained by a fundus camera, etc., a fluorescent fundus image, etc. Here, the SLO image is a frontal image of the fundus obtained by an SLO (scanning laser ophthalmoscope) optical system not shown.

[0385] Here, as for the training data of the image quality improvement model for performing image quality improvement processing on various images, similar to the training data of the image quality improvement model according to Example 6, for various images, the image before the image quality improvement processing is used as input data, and the image after the image quality improvement processing is used as the ground truth. Note that the image quality improvement processing related to the training data can be, similar to Example 6, for example, an additive averaging process, a process using a smoothing filter, a maximum a posteriori estimation process (MAP estimation process), or a grayscale conversion process. In addition, the image after the image quality improvement process can be, for example, an image that has been subjected to filtering processing such as noise removal and edge emphasis, or an image in which the contrast has been adjusted so that a low-brightness image becomes a high-brightness image can be used. In addition, since the ground truth of the training data for the image quality improvement model can be a high-quality image, the ground truth of the training data for the image quality improvement model can be an image captured by using an OCT device having a high performance when capturing an image as input data, or an image captured with a high load setting.

[0386] In addition, an image quality improvement model may be prepared for each image that has been subjected to image quality improvement processing. For example, an image quality improvement model for tomographic images, an image quality improvement model for luminance En-Face images, and an image quality improvement model for OCTA frontal images may be prepared. Furthermore, the image quality improvement model for luminance En-Face images and the image quality improvement model for OCTA frontal images may be learning models for images in different depth ranges that have been fully learned for the depth range (generation range) related to the generation of images. As for images in different depth ranges, for example, Fig.21A As shown, images for the surface layer (Im2110), deep layer (Im2120), outer layer (Im2130), and choroidal vascular network (Im1940) may be included. In addition, multiple image quality improvement models for images of different depth ranges that have been learned may be prepared for the image quality improvement model for the luminance En-Face image and the image quality improvement model for the OCTA frontal image.

[0387] In addition, when preparing an image quality improvement model for tomographic images, the image quality improvement model for tomographic images may be a learning model that has comprehensively learned tomographic images obtained at different positions in the sub-scanning (Y) direction. Fig.21BThe tomographic images Im2151 to Im2153 shown are examples of tomographic images obtained at different positions in the sub-scanning direction. However, in the case of images obtained by imaging the positions of different imaging parts (for example, the center of the macular area and the center of the optic nerve head), learning may be performed separately for each part, or learning may be performed together without paying attention to the imaging part. Note that as tomographic images for which image quality is to be improved, luminance tomographic images and tomographic images of motion contrast data may be included. However, since the image feature amount differs greatly between the luminance tomographic image and the tomographic image of motion contrast data, learning may be performed separately for each image quality improvement model.

[0388] Hereinafter, with respect to the image processing of the image processing system according to the present example, the difference from the image processing according to Example 6 will be mainly described. Note that since the configuration and processing process of the image processing system according to the present example are similar to the configuration and processing process of the image processing system 8 according to Example 6, the configuration and processing process of the image processing system according to the present example will be shown by using the same reference numerals, and the description will be omitted.

[0389] In this example, an example will be described in which the display control unit 25 displays an image on which the image quality improvement processing has been performed by the first processing unit 822 on the display unit 50. Note that in this example, although the image quality improvement processing will be performed by using Fig.22A and Fig. 22B A description is given, but the display screen is not limited to this. Image quality improvement processing (processing for improving image quality) can also be similarly applied to a display screen in which multiple images obtained at different dates and times such as in a follow-up observation are arranged side by side and displayed. In addition, the image quality improvement processing can also be similarly applied to a display screen for the inspector to confirm whether the imaging is successful immediately after the imaging, such as a imaging confirmation screen. The display control unit 25 can cause the display unit 50 to display a plurality of high-quality images generated by the first processing unit 822 and low-quality images that have not yet been improved in image quality. In addition, for a plurality of high-quality images displayed on the display unit 50 and low-quality images that have not yet been improved in image quality, the display control unit 25 can cause the display unit 50 to display each image in the low-quality image and the high-quality image selected according to the inspector's instructions. In addition, the image processing device 80 can also output the low-quality image and the high-quality image selected according to the inspector's instructions to the outside.

[0390] In the following, reference is made to Fig.22A and Fig. 22B, shows an example of a display screen 2200 of the interface according to this example. The display screen 2200 shows the entire screen, and the patient tab 220, the imaging tab 2202, the report tab 2203, and the setting tab 2204 are shown in the display screen 2200. In addition, the oblique line in the report tab 2203 indicates the activation state of the report screen. In this example, an example of displaying the report screen will be described.

[0391] exist Fig.22A The report screen shown shows an SLO image Im2205, OCTA front images Im2207 and 2208, a luminance En-Face image Im2209, tomographic images Im2211 and 2212, and a button 2220. In addition, an OCTA front image Im2206 corresponding to the OCTA front image Im2207 is superimposed and displayed in the SLO image Im2205. In addition, boundary lines 2213 and 2214 of the depth ranges of the OCTA front images Im2207 and Im2208 are superimposed and displayed in the tomographic images Im2211 and 2212, respectively. The button 2220 is a button for designating the execution of the image quality improvement process. The button 2220 may be a button for instructing the display of a high-quality image as described later.

[0392] In this example, the image quality improvement process is executed by designating the button 2220, or whether to execute the image quality improvement process is determined based on the information saved (stored) in the database. First, an example will be described in which the display of the high-quality image and the display of the low-quality image are switched by designating the button 2220 according to the instruction from the examiner. Note that hereinafter, the target image of the image quality improvement process is described as the OCTA frontal image.

[0393] Note that the depth range of the OCTA frontal images Im2207 and Im2208 can be defined by using information about the retinal layers detected by the first detection process and the second detection process. The depth range may be, for example, a range between two layer boundaries related to the detected retinal layers, or may be a range including a predetermined number of pixels in a deeper direction or a shallower direction based on one of the two layer boundaries related to the detected retinal layers. In addition, the depth range may be, for example, a range that is changed (shifted) from the range between the two layer boundaries related to the detected retinal layers according to an instruction of the operator.

[0394] When the examiner designates the report tab 2203 to transition to the report screen, the display control unit 25 displays the low image quality OCTA frontal images Im2207 and Im2208. Then, when the examiner designates the button 2220, the first processing unit 822 performs image quality improvement processing on the OCTA frontal images Im2207 and Im2208 displayed on the screen. After the image quality improvement processing is completed, the display control unit 25 displays the high-quality image generated by the first processing unit 822 on the report screen. Note that since the OCTA frontal image Im2206 is the OCTA frontal image Im2207 displayed superimposed on the SLO image Im2205, the display control unit 25 can also display the image obtained by performing the image quality improvement processing on the OCTA frontal image Im2206. In addition, the display control unit 25 can change the display of the button 2220 to an activated state so that it can be seen from the display that the image quality improvement processing has been performed.

[0395] Here, the execution of the processing in the first processing unit 822 is not necessarily limited to the timing of the examiner specifying the button 2220. Since the types of OCTA frontal images Im2207 and Im2208 displayed when the report screen is opened are known in advance, the first processing unit 822 can perform image quality improvement processing when the display screen is changed to the report screen. Then, the display control unit 25 can display the high-quality image on the report screen at the timing of pressing the button 2220. In addition, the types of images that are subjected to image quality improvement processing in response to instructions from the examiner or when changing to the report screen do not have to be two. This processing can be performed on images that are likely to be displayed, such as multiple OCTA frontal images, for example. Fig.21A The surface layer (Im2110), deep layer (Im2120), outer layer (Im2130) and choroidal vascular network (Im2140) are shown. In this case, the image that has been subjected to the image quality improvement process may be temporarily stored in a memory, or may be stored in a database.

[0396] Next, a case where image quality improvement processing is performed based on information saved (stored) in a database will be described. In a case where the state in which the image quality improvement processing is performed is saved in the database, when switching to the report screen, the display control unit 25 defaults to causing the display unit 50 to display a high-quality image obtained by performing the image quality improvement processing by the first processing unit 822. Then, the system can be constructed so that the button 2220 is displayed in an activated state by default by the display control unit 25, so that the inspector understands that the high-quality image obtained by performing the image quality improvement processing is displayed. When the inspector wants to display a low-quality image before the image quality improvement processing, by specifying the button 2220 to cancel the activation state, the display control unit 25 can cause the display unit 50 to display the low-quality image. In this case, when the inspector wants to return the displayed image to a high-quality image, by specifying the button 2220 as being in an activated state, the display control unit 25 causes the display unit 50 to display the high-quality image again.

[0397] Whether image quality improvement processing has been performed can be specified in the database for each level (for example, common to the entire data stored in the database) and each video data (each inspection). For example, when the state in which the image quality improvement processing is performed for the entire database is saved, the state in which the inspector does not perform the image quality improvement processing can be saved for a single video data (a single inspection). In this case, the single video data in which the image quality improvement processing is not performed can be displayed in a state in which the image quality improvement processing is not performed when the single video data is displayed next time. According to such a structure, when whether to perform image quality improvement processing is not specified in units of video data (in units of inspections), processing can be performed based on information specified for the entire database. In addition, when it is specified in units of video data (in units of inspections), processing can be performed individually based on this information.

[0398] Note that a user interface (e.g., a save button) not shown may be used to save the execution status of the image quality improvement process for each imaging data (each examination). In addition, when switching to other imaging data (other examinations) or other patient data (e.g., changing to a display screen other than the report screen according to the examiner's instruction), the execution status of the image quality improvement process may be saved based on the display status (e.g., the state of the button 2220).

[0399] Although an example has been shown in which the OCTA frontal images Im2207 and Im2208 are displayed as OCTA frontal images in this example, the OCTA frontal images to be displayed can be changed according to the designation of the examiner. Therefore, the change of the image to be displayed in the case where the execution of the image quality improvement process is designated (button 2220 is in an activated state) will be described.

[0400] The change of the image to be displayed can be performed by using a user interface (e.g., a combo box) not shown. For example, when the examiner changes the type of the image from the surface layer to the choroidal vascular network, the first processing unit 822 performs image quality improvement processing on the choroidal vascular network image, and the display control unit 25 displays the high-quality image generated by the first processing unit 822 on the report screen. In other words, according to the instruction from the examiner, the display control unit 25 can change the display of the high-quality image of the first depth range to the display of the high-quality image of the second depth range that is at least partially different from the first depth range. At this time, when the first depth range is changed to the second depth range according to the instruction from the examiner, the display control unit 25 can change the display of the high-quality image of the first depth range to the display of the high-quality image of the second depth range. Please note that as for the image that is highly likely to be displayed when transitioning to the report screen as described above, when a high-quality image has been generated, the display control unit 25 can display the generated high-quality image.

[0401] In addition, the method of changing the image type is not limited to the above method, and an OCTA frontal image with a different depth range set by changing the layer or offset value used as a reference can be generated, and a high-quality image obtained by performing image quality improvement processing on the generated OCTA frontal image can also be displayed. In this case, when the layer or offset value used as a reference is changed, the first processing unit 822 performs image quality improvement processing on any OCTA frontal image, and the display control unit 25 displays the high-quality image on the report screen. Note that changes to the layer and offset value used as a reference can be made by using a user interface (e.g., a combo box and a text box) not shown. In addition, the depth range (generation range) of the OCTA frontal image can also be changed by dragging any one of the boundary lines 2213 and 2214 (moving layer boundaries) superimposed and displayed on the tomographic images Im2211 and Im2212, respectively.

[0402] When the boundary line is changed by dragging, the execution command of the image quality improvement process is continuously performed. Therefore, the first processing unit 822 may always perform the process of the execution command, or may perform the process after the layer boundary is changed by dragging. Alternatively, while the execution of the image quality improvement process is continuously commanded, at the time point when the next command arrives, the previous command may be canceled and the latest command may be executed.

[0403] Note that the image quality improvement process may take a relatively long time. Therefore, whenever the command is executed at any of the above timings, it may take a relatively long time until a high-quality image is displayed. Therefore, since the depth range for generating the OCTA frontal image is set according to the instruction from the examiner, the low-image-quality OCTA frontal image (low-quality image) corresponding to the set depth range may not be displayed until the high-quality image is displayed. In other words, the system may be configured so that when the above-mentioned depth range is set, the low-image-quality OCTA frontal image (low-quality image) corresponding to the set depth range is displayed, and when the image quality improvement process is finished, the display of the low-image-quality OCTA frontal image is changed to the display of the high-quality image. In addition, since the above-mentioned depth range is set, the information indicating that the image quality improvement process is being performed may not be displayed until the high-quality image is displayed. Note that these processes are not limited to the configuration applied in the case where it is assumed that the state of performing the image quality improvement process has been specified (button 2220 is in an activated state). For example, when the image quality improvement process is instructed to be performed according to the instruction from the examiner, these processes may also be applied until the high-quality image is displayed.

[0404] In this example, although an example is shown in which OCTA frontal images Im2207 and 2108 related to different layers are displayed as OCTA frontal images and low-quality and high-quality images are displayed in a switching manner, the displayed images are not limited to these. For example, a low-image quality OCTA frontal image and a high-image quality OCTA frontal image may be displayed side by side as OCTA frontal image Im2207 and OCTA frontal image Im2208, respectively. When the images are displayed in a switching manner, since the images are switched to the same position, comparison of parts with changes can be easily performed, and when the images are displayed side by side, since the images can be displayed at the same time, the entire images can be easily compared.

[0405] Next, use Fig.22A and Fig. 22B , the execution of the image quality improvement processing in the screen transition will be described. Fig. 22B It is displayed in an enlarged manner Fig.21A An example of an OCTA frontal image of Im2207. Fig.22A , button 2220 is also displayed in Fig. 22B In. As for FIG. 22A to FIG. 22B For example, double-clicking the OCTA frontal image Im2207 to switch, and using the close button 2230 to switch from FIG. 22B to FIG. 22A Note that regarding screen transition, screen transition is not limited to the method shown here, and a user interface not shown may be used.

[0406] When the execution of the image quality improvement processing is designated during the screen transition (button 2220 is in the activated state), this state is also maintained during the screen transition. Fig.22A The screen displays a high-quality image. Fig. 22B The high-quality image is also displayed when the Fig. 22B Then, button 2220 becomes active. The same applies to Fig. 22B Transform to Fig.22A By specifying Fig. 22B The button 2220 in the figure can switch the display to a low quality image.

[0407] As for the screen transition, the screen transition is not limited to the screen shown here, and as long as it is transitioned to a screen displaying the same camera data, for example, a display screen for follow-up observation or a display screen for panorama, it is possible to perform the transition while maintaining the display state of the high-quality image. In other words, in the display screen after the transition, an image corresponding to the state of the button 2220 in the display screen before the transition can be displayed. For example, when the button 2220 in the display screen before the transition is in an activated state, a high-quality image is displayed in the display screen after the transition. In addition, for example, when the activation state of the button 2220 in the display screen before the transition is canceled, a low-quality image is displayed in the display screen after the transition. Note that when the button 2220 in the display screen for follow-up observation becomes activated, multiple images obtained at different dates and times (different examination dates) and arranged side by side and displayed on the display screen for follow-up observation can be switched to high-quality images. In other words, it can be configured so that when the button 2220 in the display screen for follow-up observation becomes activated, it can be reflected together to multiple images obtained at different dates and times.

[0408] Please note that Fig.23 An example of a display screen for follow-up observation is shown in FIG. When tab 2301 is selected in response to an instruction from the examiner, as shown in FIG. Fig.23 As shown, a display screen for follow-up observation is displayed. At this time, the depth range of the OCTA front image can be changed by selecting a set required by the examiner from the predetermined depth range sets displayed in list boxes 2302 and 2303. For example, the retinal surface layer is selected in list box 2302, and the retinal deep layer is selected in list box 2303. The analysis results of the OCTA front image of the retinal surface layer are displayed in the upper display area, and the analysis results of the OCTA front image of the retinal deep layer are displayed in the lower display area. When the depth range is selected, a plurality of images at different dates and times are collectively changed to a parallel display of the analysis results of the plurality of OCTA front images of the selected depth range.

[0409] At this time, when the display of the analysis results becomes unselected, it can be collectively changed to a parallel display of a plurality of OCTA frontal images at different dates and times. Then, when button 2220 is designated in response to an instruction from the examiner, the display of the plurality of OCTA frontal images is collectively changed to a display of a plurality of high-quality images.

[0410] In addition, when the display of the analysis results is in the selected state, when the button 2220 is designated in response to an instruction from the examiner, the display of the analysis results of the plurality of OCTA frontal images is collectively changed to the display of the analysis results of the plurality of high-quality images. Here, in the display of the analysis results, the analysis results can be displayed superimposed on the image with an arbitrary transparency. At this time, the change from the display of the image to the display of the analysis results can be, for example, a change to a state in which the analysis results are superimposed on the displayed image with an arbitrary transparency. In addition, the change from the display of the image to the display of the analysis results can be, for example, a change to display an image (e.g., a two-dimensional graph) obtained by performing a fusion process on the analysis results and the image with an arbitrary transparency.

[0411] In addition, the kind and offset position of the designated layer boundary for the depth range can be changed collectively from the user interfaces 2305 and 2306, respectively. Note that the user interfaces 2305 and 2306 for changing the kind and offset position of the layer boundary are one example, and any other form of interface may be used. Note that the depth range of a plurality of OCTA en face images of different dates and times can be collectively changed by also displaying the tomographic image and moving the layer boundary data superimposed on the tomographic image according to an instruction from the examiner. At this time, a plurality of tomographic images of different dates and times can be arranged and displayed side by side, and when the above-mentioned movement is performed on one tomographic image, the layer boundary data can be similarly moved on the other tomographic images.

[0412] In addition, the image projection method and the presence or absence of projection artifact suppression processing can be changed by, for example, selecting from a user interface such as a context menu.

[0413] In addition, a selection screen not shown can be displayed by selecting the selection button 2307, and an image selected from the image list displayed on the selection screen can be displayed. Note that Fig.23 The arrow 2304 displayed in the upper part of is a mark indicating the currently selected examination, and the baseline examination (Baseline) is the examination selected at the time of follow-up imaging ( Fig.23 Of course, a mark indicating a baseline check may be displayed in the display unit.

[0414] In addition, when the "Show Difference" check box 2308 is specified, the measurement value distribution (graph or fan graph) of the reference image is displayed on the reference image. In addition, in this case, in the area corresponding to other inspection days, the difference measurement value graph between the measurement value distribution calculated for the reference image and the measurement value distribution calculated for the image displayed in the area is displayed. As the measurement result, a trend curve graph (a curve graph of the measurement values ​​of the images of each inspection day obtained by measuring the changes over time) can be displayed on the report screen. In other words, time series data (for example, a time series curve graph) of multiple analysis results corresponding to multiple images of different dates and times can be displayed. At this time, the analysis results of dates and times other than the multiple dates and times corresponding to the multiple displayed images can also be displayed as time series data in a state distinguishable from the multiple analysis results corresponding to the multiple displayed images (for example, the color of each point on the time series curve graph is different depending on whether the image is displayed). In addition, the regression line (curve) of the trend curve graph and the corresponding formula can be displayed on the report screen.

[0415] In this example, although the OCTA frontal image has been described, the images to which the processing according to this example is applied are not limited to these. The image related to the processing such as display, image quality improvement, and image analysis according to this example may be a luminance En-Face image. In addition, the image may be not only an En-Face image but also a different image such as a tomographic image, an SLO image, a fundus image, or a fluorescent fundus image by B-scan. In this case, the user interface for performing image quality improvement processing may be a user interface for instructing image quality improvement processing to be performed on multiple images of different kinds, and a user interface for selecting an arbitrary image from multiple images of different kinds to instruct image quality improvement processing to be performed.

[0416] For example, when displaying a tomographic image by B-scan by improving image quality, Fig.22AThe tomographic images Im2211 and Im2212 shown can be displayed by improving the image quality. In addition, tomographic images whose image quality has been improved can be displayed in the area where the OCTA frontal images Im2207 and Im2208 are displayed. Note that for the tomographic images displayed by improving the image quality, only one can be displayed, or a plurality of them can be displayed. When displaying a plurality of tomographic images, tomographic images obtained at respective different positions in the sub-scanning direction can be displayed, or, when displaying a plurality of tomographic images obtained by, for example, cross scanning by improving the image quality, images in respective different scanning directions can be displayed. In addition, when displaying a plurality of tomographic images obtained by, for example, radial scanning by improving the image quality, a plurality of partially selected tomographic images (for example, two tomographic images at positions symmetrical to each other with respect to a baseline) can be displayed separately. In addition, as Fig.23 As shown, a plurality of tomographic images can be displayed on a display screen for follow-up observation, and instructions for image quality improvement and analysis results (e.g., thickness of a specific layer) can be displayed by a technique similar to the above method. In addition, image quality improvement processing can be performed on tomographic images based on information stored in a database by a technique similar to the above method.

[0417] Similarly, for example, when the SLO image is displayed by improving the image quality, the SLO image Im2205 can be displayed by improving the image quality. Also, for example, when the luminance En-Face image is displayed by improving the image quality, the luminance En-Face image 2209 can be displayed by improving the image quality. Fig.23 As shown, multiple SLO images and brightness En-Face images can be displayed on a display screen for follow-up observation, and instructions and analysis results (for example, the thickness of a specific layer) for image quality improvement can be displayed by a technique similar to the above method. In addition, the SLO images and brightness En-Face images can be processed for image quality improvement based on the information stored in the database using a technique similar to the above method. Please note that the display of tomographic images, SLO images, and brightness En-Face images is exemplary, and these images can be displayed in any form according to the desired configuration. In addition, at least two or more of the OCTA frontal images, tomographic images, SLO images, and brightness En-Face images can be displayed by improving the image quality using a single instruction.

[0418] With this configuration, the display control unit 25 can display an image, to which the image quality improvement processing has been performed according to the first processing unit 822 of this example, on the display unit 50. At this time, as described above, when at least one of the plurality of conditions related to the display of a high-quality image, the display of the analysis result, and the depth range of the displayed frontal image, etc. is selected, the selected state can be maintained even if the display screen is changed.

[0419] In addition, as described above, when at least one of the multiple conditions is selected, even if the state is changed to a state where other conditions are selected, the state where the at least one is selected can be maintained. For example, when the display of the analysis results is in the selected state, the display control unit 25 can change the display of the analysis results of the low-quality image to the display of the analysis results of the high-quality image according to an instruction from the inspector (for example, when the button 2220 is designated). In addition, when the display of the analysis results is selected, the display control unit 25 can change the display of the analysis results of the high-quality image to the display of the analysis results of the low-quality image according to an instruction from the inspector (for example, when the designation of the button 2220 is canceled).

[0420] In addition, when the display of the high-quality image is in an unselected state, the display control unit 25 can change the display of the analysis result of the low-quality image to the display of the low-quality image according to an instruction from the inspector (for example, when the designation of the display of the analysis result is canceled). In addition, when the display of the high-quality image is in an unselected state, the display control unit 25 can change the display of the low-quality image to the display of the analysis result of the low-quality image according to an instruction from the inspector (for example, when the display of the analysis result is designated). In addition, when the display of the high-quality image is in a selected state, the display control unit 25 can change the display of the analysis result of the high-quality image to the display of the high-quality image according to an instruction from the inspector (for example, when the designation of the display of the analysis result is canceled). In addition, when the display of the high-quality image is in a selected state, the display control unit 25 can change the display of the analysis result of the high-quality image to the display of the analysis result of the high-quality image according to an instruction from the inspector (for example, when the display of the analysis result is designated).

[0421] In addition, consider the following case: the display of the high-quality image is in an unselected state, and the display of the first analysis result is in a selected state. In this case, the display control unit 25 can change the display of the analysis result of the first low-quality image to the display of the analysis result of the second low-quality image according to an instruction from the examiner (for example, when the display of the second analysis result is specified). In addition, consider the following case: the display of the high-quality image is in a selected state, and the display of the first analysis result is in a selected state. In this case, the display control unit 25 can change the display of the analysis result of the first high-quality image to the display of the analysis result of the second high-quality image according to an instruction from the examiner (for example, when the display of the second analysis result is specified).

[0422] Note that the display screen for follow-up observation can be constructed so that these changes to the display can be collectively reflected on the multiple images obtained at different dates and times as described above. Here, in the display of the analysis results, the analysis results can be superimposed and displayed on the image with arbitrary transparency. At this time, the change to the display of the analysis results can be, for example, a change to a state in which the analysis results are superimposed on the displayed image with arbitrary transparency. In addition, the change to the display of the analysis results can be, for example, a change to display an image (e.g., a two-dimensional graph) obtained by fusion processing of the analysis results and the image with arbitrary transparency.

[0423] Note that in this example, the first processing unit 822 generates a high-quality image obtained by improving the image quality of the tomographic image by using a learning model (image quality improvement model) related to the image quality improvement process. However, the component that generates a high-quality image by using the image quality improvement model is not limited to the first processing unit 822. For example, a third processing unit (image quality improvement unit) different from the first processing unit 822 may be provided, and the third processing unit may generate a high-quality image by using the image quality improvement model. In this case, the third processing unit and the image quality improvement model may include a software module executed by a processor such as a CPU, an MPU, a GPU, and an FPGA, or may include a circuit that implements a specific function such as an ASIC.

[0424] (Variations of Examples 6 and 7)

[0425] In Examples 6 and 7, the display control unit 25 can cause the display unit 50 to display the high-quality image generated by the first processing unit 822 and the image selected according to the instruction from the inspector among the input images. In addition, the display control unit 25 can switch the display on the display unit 50 from the camera image (input image) to the high-quality image according to the instruction from the inspector. In other words, the display control unit 25 can change the display of the low-quality image to the display of the high-quality image according to the instruction from the inspector. In addition, the display control unit 25 can change the display of the high-quality image to the display of the low-quality image according to the instruction from the inspector.

[0426] In addition, the first processing unit 822 can start the image quality improvement process (input the image to the image quality improvement engine) through the image quality improvement engine (image quality improvement model) according to the instruction from the examiner, and the display control unit 25 can display the generated high-quality image on the display unit 50. In contrast, when the imaging device (OCT device 10) captures the input image, the first processing unit 822 can automatically generate a high-quality image based on the input image by using the image quality improvement engine, and the display control unit 25 can cause the display unit 50 to display the high-quality image according to the instruction from the examiner. Here, the image quality improvement engine includes a learning model that performs the above-mentioned process for improving image quality (image quality improvement process).

[0427] Note that these processes may also be similarly performed on the output of the analysis results. In other words, the display control unit 25 may change the display of the analysis results of the low-quality image to the display of the analysis results of the high-quality image according to the instruction from the inspector. In addition, the display control unit 25 may change the display of the analysis results of the high-quality image to the display of the analysis results of the low-quality image according to the instruction from the inspector. In addition, the display control unit 25 may change the display of the analysis results of the low-quality image to the display of the low-quality image according to the instruction from the inspector. In addition, the display control unit 25 may change the display of the analysis results of the low-quality image to the display of the analysis results of the low-quality image according to the instruction from the inspector. In addition, the display control unit 25 may change the display of the analysis results of the high-quality image to the display of the high-quality image according to the instruction from the inspector. In addition, the display control unit 25 may change the display of the analysis results of the high-quality image to the display of the analysis results of the high-quality image according to the instruction from the inspector.

[0428] In addition, the display control unit 25 can change the display of the analysis result of the low-quality image to the display of another analysis result of the low-quality image according to the instruction from the examiner. In addition, the display control unit 25 can change the display of the analysis result of the high-quality image to the display of another analysis result of the high-quality image according to the instruction from the examiner.

[0429] Here, in the display of the analysis result of the high-quality image, the analysis result of the high-quality image can be superimposed and displayed on the high-quality image with an arbitrary transparency. In addition, in the display of the analysis result of the low-quality image, the analysis result of the low-quality image can be superimposed and displayed on the low-quality image with an arbitrary transparency. At this time, the change to the display of the analysis result can be, for example, a change to a state in which the analysis result is superimposed on the displayed image with an arbitrary transparency. In addition, the change to the display of the analysis result can be, for example, a change to display an image (for example, a two-dimensional graph) obtained by performing fusion processing on the analysis result and the image with an arbitrary transparency.

[0430] Note that in this modification, the first processing unit 822 generates a high-quality image obtained by improving the image quality of the tomographic image by using a learning model (image quality improvement model) related to the image quality improvement process. However, the component that generates a high-quality image by using the image quality improvement model is not limited to the first processing unit 822. For example, a third processing unit different from the first processing unit 822 may be provided, and the third processing unit may generate a high-quality image by using the image quality improvement model. In this case, the third processing unit and the image quality improvement model may include a software module executed by a processor such as a CPU, an MPU, a GPU, and an FPGA, or may include a circuit that implements a specific function such as an ASIC.

[0431] In addition, in Example 7, an image that has been subjected to image quality improvement processing by using the image quality improvement model is displayed according to the activation state of the button 2220 in the display screen. In contrast, the system may be configured so that, according to the activation state of the button 2220, an analysis value of the result of the image segmentation processing using the learning model is displayed. In this case, for example, when the button 2220 is in an inactive state (the image segmentation processing using the learning model is in an unselected state), the display control unit 25 causes the display unit 50 to display an analysis result using the result of the image segmentation processing performed by the second processing unit 823. In contrast, when the button 2220 becomes an activated state, the display control unit 25 causes the display unit 50 to display an analysis result using the result of the image segmentation processing performed only by the first processing unit 822 or by the first processing unit 822 and the second processing unit 823.

[0432] In such a configuration, the analysis result using the result of the image segmentation processing without using the learning model and the analysis result using the result of the image segmentation processing using the learning model are switched and displayed according to the activation state of the button. Since these analysis results are respectively based on the result of the processing performed by the learning model and the result of the image processing performed by the rule base, the two results may be different. Therefore, by switching and displaying these analysis results, the examiner can compare the two results and can make a diagnosis using a more convincing analysis result.

[0433] Note that when the image segmentation processing is switched, for example, in the case where the displayed image is a tomographic image, the numerical value of the layer thickness analyzed for each layer can be switched and displayed. In addition, for example, when a tomographic image divided into layers by color, hatching pattern, etc. is displayed, a tomographic image in which the shape of the layer is changed according to the result of the image segmentation processing can be switched and displayed. In addition, when a thickness map is displayed as an analysis result, a thickness map in which the color indicating the thickness is changed according to the result of the image segmentation processing can be displayed. In addition, the button for specifying the image quality improvement processing and the button for specifying the image segmentation processing using the learning model can be set separately, only one button can be set, or both buttons can be set as one button.

[0434] In addition, similar to the switching of the above-mentioned image quality improvement processing, the switching of the image segmentation processing can be performed based on the information stored (recorded) in the database. Note that for the processing at the time of screen transition, the switching of the image segmentation processing can also be performed similarly to the above-mentioned image quality improvement processing.

[0435] In Examples 1 to 7, the obtaining unit 21 obtains the interference signal obtained by the OCT device 10 and the three-dimensional tomographic data generated by the tomographic image generating unit 221. However, the configuration in which the obtaining unit 21 obtains these signals and data is not limited thereto. For example, the obtaining unit 21 may obtain these signals from a server and an imaging device connected to the image processing device 20 via a LAN, a WAN, or the Internet. In this case, the processing related to imaging can be omitted, and the three-dimensional tomographic data that has been imaged can be obtained. Then, boundary detection processing can be performed in step S304, step S904, and the like. Therefore, a series of processing times from obtaining tomographic information to displaying a frontal image, a thickness map, and the like can be shortened.

[0436] Note that a learning model for image segmentation and a learning model for improving image quality used by the processing unit 222 and the first processing unit 822 may be set in the image processing devices 20, 80, and 152. The learning model may include, for example, a software module executed by a processor such as a CPU, an MPU, a GPU, and an FPGA, or may include a circuit such as an ASIC that implements a specific function. In addition, these learning models may be provided in another server device or the like connected to the image processing devices 20, 80, and 152. In this case, the image processing devices 20, 80, and 152 may use the learning model by connecting to a server or the like including the learning model via an arbitrary network such as the Internet. Here, the server including the learning model may be, for example, a cloud server, a fog server, and an edge server.

[0437] In addition, in Examples 2 to 7, although a label image in which each pixel is labeled has been described as a label image, a label image in which each region is labeled may be used as the label image.

[0438] Note that the configuration of the OCT apparatus 10 is not limited to the above configuration, and a part of the configuration included in the OCT apparatus 10 may be configured as a body different from the OCT apparatus 10. In addition, user interfaces such as buttons and the layout of the display are not limited to those shown above.

[0439] According to the above-described Examples 1 to 7 and their modifications, the boundary detection of the retinal layer can be performed regardless of a disease, a site, or the like.

[0440] (Example 8)

[0441] In the medical field, in order to determine the disease of the subject or observe the extent of the disease, image diagnosis is performed using images obtained by various imaging devices. As types of imaging devices, in the field of radiology, for example, there are X-ray imaging devices, X-ray computed tomography (CT), magnetic resonance imaging systems (MRI), and positron emission tomography (PET). In addition, in the field of ophthalmology, for example, there are fundus cameras, scanning laser ophthalmoscopy (SLO), optical coherence tomography (OCT) devices, and OCT angiography (OCTA) devices.

[0442] Although image diagnosis is basically performed by medical staff by observing pathological changes depicted in images, etc., in recent years, with the improvement of image analysis technology, various information useful for diagnosis can be obtained. For example, by performing image analysis, minute pathological changes that may be overlooked can be detected to help medical staff, the shape and volume of pathological changes can be quantitatively measured, and in addition, diseases can be determined without observation by medical staff.

[0443] Note that although there are a variety of techniques for image analysis, processing for specifying regions such as organs or pathological changes depicted in an image (referred to as image segmentation processing) is a processing required for performing a variety of image analysis techniques. Hereinafter, for simplicity, image segmentation processing is also referred to as segmentation processing.

[0444] Like PTL 1, conventional image segmentation processing is performed by an image processing algorithm based on medical knowledge and image features related to target organs or pathological changes. However, images obtained from an imaging device under actual medical settings may not be accurately captured due to various factors such as the clinical condition of the subject, the imaging environment of the imaging device, and insufficient skills of the person performing the imaging. Therefore, in conventional image segmentation processing, the target organ or pathological change is not depicted as expected, and in some cases, a specific area cannot be extracted with high accuracy.

[0445] Specifically, in an image obtained by photographing an eye with, for example, disappearance of the retinal layer, hemorrhage, vitiligo, and the appearance of new blood vessels by an OCT device, the shape of the retinal layer may be irregularly depicted. In this case, erroneous detection may occur in the area detection process of the retinal layer, which is one of the image segmentation processes.

[0446] One of the purposes of the following Examples 8 to 19 is to provide a medical image processing apparatus, a medical image processing method, and a program capable of performing image segmentation processing with higher accuracy than conventional image segmentation processing.

[0447] <Description of term>

[0448] Here, terms used in the present disclosure will be described.

[0449] In the network of this specification, each device can be connected by wire or wirelessly. Here, the line connecting each device in the network includes, for example, a dedicated line, a local area network (hereinafter referred to as "LAN") line, a wireless LAN line, an Internet line, Wi-Fi (registered trademark) and Bluetooth (registered trademark).

[0450] The medical image processing apparatus may include two or more devices that can communicate with each other, or may include a single device. In addition, each component of the medical image processing apparatus may include a software module executed by a processor such as a CPU, an MPU, a GPU, and an FPGA. In addition, each component may include a circuit such as an ASIC that implements a specific function. In addition, it may include a combination of any other hardware and any software.

[0451] In addition, the medical image processed by the medical image processing device or the medical image processing method is a tomographic image of the subject obtained by using an OCT device. Here, the OCT device may include a time domain OCT (TD-OCT) device and a Fourier domain OCT (FD-OCT) device. In addition, the Fourier domain OCT device may include a spectral domain OCT (SD-OCT) device and a wavelength scanning OCT (SS-OCT) device. In addition, the OCT device may include a wavefront compensation OCT (AO-OCT) device using a wavefront compensation optical system, a line OCT device that forms a measurement light irradiated on the subject in a linear shape, and a full-field OCT device that forms a surface shape measurement light.

[0452] The medical image includes a tomographic image of the eye of the subject (the subject's eye). The tomographic image of the subject's eye is not limited to a tomographic image of the retina or the like in the posterior segment of the subject's eye, and includes a tomographic image of the anterior segment of the subject's eye and an eye chamber. In addition, when the OCT apparatus is used for an endoscope or the like, a tomographic image of the skin or an organ of the subject can be used as a medical image to be processed by a medical image processing apparatus or a medical image processing method according to the following examples.

[0453] An image management system is a device and a system that receives and saves images captured by an imaging device such as an OCT device and images that have been image processed. In addition, the image management system can send images according to requests from connected devices, can perform image processing on saved images, and can request image processing requests from other devices. The image management system may include, for example, a picture archiving and communication system (PACS). In particular, the image management system according to the following example includes a database, which can also save various information associated with the received images, such as information about the subject and the imaging time. In addition, the image management system is connected to a network, and can send and receive images according to requests from other devices, can convert images, and can send and receive various information associated with the saved images.

[0454] The imaging conditions are various information when imaging the images obtained by the imaging device. The imaging conditions include, for example, information related to the imaging device, information related to the facilities for imaging, inspection information related to imaging, information related to the person performing imaging, and information about the subject. In addition, the imaging conditions include, for example, information related to the imaging date and time, the imaging part name, the imaging area, the imaging angle, the imaging system, the resolution and level of the image, the image size, the image filter applied, and the data format of the image. Please note that the imaging area may include a surrounding area offset from a specific imaging part and an area including multiple imaging parts. In addition, the imaging system may include any imaging system used for OCT, such as a spectral domain system and a wavelength scanning system.

[0455] The imaging condition may be stored in a data structure constituting an image, may be stored as imaging condition data different from the image, or may be stored in a database associated with the imaging device and the image management system. Therefore, the imaging condition may be obtained in accordance with a process corresponding to a storage unit of the imaging condition of the imaging device. Specifically, for example, the imaging condition may be obtained by analyzing the data structure of an image output by the imaging device, obtaining imaging condition data corresponding to the image, or accessing an interface for obtaining the imaging condition from a database associated with the imaging device.

[0456] Note that depending on the camera device, there are also camera conditions that cannot be obtained due to reasons such as not being saved. For example, there is a case where a function for obtaining and saving specific camera conditions does not exist in the camera device or the function is disabled. In addition, for example, there is a case where the camera conditions are not saved because they are not related to the camera device or the camera. In addition, for example, there are cases where the camera conditions are hidden, encrypted, or cannot be obtained without rights. However, there are cases where even unsaved camera conditions can be obtained. For example, the camera part name and the camera area can be specified by performing image analysis.

[0457] A region label image is a label image that gives a region label to each pixel. Specifically, Fig.24 As shown, it is the image Im2420 divided by the pixel value (hereinafter referred to as the region label value), which can specify an arbitrary region in the region depicted in the image Im2410 obtained by the camera. Here, the specified arbitrary region includes a region of interest (ROI: region of interest) and a volume of interest (VOI: volume of interest).

[0458] When the coordinates of a pixel having an arbitrary region label value are specified from the image Im2420, the coordinates of the pixel depicting the corresponding region such as the retinal layer in the image Im2410 can be specified. Specifically, for example, when the region label value indicating the ganglion cell layer constituting the retina is 1, the coordinates of the pixel value 1 among the pixels of the image Im2420 are specified, and the pixels corresponding to the coordinates are extracted from the image Im2410. Therefore, the region of the ganglion cell layer in the image Im2410 can be specified.

[0459] Note that in some examples, a process of reducing or enlarging the region label image is included. At this time, since the image completion processing technology for reducing or enlarging the region label image uses the nearest neighbor method, etc., undefined region label values ​​and region label values ​​that should not exist in the corresponding coordinates are not erroneously generated.

[0460] Image segmentation processing is processing for specifying a region called ROI or VOI such as an organ or pathological change depicted in an image to use the region for image diagnosis or image analysis. For example, according to image segmentation processing, a region of layers constituting the retina can be specified from an image obtained by imaging using the posterior segment of the eyeball as an imaging target. Note that when the region to be specified is not depicted in the image, the number of specified regions is 0. In addition, when a plurality of regions to be specified are depicted in the image, the number of specified regions may be a plurality, or may be one region surrounding the regions to include the regions.

[0461] The designated area is output as information that can be used in other processing. Specifically, for example, the coordinates of the pixels constituting each designated area can be output as a numerical data set. In addition, for example, the coordinates indicating a square area, an elliptical area, a parallelepiped area, an elliptical area, etc. including each designated area can also be output as a numerical data set. In addition, for example, the coordinates indicating a straight line, a curve, a plane or a curved surface corresponding to the boundary of the designated area can also be output as a numerical data set. In addition, for example, a region label image indicating the designated area can also be output.

[0462] Note that, in the following, when the accuracy of the image segmentation processing is expressed as high, or when the region label image is expressed as high accuracy, it indicates the fact that the percentage of correctly specified regions is high. In addition, on the contrary, when the accuracy of the image segmentation processing is expressed as low, it indicates the fact that the percentage of incorrectly specified regions is high.

[0463] The non-region label image is a type of region label image, and is a region label image in which information corresponding to the ROI or VOI to be used for image diagnosis or image analysis is not included. Specifically, as an example, the following case will be described: it is desired to find a region constituting the ganglion cell layer of the retina depicted in a medical image so as to be used for image analysis.

[0464] Here, it is assumed that the region label value indicating the region of the ganglion cell layer is 1, and the region label value indicating the remaining region is 0. When a region label image corresponding to a specific medical image is generated by image segmentation processing or the like, in the case where the ganglion cell layer is not depicted in the medical image, all pixel values ​​of the region label image are 0. In this case, the region label image is an image without a region label because there is no region corresponding to the ROI of the ganglion cell layer to be used for image analysis and having a region label value of 1 in the region label image. Note that, depending on the setting or implementation, the image without a region label may not be an image, and may be a numerical data set indicating coordinates with information similar to an image, or the like.

[0465] Here, a machine learning model refers to a learning model with a machine learning algorithm. As a specific algorithm for machine learning, the nearest neighbor method, the naive Bayes method, the decision tree, the support vector machine, etc. can be listed. In addition, deep learning (deep learning) can also be listed, which generates feature quantities for learning by itself, and generates combined weighting factors by utilizing a neural network. Appropriately, the algorithm that can be utilized in the above algorithm can be applied to the learning model according to the example. The learning model is a model that has been trained (learned) in advance according to any machine learning algorithm by using appropriate training data. However, it is assumed that the learning model is not a learning model that does not perform further learning, but a learning model that can also perform incremental learning. The training data includes one or more pairs of input data and ground truth. Please note that the format and combination of the input data and ground truth constituting the training data can be a format and combination suitable for the desired structure, such as one is an image and the other is a numerical value, one includes multiple images and the other is a string, or both are images.

[0466] Specifically, for example, training data including pairs of images obtained by OCT and imaging site labels corresponding to the images (hereinafter referred to as first training data) may be listed. Note that the imaging site label is a unique numerical value or character string representing the site. In addition, as an example of other training data, training data including pairs of images obtained by imaging using OCT with the posterior segment of the eye as an imaging target and area label images of the retinal layer corresponding to the images (hereinafter referred to as second training data) may be listed. In addition, as an example of other training data, training data including pairs of low-quality images with a large amount of noise obtained by conventional imaging of OCT and high-quality images that have undergone image quality improvement processing through multiple imaging of OCT (hereinafter referred to as third training data) may be listed.

[0467] When input data is input to the learning model, output data according to the design of the learning model is output. The learning model outputs, for example, output data that is highly likely to correspond to the input data according to a trend trained by using the training data. In addition, the learning model can output, for example, a probability corresponding to the input data as a numerical value for each output data according to a trend trained by using the training data.

[0468] Specifically, for example, when an image obtained by OCT is input to a learning model trained with the first training data, the learning model outputs an imaging site label of an imaging site imaged in the image, and outputs the probability of each imaging site label. In addition, for example, when an image depicting a retinal layer obtained by imaging with OCT using the posterior segment of the eye as an imaging target is input to a learning model trained with the second training data, the learning model outputs a regional label image of the retinal layer depicted in the image. Furthermore, for example, when a low-quality image with a large amount of noise obtained by normal imaging with OCT is input to a learning model trained with the third training data, the learning model outputs a high-quality image corresponding to an image that has been subjected to image quality improvement processing by imaging multiple times with OCT.

[0469] Machine learning algorithms include techniques related to deep learning, such as convolutional neural networks (CNNs). In techniques related to deep learning, when the parameter settings of the layers and nodes constituting the neural network are different, the degree to which the trend trained using the training data can be reproduced in the output data may vary.

[0470] For example, in a learning model of deep learning using the first training data, when more appropriate parameters are set, the probability of outputting a correct imaging part label is higher. Also, for example, in a learning model of deep learning using the second training data, when more appropriate parameters are set, a more accurate region label image can be output. Also, for example, in a learning model of deep learning using the third training data, when more appropriate parameters are set, an image with higher image quality can be output.

[0471] Specifically, the parameters in the CNN may include, for example, the kernel size of the filter set to the convolution layer, the number of filters, the stride value, and the dilation value, and the number of nodes output by the fully connected layer. Note that the parameters and the number of training epochs may be set to values ​​that are preferred for the utilization form of the learning model based on the training data. For example, based on the training data, such parameters and the number of epochs may be set that may output the correct imaging part label with a higher probability, may output a more accurate regional label image, or may output an image with higher image quality.

[0472] One of the determination methods of these parameters and the number of epochs will be described. First, 70% of the pairs constituting the training data are used for training, and the remaining 30% are randomly set for evaluation. Next, the training of the learning model is performed by using the training pairs, and the training evaluation value is calculated by using the pairs for evaluation at the end of each epoch of the training. The training evaluation value is, for example, the average value of the values ​​obtained by evaluating the output when the input data constituting each pair is input to the learning model during the training process and the ground truth value corresponding to the input data by the loss function. Finally, the parameters and the number of epochs when the training evaluation value becomes the minimum are determined as the parameters and the number of epochs of the learning model. Note that the determination of the number of epochs by dividing the pairs constituting the training data into training data for training and training data for evaluation can prevent the learning model from over-learning the pairs used for training.

[0473] The image segmentation engine is a module that performs image segmentation processing and outputs a region label image corresponding to the input image that is input. As examples of the input image, there are B-scan images of OCT, three-dimensional tomographic images (three-dimensional OCT volume images), etc. In addition, as examples of the region label image, in the case where the input image is a B-scan image of OCT, there is a region label image indicating each layer of the retinal layer, and, in the case where the input image is a three-dimensional tomographic image of OCT, there is a region label image indicating a three-dimensional region of each layer of the retinal layer.

[0474] In the image processing technology constituting the image segmentation processing technology in the following example, processing is performed using a learning model according to various machine learning algorithms such as deep learning. Note that the image processing technology can perform not only machine learning algorithms but also other existing arbitrary processing. Image processing includes processing such as various image filtering processing, matching processing using a database of region label images corresponding to similar images, image registration processing of reference region label images, and knowledge-based image processing.

[0475] In particular, there is Fig.25 The structure 2500 shown is an example of a convolutional neural network (CNN) that performs image segmentation processing on a two-dimensional image Im2510 input as an input image to generate a region label image Im2520. The structure 2500 of the CNN includes a plurality of layers responsible for processing and outputting the input values. Note that as the types of layers included in the structure 2500, such as Fig.25 As shown, there are convolution layers, downsampling layers, upsampling layers, and merging layers. Note that similar to the CNN construction 601 described in Example 1, the CNN construction 2500 used in this example is a U-net type machine learning model.

[0476] The convolution layer is a layer that performs convolution processing on input values ​​according to parameters (e.g., the kernel size of the set filter, the number of filters, the stride value, and the dilation value). The downsampling layer is a layer that performs processing to reduce the number of output values ​​to be smaller than the number of input values ​​by thinning or combining input values. As a processing performed in the downsampling layer, specifically, for example, there is a maximum pooling process.

[0477] The upsampling layer is a layer that performs a process of increasing the number of output values ​​to exceed the number of input values ​​by duplicating input values ​​or adding values ​​interpolated from input values. As a process performed in the upsampling layer, specifically, for example, there is a linear interpolation process. The merging layer is a layer that inputs values ​​such as an output value of a certain layer or pixel values ​​constituting an image from a plurality of sources, and combines these values ​​by concatenating or adding them.

[0478] Note that, for example, by setting the kernel size of the filter to 3 pixels in width and 3 pixels in height, and setting the number of filters to 64, as parameters set for the convolution layer included in the construction of the CNN, image segmentation processing with constant accuracy is achieved. However, caution is required because when the parameter settings for the layers and nodes constituting the neural network are different, the degree of tendency trained from the training data that can be reproduced in the output data may be different. In other words, in many cases, since the appropriate parameters for each layer and each node are different depending on the example, the parameters can be changed as needed.

[0479] In addition, depending on the example, CNN can obtain better characteristics not only by changing the parameters as described above, but also by changing the construction of CNN. Better characteristics are, for example, high accuracy of image segmentation processing, short time of image segmentation processing, and short time spent on learning model training. Examples of changing the construction of CNN include, for example, merging a batch normalization layer after a convolutional layer and an activation layer using a normalized linear function (Rectifier Linear Unit).

[0480] Note that as a machine learning model used as an image segmentation engine, for example, FCN or SegNet may also be used as in Example 1. In addition, a machine learning model for performing object recognition in units of regions as described in Example 1 may be used according to a desired configuration.

[0481] Note that when it is necessary to process a one-dimensional image, a three-dimensional image, and a four-dimensional image, the kernel size of the filter may correspond to one dimension, three dimensions, and four dimensions. Here, the three-dimensional image includes, for example, a three-dimensional video, and an image in which parameters at each pixel position in the three-dimensional image are indicated by different tones.

[0482] In addition, the image segmentation process may be performed by only one image processing technique, or may be performed by combining two or more image processing techniques. In addition, a plurality of image segmentation processing techniques may be performed to generate a plurality of region label images.

[0483] In addition, according to an example, there is a method for generating a region label image by dividing an input image into small regions, performing image segmentation processing on each region to obtain a region label image of the small region, and combining the region label images of the small region. Note that when the input image is a three-dimensional image, the small region may be a three-dimensional image smaller than the input image, may be a two-dimensional image, or may be a one-dimensional image. In addition, when the input image is a two-dimensional image, the small region may be a two-dimensional image smaller than the input image, or may be a one-dimensional image. In addition, depending on the example, multiple region label images may be output.

[0484] In addition, parameters may be input to the image segmentation engine along with the input image. The parameters input in this case may include, for example, parameters specifying the extent of the range for which the image segmentation process is to be performed, such as an upper limit on the size of a pathological change, and parameters specifying the size of an image filter used for the image processing technique. Note that depending on the example, the image segmentation engine may output other images and coordinate data sets that may specify a region, rather than a region label image.

[0485] Note that when performing multiple image segmentation processing techniques, and when performing image segmentation processing on multiple small areas, the processing time can be reduced by performing the image segmentation processing in parallel.

[0486] Please note that in the case of using certain image processing techniques (such as image processing using CNN), you need to be careful with the image size. In particular, it should be noted that in order to deal with problems such as segmentation processing not being performed with sufficient accuracy on the surrounding parts of the region label image, different image sizes may be required for the input image and the output region label image.

[0487] For the sake of clarity of description, although it is not explicitly mentioned in the example described later, when the image segmentation engine requires different image sizes for an image input to the image segmentation engine and an image output, it is assumed that the image size is appropriately adjusted. Specifically, the image size is adjusted by padding an image of training data for training a learning model and an input image such as an image input to the image segmentation engine, and by combining the surrounding camera areas of the input image. Note that, according to the characteristics of the image segmentation processing technology, the padded area is filled with a constant pixel value, filled with adjacent pixel values, or mirror-filled, so that the image segmentation processing can be effectively performed.

[0488] The term "camera position estimation engine" refers to a module that estimates the camera part or camera area of ​​an input image. The camera position estimation engine can output the position of the camera part or camera area depicted in the input image, or for each camera part label or camera area label at a desired level of detail, it can output the probability of being the relevant camera part or camera area.

[0489] In some cases, the imaging device does not store the imaging part or imaging area as an imaging condition, or the imaging device cannot acquire and store the imaging part or imaging area. There is also a case where, even if the imaging part or imaging area is stored, the imaging part or imaging area is not stored at the required level of detail. For example, if only the "posterior segment of the eyeball" is stored as the imaging part, it may not be known whether the detailed position of the imaging part is the "macular area", "optic nerve head", "macular area and optic nerve head", or "other areas". In addition, for another example, if only "breast" is stored as the imaging part, it may not be known in more detail whether this refers to the "right breast", "left breast", or "both". Therefore, by using the imaging position estimation engine, the imaging part or imaging area of ​​the input image in this case can be estimated.

[0490] In the image and data processing technology constituting the estimation method of the imaging position estimation engine, processing using a learning model corresponding to various machine learning algorithms such as deep learning is performed. Note that in the image and data processing technology discussed, in addition to or instead of the processing performed using the machine learning algorithm, any known estimation processing, such as natural language processing, matching processing using a database of similar images and similar data, and knowledge-based processing, may be performed. Note that an image to which a label of an imaging part or an imaging area is attached may be used as training data for training a learning model established using a machine learning algorithm. In this case, with respect to the training data, an image of the imaging part or the imaging area to be estimated is used as input data, and the label of the imaging part or the imaging area is used as the ground truth.

[0491] In particular, Fig.26 The configuration 2600 shown in FIG. 2 can be used as an example of the configuration of a CNN for estimating the imaging position of a two-dimensional input image Im2610. The configuration 2600 of the CNN includes a group of a plurality of convolution processing blocks 2620, each of which is composed of a convolution layer 2621, a batch normalization layer 2622, and an activation layer 2623 using a rectifier linear unit. The configuration 2600 of the CNN also includes a final convolution layer 2630, a fully connected layer 2640, and an output layer 2650. The fully connected layer 2640 fully connects the output value group of the convolution processing block 2620. In addition, the output layer 2650 outputs the probability of each assumed imaging site label of the input image Im210 as an estimation result (Result) 2660 using a softmax function. In this configuration 2600, for example, if the input image Im2610 is an image obtained by imaging the "macular region", the imaging site label corresponding to the "macular region" outputs the highest probability.

[0492] Note that, for example, by setting the number of convolution processing blocks 2620 to 16, and as a parameter of the group of convolution layers 2621 and 2630, setting the kernel size of the filter to a width of three pixels and a height of three pixels, and setting the number of filters to 64, the imaging part can be estimated with a certain accuracy. However, in practice, as mentioned in the description of the aforementioned learning model, a better parameter group can be set by using training data corresponding to the utilization form of the learning model. Note that in the case where it is necessary to process a one-dimensional image, a three-dimensional image, or a four-dimensional image, the kernel size of the filter can be extended to one dimension, three dimensions, or four dimensions. Note that the estimation method is sometimes performed using only one image and data processing technique, and sometimes performed using a combination of two or more image and data processing techniques.

[0493] The term "region label image evaluation engine" refers to a module that evaluates whether an input region label image has undergone possible image segmentation processing. Specifically, as an image evaluation indicator, if the input region label image is possible, the region label image evaluation engine outputs a true (True) value, and if the input region label image is unlikely, the region label image evaluation engine outputs a false (False) value. Examples of techniques for performing evaluation include processing using learning models corresponding to various machine learning algorithms such as deep learning and knowledge-based processing. One available method of knowledge-based processing is, for example, a method utilizing anatomical knowledge, for example, a method of evaluating a region label image utilizing known anatomical knowledge such as the regularity of retinal shape.

[0494] Specifically, as an example, a case will be described in which a knowledge-based process is used to evaluate a regional label image corresponding to an OCT tomographic image that targets the posterior segment of the eyeball. In the posterior segment of the eyeball, tissue groups have anatomically fixed positions. Therefore, one method that can be used is to check the coordinates of the pixel value group in the regional label image, that is, the coordinates of the regional label value group, and evaluate whether the position is correctly output. According to the evaluation method discussed, for example, within a specific range, if there is a regional label value corresponding to the lens at a coordinate close to the anterior segment of the eye, and there is a regional label value corresponding to the retinal layer group at a coordinate far from the anterior segment of the eye, it is evaluated that the regional label image has undergone possible image segmentation processing. On the other hand, if these regional label values ​​do not exist at these assumed positions, it is evaluated that the regional label image has not been appropriately processed for image segmentation.

[0495] The discussed knowledge-based evaluation method will be described in more detail using a region label image Im2710 of a layer group constituting a retinal layer, which region group image Im2710 is similar to Fig. 27 The OCT tomographic image shown in FIG. 1 corresponds to the OCT tomographic image with the posterior segment of the eyeball as the imaging target. Since the anatomical position of the tissue group in the posterior segment of the eyeball is fixed, it is possible to determine whether the regional label image is a possible image by checking the coordinates of the pixel value group in the regional label image, that is, the coordinates of the regional label value group.

[0496] The region label image Im2710 includes region Seg2711, region Seg2712, region Seg2713, and region Seg2714, each of which is composed of a group of pixels having the same region label value continuously. Although the region label values ​​of region Seg2711 and region Seg2714 are the same, because a group of layers constituting the retinal layer anatomically form a layered structure, it can be evaluated that region Seg2714 is not correctly processed for image segmentation based on the shape and positional relationship relative to other layers. In this case, the region label image evaluation engine outputs a false value as an image evaluation index.

[0497] In addition, the method of the knowledge-based evaluation processing also includes a method of evaluating whether a pixel having a region label value corresponding to a region that is definitely present in the imaging target is included in the region label image. Other available methods include, for example, a method of evaluating whether a certain number or more of pixels having a region label value corresponding to a region that is definitely present in the imaging target is included in the region label image.

[0498] In addition, when the image segmentation engine performs multiple image segmentation processing techniques and generates multiple region label images, the region label image evaluation engine can also select and output the most likely region label image among the multiple region label images.

[0499] Note that in the case where each region label image in a group of multiple region label images is a possible region label image, it may not be possible to select a single region label image to be output. In the case where a single region label image cannot be selected, the region label image evaluation engine may select a single region label image, for example, based on a predetermined priority. In addition, the region label image evaluation engine may, for example, assign weights to multiple region label images and merge the multiple region label images into a single region label image. In addition, for example, the region label image evaluation engine may display a group of multiple region label images on a user interface equipped with any display unit, etc., so that a region label image can be selected according to an instruction from an inspector (user). Note that the region label image evaluation engine may output all possible multiple region label images.

[0500] The term "region label image modification engine" refers to a module that modifies regions in an input region label image that have been incorrectly processed for image segmentation. Examples of techniques for performing the relevant modifications include knowledge-based processing. One method of available knowledge-based processing utilizes, for example, anatomical knowledge.

[0501] Will now use again Fig. 27 The knowledge-based modification method related to the modification of the region label image in question is described in more detail with reference to the region label image Im2710 of a group of layers constituting the retinal layer corresponding to an OCT tomographic image with the posterior segment of the eyeball as the imaging target. As described above, in the region label image Im2710, since a group of layers anatomically constituting the retinal layer form a layered structure, based on the shape and positional relationship relative to other layers, it is found that region Seg2714 is a region where the image segmentation processing has not been performed correctly. The region label image modification engine detects regions where the image segmentation processing has not been performed correctly, and rewrites the detected regions with different region label values. For example, in Fig. 27 In the case shown, the region tag image modification engine rewrites region Seg2714 with a region tag value indicating that region Seg2714 is not any retinal layer.

[0502] Note that the region label image modification engine can detect or identify incorrectly segmented regions by using the evaluation results obtained by the region label image evaluation engine. In addition, the region label image modification engine can rewrite the label value of the detected incorrectly segmented region with label information estimated based on label information of the region outside the relevant region. Fig. 27In the example shown, in the case where label information has been assigned to the area surrounding the area Seg2714, the label information of the area Seg2714 can be overwritten with the label information of the area in question. Note that regarding the label information of the surrounding area, the label information is not limited to the label information of the area that completely surrounds the area to be modified, and can be the label information that exists in the largest number among the label information of the area adjacent to the area to be modified.

[0503] (Configuration of Image Processing Device According to Example 8)

[0504] In the following, reference is made to Figure 28 to Figure 32 A description is given of a medical image processing apparatus of Example 8. Note that, hereinafter, in order to simplify the description, the medical image processing apparatus is simply referred to as an "image processing apparatus." Fig.28 is a diagram showing an example of a schematic configuration of an image processing apparatus according to the present example.

[0505] The image processing device 2800 is connected to the camera 2810 and the display unit 2820 via a circuit or a network. The camera 2810 and the display unit 2820 may also be directly connected. Note that although it is assumed in this example that these devices are separate devices from each other, some or all of these devices may be integrally formed with each other. In addition, these devices may be connected to any other device via a circuit or a network, and may be integrally formed with any other device.

[0506] An obtaining unit 2801, an imaging condition obtaining unit 2802, a processing possibility determining unit 2803, a segmentation processing unit 2804, an evaluation unit 2805, an analysis unit 2806, and an output unit 2807 (display control unit) are provided in the image processing device 2800. Note that the image processing device 2800 may be composed of a plurality of devices each equipped with one or more of these components.

[0507] The acquisition unit 2801 can obtain various data and images from the camera 2810 or another device, and can obtain the input of the inspector through an input device (not shown). A mouse, keyboard, touch panel and any other input device can be used as the input device. In addition, the display unit 2820 can be configured as a touch panel display.

[0508] The imaging condition obtaining unit 2802 obtains the imaging condition of the medical image (input image) obtained by the obtaining unit 2801. Specifically, according to the data format of the medical image, the imaging condition group stored in the data structure constituting the medical image is obtained. Note that in the case where the imaging condition is not stored in the medical image, the imaging information group can be obtained from the imaging device 2810 or the image management system through the obtaining unit 2801.

[0509] The processing possibility determination unit (determination unit) 2803 determines whether the relevant medical image can be processed by the segmentation processing unit 2804 using the imaging condition group obtained by the imaging condition acquisition unit 2802. The segmentation processing unit 2804 performs image segmentation processing on the medical image that can be processed using an image segmentation engine (segmentation engine) including a learning model, thereby generating a region label image (region information).

[0510] The evaluation unit 2805 uses the region label image evaluation engine (evaluation engine) to evaluate the region label image generated by the segmentation processing unit 2804, and determines whether to output the region label image based on the evaluation result. As an image evaluation index, if the input region label image is possible, the region label image evaluation engine outputs a true (True) value, and if the input region label image is unlikely, the region label image evaluation engine outputs a false (False) value. In the case where the image evaluation index determined as a result of evaluating the region label image is a true value, the evaluation unit 2805 determines to output the region label image.

[0511] The analysis unit 2806 performs image analysis processing on the input image using the region label image and the input image determined to be output by the evaluation unit 2805. The analysis unit 2806 can calculate changes in the shape or layer thickness of the tissue included in the retinal layer, for example, through the image analysis processing. Note that any known image analysis processing can be used as the image analysis processing. The output unit 2807 causes the display unit 2820 to display the region label image or the analysis result obtained by the analysis unit 2806. The output unit 2807 can also store the region label image or the analysis result in a storage device or an external device connected to the image processing device 2800.

[0512] Next, the segmentation processing unit 2804 will be described in detail. The segmentation processing unit 2804 generates a region label image corresponding to an input image (input) using an image segmentation engine. According to the image segmentation processing technique of the image segmentation engine according to this example, processing using a learning model is performed.

[0513] In this example, training data consisting of pairs of input data and ground truth values ​​are used to train a machine learning model. The input data is an image obtained under specific imaging conditions assumed to be the processing object, and the ground truth value is a region label image corresponding to the input data. Please note that the specific imaging conditions specifically include a predetermined imaging part, imaging system, imaging angle, image size, etc.

[0514] In this example, the input data of the training data is an image obtained by the same device model as the camera 2810 and using the same settings as the camera 2810. Note that the input data of the training data may be an image obtained from a camera having the same image quality tendency as the camera 2810.

[0515] In addition, the ground truth of the training data is the region label image corresponding to the input data. For example, refer to Fig.24 In the example shown in , the input data is a tomographic image Im2410 of a retinal layer photographed by OCT. In addition, the ground truth is a region label image Im2420, in which, according to the type of the retinal layer visualized in the tomographic image Im2410, region label values ​​respectively indicating the type of the corresponding retinal layer are assigned to separate each region. The region label image can be prepared in the following manner: a medical expert creates the region label image with reference to the tomographic image, creates the region label image by performing any image segmentation processing, or creates the region label image by the medical expert by modifying the region label image created by the image segmentation processing.

[0516] Note that input images having various conditions assumed to be processing objects are comprehensively included in the input data group of the pair group of training data. The term "various conditions" specifically refers to the conditions of the images generated due to differences in a combination of variations such as the disease state of the subject, the imaging environment of the imaging device, the technical level of the person performing the imaging, and the like. Since the conditions of the images included in the input data group are comprehensive, the machine learning model is trained so that high-precision image segmentation processing can also be performed on images with poor conditions, which are images for which the accuracy would be low if conventional image segmentation processing was performed. Therefore, by using an image segmentation engine including a learning model trained in this manner, the segmentation processing unit 2804 can consistently generate high-precision region label images for images of various conditions.

[0517] Note that, among the pairs constituting the training data, pairs that do not contribute to the image segmentation process may be removed from the training data. For example, in the case where the region label value of a region label image as the ground truth constituting a pair of training data is incorrect, there is a high possibility that the region label value of the region label image obtained using the learning model learned by using the relevant training data will also be incorrect. That is, the accuracy of the image segmentation process will be low. Therefore, there is a possibility that the accuracy of the learning model included in the image segmentation engine can be improved by removing the pairs of region label images having incorrect region label values ​​as the ground truth from the training data.

[0518] By using an image segmentation engine including such a learning model, when a medical image obtained by imaging is input, the segmentation processing unit 2804 can output a region label image in which an organ or lesion visualized in the medical image can be identified.

[0519] Next, we will refer to Fig.29 The shown flowchart describes a series of image processing operations according to the present example. Fig.29 29 is a flowchart showing a series of image processing operations according to the present example. First, when a series of image processing operations according to the present example is started, the process proceeds to step S2910.

[0520] In step S2910, the obtaining unit 2801 obtains, as an input image, an image photographed by the camera 2810 from the camera 2810 connected to the obtaining unit 2801 through a circuit or a network. Note that the obtaining unit 2801 can obtain the input image in response to a request from the camera 2810. For example, such a request can be issued when the camera 2810 generates an image, or when an image generated by the camera 2810 is displayed on the display unit 2820 before the image is stored in a recording device included in the camera 2810, or when a stored image is displayed on the display unit 2820 after the image is stored in the recording device, or when image analysis processing is performed using a region label image.

[0521] Note that the acquisition unit 2801 can obtain data for generating an image from the camera 2810, and the image processing device 2800 can obtain an image generated based on the relevant data as an input image. In this case, the image processing device 2800 can adopt any existing image generation method as an image generation method for generating various images.

[0522] In step S2920, the imaging condition obtaining unit 2802 obtains the imaging condition group of the input image. Specifically, according to the data format of the input image, the imaging condition obtaining unit 2802 obtains the imaging condition group stored in the data structure constituting the input image. Note that, as described above, in the case where the imaging condition is not stored in the input image, the imaging condition obtaining unit 2802 may obtain the imaging information group from the imaging device 2810 or an image management system not shown in the drawings.

[0523] In step S2930, the processing possibility determination unit 2803 uses the obtained imaging condition group to determine whether the input image can be subjected to image segmentation processing by the image segmentation engine used by the segmentation processing unit 2804. Specifically, the processing possibility determination unit 2803 determines whether the imaging part, imaging system, imaging angle of view, and image size of the input image match the conditions that can be processed using the learning model of the image segmentation engine.

[0524] The processing possibility determination unit 2803 makes a determination about all the imaging conditions, and if the processing possibility determination unit 2803 determines that the imaging conditions can be processed, the process proceeds to step S2940. On the other hand, if the processing possibility determination unit 2803 determines that the image segmentation engine cannot process the input image based on these imaging conditions, the process proceeds to step S2970.

[0525] Note that, depending on the setting or implementation form of the image processing device 2800, even if it is determined that the input image cannot be processed based on certain conditions among the imaging part, the imaging system, the imaging angle, and the image size, the processing in step S2940 may be performed. For example, such processing may be performed under the assumption that the image segmentation engine can comprehensively process any imaging part of the subject, and such processing may be implemented so that the input data can be processed even if an unknown imaging part is included in the input data. In addition, the processing possibility determination unit 2803 may determine whether at least one condition among the imaging part, the imaging system, the imaging angle, and the image size of the input image matches the condition that the image segmentation engine can process according to the desired configuration.

[0526] In step S2940, the segmentation processing unit 2804 performs image segmentation processing on the input image using the image segmentation engine, and generates a region label image from the input image. Specifically, the segmentation processing unit 2804 inputs the input image to the image segmentation engine. The image segmentation engine generates the region label image as region information that can identify organs or lesions visualized in the input image based on a learning model that performs machine learning using training data.

[0527] Note that, depending on the setting or implementation form of the image processing device 2800, the segmentation processing unit 2804 may input parameters together with the input image into the image segmentation engine according to the imaging condition group to adjust the degree of the range of the image segmentation processing, etc. In addition, the segmentation processing unit 2804 may also input parameters corresponding to the input of the inspector together with the input image into the image segmentation engine to adjust the degree of the range of the image segmentation processing, etc.

[0528] In step S2950, ​​the evaluation unit 2805 uses the region label image evaluation engine to evaluate whether the region label image generated by the segmentation processing unit 2804 is a possible image. In this example, the evaluation unit 2805 uses the region label image evaluation engine using a knowledge-based evaluation method to evaluate whether the region label image is a possible image.

[0529] Specifically, the regional label image evaluation engine checks the pixel value group in the regional label image, that is, checks the coordinates of the regional label value group, and evaluates whether the position is output at the anatomically correct position. In this case, for example, within a specific range, if there is a regional label value corresponding to the lens at a coordinate close to the anterior segment of the eye, and there is a regional label value corresponding to the retinal layer group at a coordinate far from the anterior segment of the eye, then the regional label image is evaluated to have undergone possible image segmentation processing. On the other hand, if these regional label values ​​do not exist at these assumed positions, then the regional label image is evaluated to have not been properly processed for image segmentation. For the regional label, in the case where it is evaluated that the regional label has undergone possible image segmentation processing, the regional label image evaluation engine outputs a true value as an image evaluation index, and in the case where it is evaluated that the regional label has not undergone possible image segmentation processing, the regional label image evaluation engine outputs a false value.

[0530] The evaluation unit 2805 determines whether to output the region label image based on the image evaluation index output from the region label image evaluation engine. Specifically, in the case where the image evaluation index is a true value, the evaluation unit 2805 determines to output the region label image. On the other hand, if the image evaluation index is a false value, the evaluation unit 2805 determines not to output the region label image generated by the segmentation processing unit 2804. Note that in the case where the evaluation unit 2805 determines not to output the region label image generated by the segmentation processing unit 2804, an image without a region label may be generated.

[0531] In step S2960, if the evaluation unit 2805 determines to output the region label image, the analysis unit 2806 performs image analysis processing on the input image using the region label image and the input image. The analysis unit 2806 calculates changes in layer thickness or tissue shape, etc. visualized in the input image, for example, through image analysis processing. Note that any known processing can be used as a method of image analysis processing. In addition, in the case where the evaluation unit 2805 determines not to output the region label image, or in the case of generating an image without a region label, processing is performed without performing image analysis.

[0532] In step S2970, when the evaluation unit 2805 determines that the region label image is to be output, the output unit 2807 outputs the region label image and the image analysis result, and causes the region label image and the image analysis to be displayed on the display unit 2820. Note that, instead of causing the region label image and the image analysis result to be displayed on the display unit 2820, the output unit 2807 may cause the region label image and the image analysis result to be displayed on the camera 2810 or another device, or may store the region label image and the image analysis result. In addition, depending on the setting or implementation form of the image processing device 2800, the output unit 2807 may process the region label image and the image analysis result so that they can be utilized by the camera 2810 or another device, or may convert the data format of the region label image and the image analysis result so that they can be sent to an image management system, etc. In addition, the output unit 2807 is not limited to a configuration that outputs both the region label image and the image analysis result, and may be configured to output only one of the region label image and the image analysis result.

[0533] On the other hand, if it is determined in step S2930 that the image segmentation processing cannot be performed, the output unit 2807 outputs the image without region labels as a kind of region label image so that the image without region labels is displayed on the display unit 2820. Note that instead of outputting the image without region labels, the output unit 2807 may send a signal indicating that the image segmentation processing cannot be performed to the camera 2810.

[0534] Furthermore, in the case where it is determined in step S2950 that the image segmentation process cannot be performed correctly, the output unit 2807 outputs the image without the region label so that the image without the region label is displayed on the display unit 2820. Also in this case, instead of outputting the image without the region label, the output unit 2807 may send a signal indicating that the image segmentation process cannot be performed correctly to the camera 2810. When the output process in step S2970 ends, a series of image processing operations ends.

[0535] As described above, the image processing device 2800 according to this example includes an acquisition unit 2801 and a segmentation processing unit 2804. The acquisition unit 2801 acquires an input image, which is a tomographic image of a predetermined part of a subject. The segmentation processing unit 2804 uses an image segmentation engine including a learning model to generate a region label image, which is region information that can distinguish anatomical regions from the input image. The image segmentation engine includes a learning model that uses tomographic images and region label images obtained under different conditions as training data. The tomographic image is used as an input to the image segmentation engine, and the region label image is used as an output.

[0536] The image processing apparatus 2800 further includes an evaluation unit 2805. The evaluation unit 2805 evaluates the region label image using a knowledge-based evaluation engine using anatomical features, and determines whether to output the region label image according to the evaluation result.

[0537] Through this construction, the image processing device 2800 according to this example can use a segmentation engine including a learning model to generate a region label image as region information for image diagnosis or for identifying an ROI or VOI that can be used for image analysis. Therefore, for an input image whose condition is also unfavorable relative to conventional segmentation processing, a high-precision region label image can be output, and it can be provided for image diagnosis, or an ROI or VOI that can be used for image analysis can be provided.

[0538] In addition, by evaluating whether the region label image is a possible image using the evaluation unit 2805, the image processing apparatus 2800 can prevent an inappropriate region label image from being used for image diagnosis or image analysis.

[0539] In addition, the image processing device 2800 also includes a photographing condition obtaining unit 2802 and a processing possibility determining unit 2803. The photographing condition obtaining unit 2802 obtains a photographing condition including at least one of a photographing part of an input image, a photographing system, a photographing angle of view, and an image size. The processing possibility determining unit 2803 determines whether a region label image can be generated from the input image using an image segmentation engine. The processing possibility determining unit 2803 makes the determination in question based on the photographing condition of the input image.

[0540] With this configuration, the image processing apparatus 2800 according to the present example can omit an input image that cannot be processed by the segmentation processing unit 2804 from the image segmentation processing, and thus can reduce the processing load of the image processing apparatus 2800 and the occurrence of errors.

[0541] In addition, the image processing device 2800 also includes an analysis unit 2806 that performs image analysis on the input image using the region label image as region information. With this configuration, the image processing device 2800 can perform image analysis using the high-precision region label image generated by the segmentation processing unit 2804, and can obtain a high-precision analysis result.

[0542] In this example, the processing possibility determination unit 2803 determines whether the input image can be subjected to the image segmentation processing by the image segmentation engine. Thereafter, if the processing possibility determination unit 2803 determines that the input image is an image that can be processed by the segmentation processing unit 2804, the segmentation processing unit 2804 performs the image segmentation processing. In this regard, in the case where the image is captured only by the camera 2810 or the like under the imaging condition that enables the image segmentation processing, the image obtained from the camera 2810 can be subjected to the image segmentation processing unconditionally. In this case, as Fig.30 As shown, the processing in step S2920 and step S2930 can be omitted, and step S2940 can be performed after step S2910.

[0543] Note that although the output unit 2807 (display control unit) is configured in this example so that the generated region label image or analysis result is displayed on the display unit 2820, the operation of the output unit 2807 is not limited thereto. For example, the output unit 2807 may also output the region label image or analysis result to the camera 2810 or another device connected to the image processing device 2800. Therefore, the region label image or analysis result may be displayed on a user interface in these devices, may be stored in any recording device, may be used for any image analysis, or may be sent to an image management system.

[0544] Furthermore, in this example, the output unit 2807 is configured so that the region label image or the image analysis result is displayed on the display unit 2820. However, in response to an instruction from the inspector, the output unit 2807 may cause the region label image or the image analysis result to be displayed on the display unit 2820. For example, in response to the inspector pressing any button on the user interface of the display unit 2820, the output unit 2807 may display the region label image or the image analysis result on the display unit 2820. In this case, the output unit 2807 may switch from displaying the input image to displaying the region label image. Furthermore, as Fig.31 As shown, the output unit 2807 can display the region label image UI 3120 and the input image UI 3110 side by side, or as shown in FIG. Fig.32 As shown in any one of UI3210 to UI3240 in FIG. 32, the region label image that has become semi-transparent can be displayed on the input image in an overlay manner. Note that any known method can be used as a method of making the region label image semi-transparent, and for example, the region label image can be made semi-transparent by setting the transparency of the region label image to a desired value.

[0545] Furthermore, the output unit 2807 may cause information to the effect that the region label image is an image generated using the learning model, or information to the effect that the image analysis result is a result obtained by performing image analysis based on the region label image generated using the learning model, to be displayed on the display unit 2820. In addition, the output unit 2807 may cause a display indicating which kind of training data the learning model used for learning to be displayed on the display unit 2820. The display in question may include a description of the kind of input data and the ground truth value of the training data, or any display related to the training data, such as an imaging site and a ground truth value included in the input data.

[0546] In this example, if the processing possibility determination unit 2803 determines that the input image can be processed by the image segmentation engine, the process proceeds to step S2940, and the image segmentation process by the segmentation processing unit 2804 is started. In this regard, it is also possible to adopt a configuration in which the output unit 2807 causes the determination result of the processing possibility determination unit 2803 to be displayed on the display unit 2820, and the segmentation processing unit 2804 starts the image segmentation process in response to an instruction from the examiner. At this time, the output unit 2807 may also cause the input image or the imaging condition such as the imaging part obtained with respect to the input image to be displayed on the display unit 2820 together with the determination result. In this case, since the image segmentation process is performed after the examiner has determined whether the determination result is correct, the image segmentation process based on erroneous determination can be reduced.

[0547] Furthermore, in this regard, a configuration may be adopted in which the processing possibility determination unit 2803 does not make the determination, and the output unit 2807 causes the input image or imaging conditions such as an imaging part obtained with respect to the input image to be displayed on the display unit 2820. In this case, the segmentation processing unit 2804 may start the image segmentation processing in response to an instruction from the examiner.

[0548] In addition, according to the present example, the evaluation unit 2805 uses a region label image evaluation engine that adopts a knowledge-based evaluation method to evaluate the region label image generated by the segmentation processing unit 2804. In this regard, the evaluation unit 2805 may evaluate whether the region label image is a possible image using the region label image evaluation engine that includes a learning model that is trained using the region label image and an image evaluation index obtained by a predetermined evaluation technique as training data.

[0549] In this case, the training data of the machine learning model included in the region label image evaluation engine uses a region label image and a fake image that looks like a region label image as input data, and uses an image evaluation index about each image as a ground truth. As the image evaluation index, a true value is used when the input data is an appropriate region label image, and a false value is used when the input data is a false image. Note that the method for generating a false image may be a method of using an arbitrary generator of a region label image for which an inappropriate condition is set, or a method of generating a false image by intentionally rewriting an inappropriate region label onto an appropriate region label image, etc.

[0550] When the evaluation unit 2805 evaluates whether a region label image is a possible image using a region label image evaluation engine that also includes a learning model that has undergone such learning, image diagnosis or image analysis using an inappropriate region label image can be prevented.

[0551] Note that, in the case where it is determined by the processing possibility determination unit 2803 that image segmentation processing cannot be performed, an image without region labels may be generated by the output unit 2807, or may be generated by the processing possibility determination unit 2803. Furthermore, in the case where it is determined by the evaluation unit 2805 that image segmentation is not properly performed, in step S2970, the output unit 2807 may cause information to the effect that image segmentation cannot be correctly performed to be displayed on the display unit 2820.

[0552] (Example 9)

[0553] Next, we will refer to Fig.28 and Fig.33 An image processing apparatus according to Example 9 is described. In Example 8, the segmentation processing unit 2804 is provided with one image segmentation engine. In contrast, in this example, the segmentation processing unit generates a plurality of region label images with respect to an input image using a plurality of image segmentation engines including learning models that perform machine learning using mutually different training data.

[0554] Unless otherwise explicitly stated, the configuration and processing of the image processing device according to this example are the same as those of the image processing device 2800 according to Example 8. Therefore, hereinafter, the image processing device according to this example will be described centering on the differences from the image processing device according to Example 8. Note that since the configuration of the image processing device according to this example is the same as that of the image processing device according to Example 8, Fig.28 The components shown in are denoted by the same figure marks as those in Example 8, and the description of these components is omitted below.

[0555] The segmentation processing unit 2804 according to the present example performs image segmentation processing on an input image using two or more image segmentation engines including learning models that have performed machine learning using mutually different training data.

[0556] A method for creating a training data set according to this example will now be described. Specifically, first, imaging of various imaging sites is performed, and pairs consisting of images as input data and region label images as ground truth values ​​are prepared. Next, the pairs are grouped for each imaging site to create a training data set. For example, a training data set is created by creating first training data consisting of pairs obtained by imaging a first imaging site, and creating second training data consisting of pairs obtained by imaging a second imaging site.

[0557] Thereafter, the machine learning models included in each image segmentation engine are made to perform machine learning using each training data. For example, a first image segmentation engine including a learning model trained using first training data is prepared. In addition, a second image segmentation engine including a learning model trained using second training data is prepared. In this way, an image segmentation engine group is prepared.

[0558] The training data used to train the learning model included in each image segmentation engine is different for each image segmentation engine. Therefore, the degree of accuracy with which each of these image segmentation engines can perform image segmentation processing on an input image input to the image segmentation engine will differ depending on the imaging conditions of the input image. Specifically, in the case of the first image segmentation engine, the accuracy of the image segmentation processing for the input image obtained by imaging the first imaging part is high, and the accuracy of the image segmentation processing for the input image obtained by imaging the second imaging part is low. Similarly, in the case of the second image segmentation engine, the accuracy of the image segmentation processing for the input image obtained by imaging the second imaging part is high, and the accuracy of the image segmentation processing for the image obtained by imaging the first imaging part is low.

[0559] Since each training data set is composed of pairs grouped according to the imaging part, there will be similar image quality trends between the images in the image groups constituting the relevant pair groups. Therefore, if the imaging part corresponds to the relevant image segmentation engine, the image segmentation engine can perform image segmentation processing more accurately than the image segmentation engine according to Example 8. Note that the imaging condition used to group the paired training data is not limited to the imaging part, and may be the imaging angle of view or resolution of the image, or a combination of two or more of these conditions.

[0560] In the following, reference will be made to Fig.33 A series of image processing operations according to this example is described. Fig.33 is a flowchart showing a series of image processing operations according to this example. Note that the processing in step S3310 and step S3320 is the same as the processing in step S2910 and step S2920 according to Example 8, so the description of the processing is omitted here. Note that in the case where image segmentation processing is to be performed unconditionally on the input image, after the processing in step S3320 is performed, the processing in step S3330 can be omitted, and the processing can proceed to step S3340.

[0561] After obtaining the imaging conditions of the input image in step S3320, the process proceeds to step S3330. In step S3330, the processing possibility determination unit 2803 uses the imaging condition group obtained in step S3320 to determine whether any one of the above-mentioned image segmentation engine groups can process the input image.

[0562] If the processing possibility determination unit 2803 determines that none of the image segmentation engine group can process the input image, the process proceeds to step S3380. On the other hand, if the processing possibility determination unit 2803 determines that any one of the image segmentation engine group can process the input image, the process proceeds to step S3340. Note that, depending on the setting or implementation form of the image processing device 2800, similar to Example 8, even if it is determined that the image segmentation engine cannot process certain imaging conditions, the process in step S3340 may be performed.

[0563] In step S3340, the segmentation processing unit 2804 selects an image segmentation engine for processing based on the imaging conditions of the input image obtained in step S3320 and the information related to the training data of the image segmentation engine group. Specifically, for example, the segmentation processing unit 2804 selects an image segmentation engine that has information on the training data related to the same imaging part or surrounding imaging parts for the imaging part in the imaging condition group obtained in step S3320 and can perform high-precision image segmentation processing. In the above example, if the imaging part is the first imaging part, the segmentation processing unit 2804 selects the first image segmentation engine.

[0564] In step S3350, the segmentation processing unit 2804 generates a region label image by using the image segmentation engine selected in step S3340 to perform image segmentation processing on the input image. Steps S3360 and S3370 are the same as steps S2950 and S2960 in Example 8, so descriptions of these steps are omitted here.

[0565] After image analysis is performed in step S3370, the process proceeds to step S3380. In step S3380, if the evaluation unit 2805 determines that the region label image is to be output, the output unit 2807 outputs the region label image and the analysis result so that the region label image and the analysis result are displayed on the display unit 2820. Note that when the region label image is displayed on the display unit 2820, the output unit 2807 may also cause information to be displayed to the effect that the region label image is a region label image generated using the image segmentation engine selected by the segmentation processing unit 2804. Note that the output unit 2807 may also output only one of the region label image and the analysis result.

[0566] On the other hand, if it is determined in step S3330 that the image segmentation processing cannot be performed, the output unit 2807 outputs an image without region labels as a type of region label image so that the image without region labels is displayed on the display unit 2820. Note that instead of generating an image without region labels, the output unit 2807 may send a signal indicating that the image segmentation processing cannot be performed to the camera 2810.

[0567] Furthermore, in the case where it is determined in step S3360 that the image segmentation process cannot be performed correctly either, the output unit 2807 outputs an image without a region label as a type of region label image so that the image without a region label is displayed on the display unit 2820. Also in this case, instead of generating an image without a region label, the output unit 2807 may send a signal indicating that the image segmentation process cannot be performed correctly to the camera 2810. When the output process in step S3380 ends, a series of image processing operations ends.

[0568] As described above, the segmentation processing unit 2804 according to this example uses at least one of a plurality of image segmentation engines to generate a region label image, and each of the plurality of image segmentation engines includes a learning model that uses training data different from the training data used by the learning models of other image segmentation engines for learning. In this example, each of the plurality of image segmentation engines includes a learning model that uses training data different from the training data used by other learning models for learning for at least one of the imaging part, the imaging angle, and the image resolution. The segmentation processing unit 2804 uses the image segmentation engine to generate a region label image based on at least one imaging condition of the imaging part, the imaging angle, and the image resolution of the input image.

[0569] According to this configuration, the image processing device 2800 according to the present example can perform more accurate image segmentation processing according to the imaging conditions.

[0570] Although in this example, the segmentation processing unit 2804 selects an image segmentation engine to be used for the image segmentation processing based on the imaging condition of the input image, the process for selecting the image segmentation engine is not limited thereto. For example, the output unit 2807 may cause the imaging condition of the input image obtained and the image segmentation engine group to be displayed on the user interface of the display unit 2820. In addition, the segmentation processing unit 2804 may select an image segmentation engine to be used for the image segmentation processing according to an instruction from the examiner.

[0571] Note that the output unit 2807 can cause information related to training data used for learning by each image segmentation engine to be displayed together with the image segmentation engine group on the display unit 2820. Note that the information related to training data used for learning by the image segmentation engine can be displayed in any form, and, for example, the image segmentation engine group can be displayed using a name associated with the training data used for learning.

[0572] In addition, the output unit 2807 may cause the image segmentation engine selected by the segmentation processing unit 2804 to be displayed on the user interface of the display unit 2820, and may accept instructions from the examiner. In this case, the segmentation processing unit 2804 may determine whether to finally select the relevant image segmentation engine as the image segmentation engine to be used for image segmentation processing according to the instructions from the examiner.

[0573] Note that, similar to Example 8, the output unit 2807 may output the generated region label image and evaluation result to the camera 2810 or another device connected to the image processing device 2800. Furthermore, depending on the setting or implementation form of the image processing device 2800, the output unit 2807 may process the region label image and evaluation result so that they can be utilized by the camera 2810 or another device, or may convert the data format of the region label image and evaluation result so that they can be sent to an image management system or the like.

[0574] (Example 10)

[0575] Next, we will refer to Fig.28 and Fig.33 An image processing apparatus according to Example 10 is described. In Examples 8 and 9, the imaging condition obtaining unit 2802 obtains an imaging condition group from the data structure of an input image, etc. In contrast, in this example, the imaging condition obtaining unit estimates an imaging part or imaging area of ​​the input image based on the input image using an imaging position estimation engine.

[0576] Unless otherwise explicitly stated, the configuration and processing of the image processing device according to this example are the same as those of the image processing device 2800 according to Example 9. Therefore, hereinafter, the image processing device according to this example will be described centering on the differences from the image processing device according to Example 9. Note that since the configuration of the image processing device according to this example is the same as that of the image processing devices according to Examples 8 and 9, Fig.28 The components shown in are denoted by the same figure marks as Examples 8 and 9, and the description of these components is omitted below.

[0577] The imaging condition obtaining unit 2802 according to the present example uses an imaging position estimation engine (estimation engine) to estimate and obtain an imaging part or imaging area depicted in the input image obtained by the obtaining unit 2801. According to the technique for estimating the imaging position of the imaging position estimation engine of the present example, a machine learning algorithm is used to perform estimation processing.

[0578] In this example, training data consisting of a pair group is used to train a learning model related to a camera position estimation technique using a machine learning algorithm, and the pair group consists of input data as an image and a ground truth as a camera part label corresponding to the input data. Here, the term "input data" refers to an image (input image) having specific camera conditions assumed for a processing object. An image obtained from a camera having the same image quality trend as the camera 2810 can be used as input data, and it is better if the camera has the same device model as the camera 2810 and is provided with the same settings as the camera 2810. The type of the camera part label used as the ground truth value can be a related camera part at least partially included in the input data. For example, the type of the camera part label used as the ground truth value can be "macular area", "optic nerve head", "macular area and optic nerve head" and "other".

[0579] By including a learning model learned using such training data, the imaging position estimation engine according to this example can output information indicating the position of the imaging part or imaging area visualized in the input image. In addition, for each imaging part label or imaging area label at a desired level of detail, the imaging position estimation engine can also output the probability of being the relevant imaging part or imaging area.

[0580] By using the imaging position estimation engine, the imaging condition acquisition unit 2802 can estimate the imaging part or imaging area of ​​the input image based on the input image, thereby obtaining the imaging part or imaging area as the imaging condition of the input image. Note that in the case where the imaging position estimation engine outputs the probability of the relevant imaging part or imaging area for each imaging part label or imaging area label, the imaging condition acquisition unit 2802 obtains the imaging part or imaging area with the highest probability as the imaging condition of the input image.

[0581] Next, similar to Example 9, refer to Fig.33 The flowchart in describes a series of image processing operations according to this example. Note that since the processing of step S3310 and steps S3330 to S3380 according to this example is the same as the processing in these steps in Example 9, the description of the processing is omitted here. Note that in the case where image segmentation processing is to be performed unconditionally on the input image, after the processing in step S3320, the processing in step S3330 can be omitted, and the processing can proceed to step S3340.

[0582] When the input image is obtained in step S3310, the process proceeds to step S3320. In step S3320, the imaging condition obtaining unit 2802 obtains an imaging condition group of the input image obtained in step S3310.

[0583] Specifically, the imaging condition group stored in the data structure constituting the input image is obtained according to the data format of the input image. In addition, if the information related to the imaging part or the imaging area is not included in the imaging condition group, the imaging condition acquisition unit 2802 inputs the input image to the imaging position estimation engine to estimate which imaging part or imaging area is to be imaged to obtain the input image. Specifically, the imaging condition acquisition unit 2802 inputs the input image to the imaging position estimation engine, evaluates the probability of each imaging part or imaging area output for the imaging part label group or the imaging area label group, and sets and obtains the imaging part or imaging area with the highest probability as the imaging condition of the input image.

[0584] Note that, in the case where the imaging conditions other than the imaging part or the imaging area are not stored in the input image, the imaging condition obtaining unit 2802 may obtain the imaging information group from the imaging device 2810 or the image management system (not shown). The subsequent processing is the same as the series of image processing operations according to Example 9, so the description thereof is omitted here.

[0585] As described above, the imaging condition obtaining unit 2802 according to the present example functions as an estimation unit that estimates at least one of an imaging part and an imaging area of ​​an input image using an imaging position estimation engine including a learning model. The imaging condition obtaining unit 2802 estimates the imaging part or the imaging area of ​​an input image by inputting the input image to the imaging position estimation engine including a learning model that performs learning using images each with a label attached to the imaging part or the imaging area as training data.

[0586] Therefore, the image processing device 2800 according to the present example can obtain the imaging conditions regarding the imaging part or the imaging area of ​​the input image based on the input image.

[0587] Note that in this example, in the case where the imaging condition group does not include information about the imaging part or the imaging area, the imaging condition obtaining unit 2802 uses the imaging position estimation engine to perform estimation on the imaging part or the imaging area of ​​the input image. However, the case where the imaging position estimation engine is used to perform estimation on the imaging part or the imaging area is not limited to this case. In the case where the information about the imaging part or the imaging area included in the data structure of the input image is not sufficient as information of the required level of detail, the imaging condition obtaining unit 2802 may also use the imaging position estimation engine to perform estimation on the imaging part or the imaging area.

[0588] Furthermore, regardless of whether the data structure of the input image includes information about the imaging part or the imaging area, the imaging condition obtaining unit 2802 can estimate the imaging part or the imaging area of ​​the input image using the imaging position estimation engine. In this case, the output unit 2807 can cause the display unit 2820 to display the estimation result output from the imaging position estimation engine and the information about the imaging part or the imaging area included in the data structure of the input image, and the imaging condition obtaining unit 2802 can make determinations about these imaging conditions according to instructions from the examiner.

[0589] (Example 11)

[0590] Next, refer to Fig.28 , Fig.29 , Fig.34 and Fig.35 An image processing device according to Example 11 is described. In this example, the segmentation processing unit enlarges or reduces the input image so that the size of the input image becomes an image size that can be processed by the image segmentation engine. In addition, the segmentation processing unit generates a region label image by reducing or enlarging the output image from the image segmentation engine so that the image size of the output image becomes the image size of the input image.

[0591] Unless otherwise explicitly stated, the configuration and processing of the image processing device according to this example are the same as those of the image processing device 2800 according to Example 8. Therefore, hereinafter, the image processing device according to this example will be described centering on the differences from the image processing device according to Example 8. Note that since the configuration of the image processing device according to this example is the same as that of the image processing device according to Example 8, Fig.28 The components shown in are denoted by the same figure marks as those in Example 8, and the description of these components is omitted below.

[0592] The segmentation processing unit 2804 according to this example includes an image segmentation engine similar to that according to Example 8. However, in this example, the training data used for learning by the machine learning model included in the image segmentation engine is different from the training data in Example 8. Specifically, in this example, a pair of input data and ground truth values ​​consisting of an image group in which each image of the input data and each image of the ground truth value are enlarged or reduced to a specific image size is used as the training data.

[0593] Now refer to Fig.34 Describe the training data of the learning model included in the image segmentation engine according to this example. Fig.34 As shown, for example, let us consider the case where there are input images Im3410 and region label images Im3420 that are smaller than the specific image size set for training data. In this case, the input image Im3410 and the region label image Im3420 are each enlarged to the specific image size set for training data. Then the enlarged image Im3411 and the enlarged region label image Im3421 are taken as a pair, and the relevant pair is used as a piece of training data.

[0594] Note that, similar to Example 8, for the input data of the training data, an image (input image) having a specific imaging condition assumed to be a processing object is used, and the relevant specific imaging conditions are a predetermined imaging part, an imaging system, and an imaging angle. In other words, unlike Example 8, according to this example, the image size is not included in the specific imaging condition.

[0595] The segmentation processing unit 2804 according to the present example generates a region label image by performing image segmentation processing on the input image using the image segmentation engine, which is trained using such training data. At this time, the segmentation processing unit 2804 generates a modified image by enlarging or reducing the input image so that it becomes a specific image size set for the training data, and inputs the modified image to the image segmentation engine.

[0596] In addition, the segmentation processing unit 2804 generates a region label image by reducing or enlarging the output image from the image segmentation engine to the image size of the input image. Therefore, even in the case where the input image has an image size that cannot be processed according to Example 8, the segmentation processing unit 2804 according to this example can generate a region label image by image segmentation processing using the image segmentation engine.

[0597] Next, refer to Fig.29 and Fig.35 A series of image processing operations according to this example is described. Fig.35 is a flowchart showing the segmentation processing according to the present example. Note that the processing in step S2910, step S2920, and steps S2950 to S2970 according to the present example is the same as the processing in these steps in Example 8, so the description of the processing is omitted here. Note that in the case where the image segmentation processing is to be performed unconditionally on the input image with respect to the imaging conditions other than the image size, after the processing in step S2920 is performed, the processing in step S2930 may be omitted, and the processing may proceed to step S2940.

[0598] In step S2920, similar to Example 8, when the imaging condition obtaining unit 2802 obtains the imaging condition group of the input image, the process proceeds to step S2930. In step S2930, the processing possibility determination unit 2803 uses the obtained imaging condition group to determine whether the image segmentation engine can process the input image. Specifically, for the imaging condition of the input image, the processing possibility determination unit 2803 determines whether the image segmentation engine can process the imaging part, the imaging system, and the imaging angle of view. Unlike Example 8, the processing possibility determination unit 2803 does not perform determination regarding the image size.

[0599] The processing possibility determination unit 2803 performs determination regarding the imaging part, the imaging system, and the imaging angle of view, and if it is determined that the input image can be processed, the process proceeds to step S2940. On the other hand, in the case where the processing possibility determination unit 2803 determines that the image segmentation engine cannot process the input image based on these imaging conditions, the process proceeds to step S2970. Note that, depending on the setting and implementation form of the image processing device 2800, even if it is determined that the input image cannot be processed based on one or more of the imaging part, the imaging system, and the imaging angle of view, the image segmentation process in step S2940 may be performed.

[0600] When the process enters step S2940, Fig.35The image segmentation processing according to this example is shown. In the image segmentation processing according to this example, first, in step S3510, the segmentation processing unit 2804 enlarges or reduces the input image to a specific image size set for the training data, thereby generating a modified image.

[0601] Next, in step S3520, the segmentation processing unit 2804 inputs the generated modified image into the image segmentation engine to obtain a first region label image that has undergone image segmentation processing.

[0602] Thereafter, in step S3530, the segmentation processing unit 2804 reduces or enlarges the first region label image to the image size of the input image to generate a final region label image. When the segmentation processing unit 2804 generates the final region label image in step S3530, the image segmentation processing according to this example ends, and the processing proceeds to step S2950. Since the processing from step S2950 is the same as the processing from step S2950 in Example 8, the description of the processing is omitted here.

[0603] As described above, the segmentation processing unit 2804 according to this example adjusts the image size of the input image to an image size that can be processed by the image segmentation engine, and inputs the resulting image whose size is adjusted into the image segmentation engine. In addition, the segmentation processing unit 2804 generates a region label image by adjusting the image size of the output image from the image segmentation engine to the original image size of the input image. Therefore, the image processing device 2800 of this example can also perform image segmentation processing on an input image having an image size that cannot be processed according to Example 8, and can generate a region label image including information related to an ROI or VOI that can be used for image diagnosis or image analysis.

[0604] (Example 12)

[0605] Next, refer to Fig.28 , Fig.29 , Fig.36 and Fig.37 An image processing apparatus according to Example 12 is described. In this example, the segmentation processing unit generates a region label image through image segmentation processing based on a specific resolution performed by an image segmentation engine.

[0606] Unless otherwise explicitly stated, the configuration and processing of the image processing device according to this example are the same as those of the image processing device 2800 according to Example 8. Therefore, hereinafter, the image processing device according to this example will be described centering on the differences from the image processing device according to Example 8. Note that since the configuration of the image processing device according to this example is the same as that of the image processing device according to Example 8, Fig.28The components shown in are denoted by the same figure marks as those in Example 8, and the description of these components is omitted below.

[0607] The segmentation processing unit 2804 according to the present example includes an image segmentation engine similar to that of Example 8. However, in the present example, the training data used for learning by the machine learning model included in the image segmentation engine is different from the training data in Example 8. Specifically, after an image group consisting of a pair of input data of the training data and a ground truth value is enlarged or reduced to an image size such that the resolution of the image group becomes an image size of a specific resolution, padding is performed so that the image size of each image of the image group becomes a sufficiently large specific image size. Here, the phrase "resolution of the image group" refers to, for example, the spatial resolution of the camera device or the resolution for the camera area.

[0608] Now refer to Fig.36 Describe the training data of the image segmentation engine according to this example. For example, Fig.36 As shown, let us consider the case where there are images Im3610 and region label images Im3620 having a resolution lower than the specific resolution set for the training data. In this case, the image Im3610 and the region label image Im3620 are each enlarged so that the resolution becomes the specific resolution set for the training data. In addition, the enlarged image Im3610 and the region label image Im3620 are each padded to become the specific image size set for the training data. Then the enlarged and padded image Im3611 and the region label image Im3621 are taken as a pair, and the pair is used as a piece of training data.

[0609] Note that the phrase "specific image size set for training data" means that the image (input image) assumed to be the object of processing can become the maximum image size when it is enlarged or reduced so that the resolution of the image is a specific resolution. In the case where the specific image size is not large enough, there is a possibility that when the image input to the image segmentation engine is enlarged, the image will become an image size that cannot be processed by the learning model.

[0610] In addition, according to the characteristics of the learning model, the filled area is filled with a fixed pixel value, filled with a neighboring pixel value, or filled with a mirror image, so that the image segmentation process can be effectively performed. Note that similar to Example 8, for the input data, an image with a specific imaging condition assumed to be the processing object is used, and the specific imaging condition in question is a predetermined imaging part, an imaging system, and an imaging angle. In other words, unlike Example 8, according to this example, the image size is not included in the specific imaging condition.

[0611] The segmentation processing unit 2804 according to this example generates a region label image by performing image segmentation processing on the input image using an image segmentation engine including a learning model, and the learning model is learned using such training data. At this time, the segmentation processing unit 2804 generates a modified image by enlarging or reducing the input image to make it a specific resolution set for the training data. In addition, the segmentation processing unit 2804 performs padding on the modified image so that the modified image becomes a specific image size set for the training data, thereby generating a padded image, and inputs the padded image to the image segmentation engine.

[0612] In addition, for the first region label image output from the image segmentation engine, the segmentation processing unit 2804 trims only the area corresponding to the filled area to generate a second region label image. Thereafter, the segmentation processing unit 2804 reduces or enlarges the generated second region label image to the image size of the input image to generate a final region label image.

[0613] Therefore, even in the case where an input image has an image size that cannot be processed according to Example 8, the segmentation processing unit 2804 according to this example can generate a region label image through the image segmentation process by the image segmentation engine.

[0614] Next, refer to Fig.29 and Fig.37 A series of image processing operations according to this example is described. Fig.37 is a flowchart showing the image segmentation processing according to the present example. Note that the processing in step S2910, step S2920, and steps S2950 to S2970 according to the present example is the same as the processing in these steps in Example 8, so the description of the processing is omitted here. Note that in the case where the image segmentation processing is to be performed unconditionally on the input image with respect to the imaging conditions other than the image size, after the processing in step S2920 is performed, the processing in step S2930 may be omitted, and the processing may proceed to step S2940.

[0615] In step S2920, similar to Example 8, when the imaging condition obtaining unit 2802 obtains the imaging condition group of the input image, the process proceeds to step S2930. In step S2930, the processing possibility determination unit 2803 uses the obtained imaging condition group to determine whether the image segmentation engine can process the input image. Specifically, for the imaging condition of the input image, the processing possibility determination unit 2803 determines whether the image segmentation engine can process the imaging part, the imaging system, and the imaging angle of view. Unlike Example 8, the processing possibility determination unit 2803 does not perform determination regarding the image size.

[0616] The processing possibility determination unit 2803 performs determination regarding the imaging part, the imaging system, and the imaging angle of view, and if it is determined that the input image can be processed, the process proceeds to step S2940. On the other hand, in the case where the processing possibility determination unit 2803 determines that the image segmentation engine cannot process the input image based on these imaging conditions, the process proceeds to step S2970. Note that, depending on the setting and implementation form of the image processing device 2800, even if it is determined that the input image cannot be processed based on one or more of the imaging part, the imaging system, and the imaging angle of view, the image segmentation process in step S2940 may be performed.

[0617] When the process enters step S2940, Fig.37 The image segmentation processing according to this example is shown. In the image segmentation processing according to this example, first, in step S3710, the segmentation processing unit 2804 enlarges or reduces the input image to a specific resolution set for the training data, thereby generating a modified image.

[0618] Next, in step S3720, the segmentation processing unit 2804 fills the generated modified image so that the modified image becomes the image size set for the training data, thereby generating a filled image. At this time, for the area to be filled, the segmentation processing unit 2804 fills the area by using fixed pixel values ​​or using adjacent pixel values ​​or by mirror filling according to the characteristics of the learning model, thereby effectively performing image segmentation processing.

[0619] In step S3730, the segmentation processing unit 2804 inputs the filled image into the image segmentation engine, thereby obtaining a first region label image that has undergone image segmentation processing.

[0620] Next, in step S3740, for the first region label image, the segmentation processing unit 2804 trims only the region corresponding to the region filled in step S3720, thereby generating a second region label image.

[0621] Thereafter, in step S3750, the segmentation processing unit 2804 reduces or enlarges the second region label image to the image size of the input image to generate a final region label image. When the segmentation processing unit 2804 generates the final region label image in step S3750, the image segmentation processing according to this example ends, and the processing proceeds to step S2950. Since the processing from step S2950 is the same as the processing from step S2950 in Example 8, the description of the processing is omitted here.

[0622] As described above, according to the segmentation processing unit 2804 of this example, the image size of the input image is adjusted so that the resolution of the input image becomes a predetermined resolution. In addition, for the input image whose image size is adjusted, the segmentation processing unit 2804 generates a padded image that is padded so that the adjusted image size becomes an image size that can be processed by the image segmentation engine, and then inputs the padded image to the image segmentation engine. Thereafter, the segmentation processing unit 2804 trims the output image from the image segmentation engine to trim only the area corresponding to the padded area. Then, the segmentation processing unit 2804 adjusts the image size of the trimmed image to the original image size of the input image, thereby generating a regional label image.

[0623] Therefore, even in the case of an input image having an image size that cannot be processed according to Example 8, the segmentation processing unit 2804 according to this example can perform image segmentation processing on the input im...

Claims

1. A medical image processing device, comprising: an acquisition unit configured to acquire a tomographic image of an eye to be inspected; a first processing unit configured to perform a first detection process for outputting a detection result related to at least one of a plurality of layers in the obtained tomographic image as output data of a learning model by using the obtained tomographic image as input data of the learning model, wherein the learning model is obtained by using training data including data indicating information on at least one of a plurality of layers in the tomographic image of the eye to be inspected, and a second processing unit configured to, after performing the first detection processing, perform a second detection processing for detecting information on at least one layer among a plurality of layers in the obtained tomographic image without using a learning model obtained by machine learning, wherein the second detection processing is processing for detecting information on at least one layer that is not obtained in a detection result output by performing the first detection processing, The first detection processing is a processing for detecting a layer from a boundary between the inner limiting membrane and the nerve fiber layer of the eye to a photoreceptor inner segment-outer segment junction, a retinal pigment epithelium, a Bruch's membrane, a choroid, and a sclera, and The second detection process is a process of detecting at least one layer between the detected layers.

2. The medical image processing apparatus according to claim 1, wherein: The first detection process is a process of detecting a retinal area in the obtained tomographic image as the at least one layer, and The second detection process is a process of detecting at least one layer in the detected retinal area.

3. The medical image processing device according to claim 1 further includes a display control unit, which is configured to display on the display unit at least one of the processing results of the first detection processing and the second detection processing and the measurement results of the shape features of the eye to be inspected obtained based on the processing results of the first detection processing and the second detection processing.

4. The medical image processing apparatus according to claim 1, further comprising: a display control unit configured to display a processing result of the first detection processing on a display unit; as well as a modification unit configured to modify the information of at least one layer detected by the first detection process according to an instruction of an operator, Among them, the modified information is used for incremental learning of the learning model.

5. The medical image processing apparatus according to claim 3, wherein: The display control unit causes information indicating that a processing result of the first detection processing is a processing result detected by using a learning model to be displayed on the display unit.

6. The medical image processing apparatus according to claim 1, wherein: The first processing unit performs the first detection process by using a learning model selected based on the imaging condition related to the obtained tomographic image, among a plurality of learning models that have been machine-learned using different kinds of training data corresponding to a plurality of imaging conditions.

7. The medical image processing apparatus according to claim 1, further comprising: A generating unit is configured to generate an En-Face image corresponding to a depth range of at least a portion of a three-dimensional tomographic image of a subject's eye, the depth range being determined based on the detected at least one layer.

8. The medical image processing apparatus according to claim 7, wherein: The generation unit generates a motion contrast En-Face image corresponding to the determined depth range by using three-dimensional motion contrast data corresponding to the three-dimensional tomographic image.

9. The medical image processing apparatus according to claim 1, further comprising: an image quality improving unit configured to generate, from the acquired tomographic image, a tomographic image having improved image quality compared with the acquired tomographic image, by using a learning model for improving image quality, The first processing unit performs a first detection process on the generated tomographic image.

10. The medical image processing apparatus according to claim 1, further comprising: A diagnosis result generating unit is configured to generate a diagnosis result of the obtained tomographic image from a processing result of the first detection processing by using a learning model for diagnosis result generation.

11. The medical image processing apparatus according to claim 1, wherein: The first processing unit inputs, into the learning model, an image obtained by adjusting the obtained tomographic image to a state in which the learning model can operate using images of a plurality of regions obtained by dividing the obtained tomographic image.

12. The medical image processing apparatus according to claim 1, wherein: The first processing unit inputs, into the learning model, an image obtained by performing padding processing on the obtained tomographic image so that the image size of the obtained tomographic image becomes an image size that can be operated by the learning model.

13. The medical image processing apparatus according to claim 1, wherein: The medical image processing apparatus further includes an evaluation unit configured to evaluate the detected at least one layer as the region information by an evaluation engine that performs knowledge-based processing using anatomical knowledge or a learning model for evaluation.

14. The medical image processing apparatus according to claim 13, further comprising: A modification unit is configured to modify the region information by processing based on anatomical knowledge.

15. A medical image processing method, comprising: obtaining a tomographic image of the eye to be examined; performing a first detection process for outputting a detection result related to at least one of a plurality of layers in the obtained tomographic image as output data of the learning model by using the obtained tomographic image as input data of a learning model obtained by using training data including data indicating information on at least one of a plurality of layers in the tomographic image of the eye to be inspected, and After performing the first detection processing, performing a second detection processing for detecting information on at least one of a plurality of layers in the obtained tomographic image without using a learning model obtained by machine learning, wherein the second detection processing is processing for detecting information on at least one layer that is not obtained in a detection result output by performing the first detection processing, The first detection processing is a processing for detecting a layer from a boundary between the inner limiting membrane and the nerve fiber layer of the eye to a photoreceptor inner segment-outer segment junction, a retinal pigment epithelium, a Bruch's membrane, a choroid, and a sclera, and The second detection process is a process of detecting at least one layer between the detected layers. 16 . A non-transitory computer-readable medium storing a program which, when executed by a processor, causes the processor to perform the steps of the medical image processing method according to claim 15 .

Images