Medical image processing device, trained model, medical image processing method and program

Abstract

To provide a medical image processing device capable of detecting a region or a boundary of an object in a tomographic image in a short processing period.SOLUTION: A medical image processing device comprises: an acquisition unit for acquiring a tomographic image of an eye to be examined; and a processing unit for detecting at least one of a layer and a boundary in the acquired tomographic image. The processing unit generates a composite image by combining images of partial regions in the acquired tomographic image, detects at least one of the layer and the boundary in the composite image by using a learned model, and detects at least one of the layer and the boundary in the acquired tomographic image by using a detection result acquired using the learned model.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to a medical image processing device, a trained model, a medical image processing method, and a program. [Background technology]

[0002] Tomographic imaging devices for the eye, such as devices using optical coherence tomography (OCT devices), are capable of observing the state of the inside of the retina in three dimensions. These tomographic imaging devices have been attracting attention in recent years because they are useful for more accurate diagnosis of diseases.

[0003] One form of OCT is TD-OCT (Time Domain OCT), which combines a broadband light source with a Michelson interferometer. TD-OCT is configured to obtain information in the depth direction by scanning the delay of the reference arm to measure the interference light with the backscattered light of the signal arm. However, with TD-OCT, high-speed image acquisition is difficult due to the scanning of the delay of the reference arm.

[0004] Therefore, as a method for acquiring images at higher speeds, Spectral Domain OCT (SD-OCT) is known, which uses a broadband light source and acquires an interferogram with a spectrometer. Also, Swept Source OCT (SS-OCT) is known, which uses a high-speed wavelength swept light source as a light source and measures the spectral interference with a single channel photodetector.

[0005] When a tomographic image is acquired using OCT, if the thickness of the nerve fiber layer can be measured, the progression of a disease such as glaucoma and the degree of recovery after treatment can be quantitatively diagnosed. In this regard, Patent Document 1 discloses a technology for quantitatively measuring the thickness of each retinal layer by detecting the region of each retinal layer from a tomographic image using a trained model and measuring the thickness of each layer. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2020-93076 A Summary of the Invention [Problem to be solved by the invention]

[0007] However, in the technology described in Patent Document 1, the retinal layer area is detected using a trained model for the entire area of ​​a tomographic image captured by OCT. Therefore, even unimportant areas of the tomographic image are processed, which places an extra load on the data transfer and preparation and the detection process of the retinal layer area, and takes a lot of processing time.

[0008] Therefore, in one embodiment of the present invention, one of the objectives is to provide a medical image processing device, a trained model, a medical image processing method, and a program that can detect the area or boundary of an object in a tomographic image in a short processing time. [Means for solving the problem]

[0009] A medical image processing apparatus according to an embodiment of the present invention includes an acquisition unit that acquires a tomographic image of a subject's eye, and a processing unit that detects at least one of a layer and a boundary in the acquired tomographic image. By the acquisition department Images of multiple partial regions in the acquired tomographic image Lined up A composite image is generated by synthesizing the layers and the boundaries in the composite image, and at least one of a layer and a boundary is detected in the composite image by using a trained model. At least one of a layer and a boundary is detected in the acquired tomographic image by using a detection result obtained by using the trained model. death , Each of the plurality of partial regions is a region of interest including a retina region and a choroid. . Effect of the Invention

[0010] According to one embodiment of the present invention, it is possible to detect an object region or boundary in a tomographic image in a short processing time.

[0011] Further features of the invention will become apparent from the following description of exemplary embodiments with reference to the accompanying drawings. [Brief description of the drawings]

[0012] [Figure 1] 1 shows an example of a schematic configuration of an image processing system according to a first embodiment. [Diagram 2] 2A to 2C are diagrams for explaining the structure of an eye, a tomographic image, and a fundus image. [Diagram 3] FIG. 4 is a diagram for explaining detection processing of an attention area according to the first embodiment. [Figure 4] FIG. 4 is a diagram for explaining packing processing of a region of interest according to the first embodiment; [Diagram 5] FIG. 4 is a diagram for explaining an unpacking process according to the first embodiment. [Figure 6] 10A and 10B are diagrams for explaining an example of learning data for area detection processing; [Figure 7] 13 is a diagram for explaining another example of learning data for area detection processing. FIG. [Figure 8] 4 illustrates an example of learning data according to the first embodiment. [Figure 9] FIG. 2 is a diagram for explaining an example of learning data according to the first embodiment. [Figure 10] FIG. 2 is a diagram for explaining an example of a machine learning model according to the first embodiment. [Figure 11] 4 is a flowchart of a series of processes according to the first embodiment. [Figure 12] 3 shows an example of a display screen according to the first embodiment. [Figure 13] 6A to 6C are diagrams for explaining the results of post-processing according to the first embodiment. [Figure 14] FIG. 11 is a diagram for explaining an example of detection of a boundary line according to a modified example of the first embodiment. [Figure 15] 13A to 13C are diagrams for explaining another example of boundary detection according to the modified example of the first embodiment. [Figure 16] 13 is a flowchart of a retinal layer detection process according to the second embodiment. [Figure 17] FIG. 11 is a diagram illustrating an example of a stripe division process according to the second embodiment. [Figure 18] 13A to 13C are diagrams for explaining packing processing and unpacking processing of a region of interest according to the second embodiment; [Figure 19] 13A to 13C are diagrams illustrating an example of stripe concatenation processing and boundary detection processing according to the second embodiment. [Figure 20] 13A to 13C are diagrams illustrating an example of a process for creating stripes having overlapping regions according to a modified example of the second embodiment. [Figure 21] 13 is a flowchart of a series of processes according to the third embodiment. [Figure 22] 13A to 13C are diagrams illustrating an example of a reconstruction process of a tomographic image according to the third embodiment. [Diagram 23] 13 is a flowchart of a retinal layer detection process according to the fourth embodiment. [Figure 24] FIG. 13 is a diagram illustrating an example of a processing result according to the fourth embodiment. [Diagram 25] 13 is a flowchart of a retinal layer detection process according to the sixth and seventh embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, exemplary embodiments for carrying out the present invention will be described in detail with reference to the drawings. However, the dimensions, materials, shapes, and relative positions of components described in the following embodiments are arbitrary and can be changed according to the configuration of an apparatus to which the present invention is applied or various conditions. In addition, in the drawings, the same reference numerals are used between the drawings to indicate elements that are the same or functionally similar.

[0014] In the following, a machine learning model refers to a learning model based on a machine learning algorithm such as deep learning. A trained model is a model that has been trained (learned) in advance using appropriate teacher data (learning data) for a machine learning model based on an arbitrary machine learning algorithm. However, a trained model is not one that does not undergo further learning, but rather is capable of undergoing additional learning.

[0015] (First embodiment) Hereinafter, a medical image processing system and a medical image processing method including a medical image processing device for processing a tomographic image of an eye according to a first embodiment of the present invention will be described with reference to Figs. 1 to 13. In the retinal layer region detection process according to this embodiment, a trained model related to a machine learning model is used to perform region detection (segmentation) process of the retinal layer in a region of interest including a retinal region. Here, in this embodiment, for input data for the region detection process, a region of interest is extracted from a tomographic image, and a composite image in which data of the extracted multiple regions of interest are compiled is used. As a result, according to this embodiment, the data to be transferred and the data to be subjected to the region detection process can be reduced, the efficiency of region detection can be improved, and processing can be accelerated.

[0016] Fig. 1 shows an example of a schematic configuration of an image processing system including an image processing device (medical image processing device) according to the present embodiment. As shown in Fig. 1, the image processing system includes an OCT device 10, which is an example of a tomographic image capturing device, an image processing device 20, a fundus image capturing device 30, an external storage device 40, a display unit 50, and an input unit 60.

[0017] The OCT device 10 is an example of a tomographic imaging device that is a device for capturing a tomographic image of a subject's eye. As the OCT device, any type of OCT device can be used, such as an SD-OCT device or an SS-OCT device.

[0018] The image processing device 20 is connected to the OCT device 10, the fundus image capturing device 30, the external storage device 40, the display unit 50, and the input unit 60 via an interface, and can control these. The image processing device 20 can generate various images such as a tomographic image and an En-Face image (front image) of the subject's eye based on various signals acquired from the OCT device 10, the fundus image capturing device 30, and the external storage device 40. The image processing device 20 can also perform image processing on these images.

[0019] The image processing device 20 may be configured by a general-purpose computer, or may be configured by a dedicated computer for an image processing system. The image processing device 20 includes a processor (not shown) and a storage medium including a memory such as an optical disk or a read-only memory (ROM). The processor may be a central processing unit (CPU) or a micro processing unit (MPU). The processor is not limited to a CPU or an MPU, and may be a graphics processing unit (GPU) or a field-programmable gate array (FPGA).

[0020] The fundus image capturing device 30 is a device for capturing a fundus image of the subject's eye, and as the device, for example, a fundus camera, a scanning laser ophthalmoscope (SLO), etc. may be used. The OCT device 10 and the fundus image capturing device 30 may be configured as an integrated device or as separate devices.

[0021] The external storage device 40 stores information about the subject's eye (such as the patient's name, age, and sex), various captured image data, shooting parameters, image analysis parameters, and parameters set by the operator, each associated with each other. The external storage device 40 may be configured by any storage device, and may be configured by a storage medium such as an optical disk or a memory.

[0022] The display unit 50 is configured by any display, and can display information about the subject's eye, various images, etc., under the control of the image processing device 20. The input unit 60 is, for example, a mouse, a keyboard, or a touch display, and the operator can input instructions to the image processing device 20, the OCT device 10, and the fundus image capturing device 30, via the input unit 60, to the image processing device 20.

[0023] Although these components are shown as separate entities in Fig. 1, some or all of these components may be configured as an integrated unit. For example, when the input unit 60 is a touch display, the input unit 60 can be configured as an integrated unit with the display unit 50.

[0024] Next, the OCT device 10 will be described. The OCT device 10 is provided with a light source 11, a galvanometer mirror 12, a focus lens stage 13, a coherence gate stage 14, a detector 15, and an internal fixation lamp 16. Note that since the OCT device 10 is a known device, a detailed description thereof will be omitted, and the following describes the capturing of a tomographic image performed in response to an instruction from an image processing device 20.

[0025] When an instruction to capture an image is transmitted from the image processing device 20, the light source 11 emits light. The light from the light source 11 is split into measurement light and reference light by a splitting unit (not shown). The OCT device 10 irradiates the subject (subject's eye) with the measurement light and detects interference light between the return light from the subject and the reference light, thereby making it possible to obtain an interference signal including tomographic information of the subject.

[0026] The galvanometer mirror 12 is an example of a scanning unit used to scan the fundus of the subject's eye with the measurement light, and the imaging range of the fundus by OCT imaging can be determined by the scanning range of the measurement light by the galvanometer mirror 12. The image processing device 20 can determine the imaging range in the planar direction of the fundus and the number of scanning lines (scanning speed in the planar direction) by controlling the driving range and speed of the galvanometer mirror 12. In FIG. 1, the galvanometer mirror 12 is shown as one unit for the sake of simplicity of explanation, but the galvanometer mirror 12 is actually composed of a mirror for X-scanning and two mirrors for Y-scanning, and can scan a desired range on the fundus with the measurement light. Note that the configuration of the scanning unit for scanning the measurement light is not limited to the galvanometer mirror, and any other deflection mirror can be used. In addition, as the scanning unit, for example, a deflection mirror that can scan the measurement light in a two-dimensional direction with one mirror such as a MEMS mirror may be used.

[0027] The focus lens stage 13 is provided with a focus lens (not shown). By moving the focus lens stage 13, the focus lens can be moved along the optical axis of the measurement light. Therefore, the measurement light can be focused on the retinal layer of the fundus via the anterior segment of the subject's eye by the focus lens. The measurement light irradiated to the fundus is reflected and scattered by each retinal layer, and travels back along the optical path as return light.

[0028] The coherence gate stage 14 is used to adjust the length of the optical path of the reference light or the measurement light to accommodate differences in axial length of the subject's eye. In this embodiment, the coherence gate stage 14 is configured by a stage provided with a mirror, and can correspond to the optical path length of the reference light to the optical path length of the measurement light by moving in the optical axis direction in the optical path of the reference light. Here, the position of the coherence gate represents the position where the optical distances of the measurement light and the reference light in OCT are equal. The coherence gate stage 14 can be controlled by the image processing device 20. The image processing device 20 can control the imaging range in the depth direction of the subject's eye by controlling the position of the coherence gate by the coherence gate stage 14, and can control imaging of the retinal layer side or imaging of a deeper side than the retinal layer.

[0029] The detector 15 detects interference light between the return light of the measurement light from the subject's eye and the reference light, which occurs in an interference unit (not shown), and generates an interference signal. The image processing device 20 acquires the interference signal from the detector 15 and performs Fourier transform or the like on the interference signal to generate a tomographic image of the subject's eye.

[0030] The internal fixation light 16 is provided with a display unit 161 and a lens 162. In this embodiment, a display unit 161 having a plurality of light emitting diodes (LD) arranged in a matrix is ​​used as an example of the display unit 161. The lighting positions of the light emitting diodes are changed according to the part to be photographed under the control of the image processing device 20. The light from the display unit 161 is guided to the subject's eye via the lens 162. The light emitted from the display unit 161 has a wavelength of, for example, 520 nm, and is displayed in a desired pattern under the control of the image processing device 20.

[0031] It should be noted that the OCT device 10 may include other components. For example, the OCT device 10 may further include a drive control unit for the OCT device 10 that controls the drive of each component based on the control by the image processing device 20, a shutter that is driven to block the measurement light, and the like.

[0032] Next, the structure of the eye and images acquired by the image processing system will be described with reference to Fig. 2(a) to Fig. 2(c). Fig. 2(a) is a schematic diagram of an eyeball. Fig. 2(a) shows a cornea C, a lens CL, a vitreous body V, a macular region M (the center of the macula represents the fovea), and an optic disc D. In this embodiment, a case where the posterior pole of the retina including the vitreous body V, the macular region M, and the optic disc D is mainly described. Although not described below, the OCT device 10 is also capable of photographing anterior segments such as the cornea and the lens.

[0033] FIG. 2(b) shows an example of a tomographic image obtained by photographing a retina using the OCT device 10. In FIG. 2(b), AS indicates an image unit obtained by one A-scan. Here, the A-scan refers to obtaining tomographic information in the depth direction at one point of the subject's eye by the above-mentioned series of operations of the OCT device 10. In addition, performing A-scans multiple times in any transverse direction (main scanning direction) to obtain two-dimensional tomographic information in the transverse direction and depth direction of the subject's eye is called a B-scan. A single B-scan image can be constructed by collecting multiple A-scan images obtained by A-scans. Hereinafter, this B-scan image will be referred to as a two-dimensional tomographic image. In the tomographic image in this example, the horizontal axis (main scanning direction of OCT) is the X-axis, and the vertical axis (depth direction) is the Z-axis.

[0034] Furthermore, by repeating scanning in the main scanning direction while moving the scanning position in a predetermined direction (sub-scanning direction) by the scanning unit with respect to the imaging range on the subject's eye, multiple B-scan images can be obtained. For example, by repeating B-scanning of the XZ plane while moving the scanning position in the Y direction, three-dimensional information in the XYZ space can be obtained. Data composed of the multiple obtained B-scan images is called volume data. The image processing device 20 can generate a three-dimensional tomographic image of the subject's eye from this volume data. Such a scanning method is called C-scan, and the obtained three-dimensional tomographic image is called a C-scan image. The image processing device 20 can also obtain an En-Face image (front image) of the fundus by projecting or integrating information in at least a part of the depth range of the three-dimensional information.

[0035] In Fig. 2(b), blood vessels Ve, vitreous body V, macula M, and optic nerve head D are shown. Boundary line L1 represents the internal limiting membrane (ILM), boundary line L2 represents the boundary between the nerve fiber layer (NFL) and ganglion cell layer (GCL), boundary line L3 represents the boundary between the intimal layer (IPL) and inner nuclear layer (INL), boundary line L4 represents the retinal pigment epithelium layer (RPE), boundary line L5 represents Bruch's membrane (BM), and boundary line L6 represents the choroid.

[0036] Fig. 2(c) shows an example of a fundus image acquired by photographing the fundus of the examinee's eye using the fundus image photographing device 30. Fig. 2(c) shows the macular region M and optic disc D, and retinal blood vessels are shown by thick curves. In the fundus image, the horizontal axis (main scanning direction of OCT) is the X-axis, and the vertical axis (sub-scanning direction of OCT) is the Y-axis.

[0037] Next, the image processing device 20 will be described with reference to Fig. 1 again. The image processing device 20 includes an acquisition unit 21, an image processing unit 22, a drive control unit 23, a storage unit 24, and a display control unit 25.

[0038] The acquiring unit 21 can acquire data of the interference signal of the subject's eye from the OCT device 10. The interference signal data acquired by the acquiring unit 21 may be an analog signal or a digital signal. When the acquiring unit 21 acquires an analog signal, the image processing device 20 can convert the analog signal into a digital signal. The acquiring unit 21 can acquire various images such as tomographic data generated by the image processing unit 22, tomographic images, and En-Face images. Here, the tomographic data refers to data including information about the tomography of the subject, including data based on the interference signal by OCT, and data obtained by performing a fast Fourier transform (FFT) or any signal processing on the data.

[0039] Furthermore, the acquisition unit 21 acquires a group of imaging conditions for the tomographic image to be image-processed (for example, imaging date and time, name of imaging part, imaging area, imaging angle of view, imaging method, image resolution and gradation, image pixel size, image filter, information on image data format, etc.). Note that the group of imaging conditions is not limited to those exemplified. Also, the group of imaging conditions does not need to include all of the exemplified ones, and may include some of them.

[0040] The acquisition unit 21 can also acquire data including fundus information acquired by the fundus image photographing device 30, a front image of the fundus, etc. The acquisition unit 21 can also acquire information for identifying the subject's eye, such as a subject identification number, from the input unit 60, the external storage device 40, etc. The acquisition unit 21 can also acquire various images stored in the external storage device 40 or an external device (not shown), information obtained using other photographing devices, etc. The acquisition unit 21 can store the various acquired data and images in the storage unit 24.

[0041] The image processing unit 22 can generate a tomographic image, an En-Face image, or the like from the data acquired by the acquisition unit 21 or the data stored in the storage unit 24, and can perform image processing on the generated or acquired image. The image processing unit 22 is provided with a tomographic image generation unit 221 and a processing unit 222.

[0042] The tomographic image generating unit 221 can generate tomographic data by performing processing such as Fourier transform on the interference signal acquired by the acquiring unit 21, and generate a tomographic image based on the tomographic data. Note that any known method may be adopted as a method for generating a tomographic image, and detailed description thereof will be omitted.

[0043] The processing unit 222 performs a process of detecting a region of interest in the tomographic image. In this embodiment, the processing unit 222 detects a region including a retina region and a choroid as a region of interest. The processing unit 222 also performs a packing process of generating a composite image in which images of at least two of the detected regions of interest are arranged vertically (in the depth direction of the subject's eye). Furthermore, the processing unit 222 performs an unpacking process of dividing the images of the regions of interest from the composite image and returning the divided images of the regions of interest to a tomographic image. Details of these processing methods will be described later.

[0044] Furthermore, the processing unit 222 includes a trained model for a machine learning model based on a machine learning algorithm such as deep learning. The processing unit 222 uses the trained model to perform segmentation processing for detecting a layer structure and layer boundaries in the retina region of the subject's eye in the tomographic image. The details of the machine learning model and the segmentation processing will be described later.

[0045] The drive control unit 23 can control the driving of each component of the OCT device 10 and the fundus image capturing device 30 connected to the image processing device 20. The storage unit 24 can store various images and data such as the tomographic data acquired by the acquisition unit 21 and the tomographic images generated and processed by the image processing unit 22. The storage unit 24 can also store programs and the like for performing the functions of each component of the image processing device 20 by being executed by the processor.

[0046] The display control unit 25 can control the display on the display unit 50 of various information acquired by the acquisition unit 21, various images such as tomographic images generated and processed by the image processing unit 22, and information input by the operator.

[0047] Each component other than the storage unit 24 in the above-described image processing device 20 may be configured by a software module executed by a processor such as a CPU, MPU, or GPU. Moreover, each component may be configured by a circuit performing a specific function such as an ASIC, or an independent device. The storage unit 24 may be configured by any storage medium such as an optical disk or a memory.

[0048] Next, various processes by the processing unit 222 will be described with reference to Fig. 3(a) to Fig. 10. As described above, in the conventional retinal layer detection process using a trained model, the entire area of ​​the tomographic image is processed, so that extra load is placed on the transfer and preparation of data and the area detection process of the retinal layer, and a lot of processing time is required. Therefore, in this embodiment, an area of ​​interest where the area detection process of the retinal layer is to be performed is extracted from the tomographic image, and a packing process is performed to combine the images of the extracted multiple areas of interest to generate a composite image, and the retinal layer is detected from the composite image using the trained model.

[0049] (Region of interest detection) First, the detection process of the region of interest according to this embodiment will be described with reference to Fig. 3(a) to Fig. 3(e). As described above, in this embodiment, a region including the retina region and the choroid in the tomographic image is detected as the region of interest. When the ratio of the region of interest to the entire region of the input tomographic image is small, there are many regions without useful information and regions that do not require the process of detecting the region of the retina layer. If the region detection process is performed on such regions, extra processing time will be generated, and the overall processing time will be increased. Therefore, by detecting the region of interest (the region where the retina layer exists) in advance, the region of the retina layer can be detected efficiently.

[0050] The processing unit 222 performs binarization processing on the tomographic image as the region of interest detection processing. FIG. 3(a) shows an example of the tomographic image to be processed. First, the processing unit 222 performs Gaussian filtering processing on the tomographic image. FIG. 3(b) shows an example of the tomographic image subjected to the Gaussian filtering processing. Note that in this embodiment, the kernel size of the Gaussian filter is 3×3, but the kernel size of the Gaussian filter is not limited to this and may be, for example, 5×5 or other sizes. Furthermore, other filters may be used as long as they are noise reduction filters such as a Gaussian filter.

[0051] Next, the processing unit 222 performs a normalization process on the tomographic image that has been subjected to the filtering process. Here, the processing unit 222 normalizes the pixel values ​​of the tomographic image in the range of 0 to 255. Furthermore, the processing unit 222 performs a binarization process on the normalized image to generate a mask image. FIG. 3(c) shows an example of the generated mask image. In the mask image generation process, if the pixel value of the normalized tomographic image is equal to or greater than a threshold value, the processing unit 222 sets the pixel value of the mask image corresponding to the position of the pixel to 1. Also, if the pixel value of the tomographic image is less than the threshold value, the processing unit 222 sets the pixel value of the mask image corresponding to the position of the pixel to 0. Here, the threshold value is set to 50. However, the threshold value may be determined based on experience as in this case, or may be dynamically determined by statistical processing of the tomographic image, for example, based on the noise level of the tomographic image. Note that the method of binarization based on the noise level of the tomographic image is not limited to the above method, and may be performed using any known method. The processing unit 222 sets a region in the mask image generated by the binarization process, in which the pixel value is 1, as a region of interest.

[0052] Thereafter, the processing unit 222 sets a mask image of the smallest rectangle surrounding the region of interest in order to cut out the region of interest in the tomographic image to be processed. FIG. 3(d) shows an example of the set mask image. The processing unit 222 uses the set mask image to extract the region of interest from the tomographic image to be processed. Furthermore, when extracting the image of the region of interest from the tomographic image, the processing unit 222 stores information on the tomographic image from which the extraction was performed and position information of the extracted region of interest in the tomographic image in the storage unit 24. FIG. 3(e) shows an example of an image of the extracted region of interest.

[0053] In this embodiment, the attention area detection process is performed using a binarization process. However, the method of the attention area detection process is not limited to the binarization process, and for example, rule-based area detection may be used. Here, the rule-based process refers to a process that uses a known regularity, and the rule-based area detection refers to an area detection process that uses a known regularity such as the regularity of the shape of the retina.

[0054] (Packing and unpacking) Next, the packing process and the unpacking process according to this embodiment will be described with reference to Fig. 4 and Fig. 5. Fig. 4 shows an outline of the packing process according to this embodiment, and Fig. 5 shows an outline of the unpacking process according to this embodiment.

[0055] As shown in Fig. 4, the processing unit 222 performs packing processing by vertically arranging (packing) images of a plurality of regions of interest 406-409 extracted from a plurality of tomographic images 401-404 in one image (depth direction of the subject's eye) to generate a composite image 405. Since the generated composite image 405 is an input image to a trained model described later, the image size of the composite image 405 can be an image size that can be processed by the trained model. In this embodiment, the image size of the composite image 405 can be the same as the image size of an image used as input data for training data related to the trained model.

[0056] Here, the size of the attention region in the depth direction (Z direction) changes depending on the state of the retina of the subject's eye. Therefore, the processing unit 222 can vary the number of images of the attention region packed into the composite image 405. Here, the processing unit 222 according to this embodiment is configured to pack as many images of the attention region as possible into the composite image 405.

[0057] In addition, as an unpacking process, the processing unit 222 unpacks (divides) the label image 505 obtained by the segmentation process using the trained model into images of the respective regions of interest, and overwrites the images of the respective divided regions onto the regions of interest of the original tomographic image. At this time, the processing unit 222 specifies the position of the region of interest in the corresponding tomographic image using the information and position information of the tomographic image corresponding to each region of interest stored in the storage unit 24, and returns the images of the divided regions of interest to the positions of the regions of interest in the corresponding tomographic image. By the unpacking process, the processing unit 222 can generate detection result images 501 to 504 corresponding to the tomographic images 401 to 404 to be processed, as shown in FIG. 5.

[0058] (Learning data) Next, learning of the machine learning model according to this embodiment will be described with reference to Figs. 6(a) to 9(c). The learning data of the machine learning model is composed of a pair group of one or more input data and output data. In the learning data according to this embodiment, a composite image obtained by packing images of the region of interest in a plurality of tomographic images is used as input data, and a label image corresponding to the composite image is used as output data. Here, the label image is an image in which a label is assigned to each pixel, and in this embodiment, it refers to an image in which a label is assigned to each pixel regarding the image appearing (photographed) at the pixel. In this embodiment, the label image with a label assigned to each region is used as output data of the learning data to perform region detection, but it is not limited to the region, and a boundary image indicating a boundary or with a label assigned to each boundary may be used as output data of the learning data to perform boundary detection.

[0059] 6(a) and 6(b) are diagrams for explaining an example of learning data for region detection processing. However, in FIG. 6(a) and FIG. 6(b), for the sake of simplicity, a tomographic image and a label image corresponding to the tomographic image are shown as an example of learning data. FIG. 6(a) shows an example of a tomographic image 601 acquired by OCT imaging. FIG. 6(b) shows a label image 602 in which an arbitrary layer is given a label from the tomographic image 601. Hereinafter, an example of learning data constituted by a pair group of the tomographic image 601 and the label image 602 will be explained.

[0060] In the label image 602, the label between the boundary line L1 and the boundary line L2 is the NFL region 603. The label between the boundary line L2 and the boundary line L3 is the GCL+IPL region 604. The label between the boundary line L3 and the boundary line L4 is the INL+OPL+ONL+photoreceptor layer region 605. The label of the region below the boundary line L4 is the RPE layer region 606. Note that as other regions, the ganglion cell layer (GCL), inner plexiform layer (IPL), inner nuclear layer (INL), outer plexiform layer (OPL), outer nuclear layer (ONL), photoreceptor layer, and retinal pigment epithelium layer (RPE) may be shown as individual regions or regions of a collection of multiple layers.

[0061] Here, for example, by not including a region label for the choroid in the label image used in the learning data, the label for the choroid region can be omitted from the label image output from the trained model. In this case, the trained model can be used to perform segmentation processing that appropriately distinguishes between the choroid and retina regions.

[0062] The learning data may be configured with a group of multiple output data for one input data. For example, the tomographic image 601 may be used as input data, and the label images 707 to 710 that identify the respective regions shown in Fig. 7(a) to Fig. 7(d) may be used as output data.

[0063] The labeled image 602 used as output data may be an image in which a doctor or the like has labeled a region in a tomographic image, or an image in which a region has been detected by a rule-based region detection process. Furthermore, the labeled image 602 used as output data may be an image in which a doctor or the like has corrected an image in which a region has been detected by a rule-based region detection process. However, if machine learning is performed using an image in which region detection has not been performed appropriately as output data of training data, the image output by the trained model trained using the training data may also be an image in which region detection has not been performed appropriately. Therefore, by removing pairs including such region images from the training data, the possibility that the trained model will generate an inappropriate region image can be reduced.

[0064] As described above, a boundary image may be used as output data of the learning data. In this case, the boundary image used as the output data may be an image in which a doctor or the like indicates a boundary in a tomographic image or a boundary image in which a label is attached to the boundary, or may be a boundary image in which a boundary is detected by a rule-based region detection process. In this case, the boundary image used as the output data may also be an image in which a doctor or the like has modified an image in which a region is detected by a rule-based region detection process.

[0065] Further, in FIG. 6(a) and FIG. 6(b), an example of one XZ cross section in the XY plane of the retina is shown, but the cross section is not limited to this. Although not shown, any multiple XZ cross sections in the XY plane can be learned in advance so as to be able to handle cross sections captured with various different scan patterns such as raster scan and radial scan. For example, when using data such as a three-dimensional tomographic image obtained by three-dimensionally capturing the retina with raster scan, volume data in which adjacent multiple tomographic images are aligned can be used as learning data. In this case, it is possible to generate a pair image group at any angle from one piece of volume data and one corresponding piece of three-dimensional label data (three-dimensional label image). In addition, the machine learning model may be trained using images actually captured with various scan patterns as learning data.

[0066] In the above, for the sake of simplicity, a label image corresponding to a tomographic image is used as an example of learning data. In this embodiment, a composite image generated by packing a plurality of regions of interest and a label image corresponding to the composite image are used as learning data. Here, FIG. 8 shows an example of a composite image 801 used as input data for learning data according to this embodiment, and a label image 802 corresponding to the composite image 801. The method of generating a label image corresponding to the composite image according to this embodiment may be the same as the above-mentioned method of generating a label image. In addition, the composite image used as learning data is not limited to one in which the regions of interest of a tomographic image related to a cross section in the XZ plane are packed, and images captured in a cross section in the YZ plane or with various scan patterns may be used as learning data.

[0067] Here, as described above, the size of the attention area in the depth direction (Z direction) changes depending on the state of the retina of the subject's eye, so the processing unit 222 according to this embodiment is configured to pack as many images of the attention area as possible into the composite image. Correspondingly, the number of attention areas included in the composite image used as input data of the learning data can also be made variable. Therefore, as the composite image 801 used for the learning data, a composite image with various numbers of attention areas greater than or equal to two can be used. For example, a group of composite images used as output data of the trained model may include a composite image including two attention areas, a composite image including three attention areas, and a composite image including four attention areas. Note that the label image used as output data of the learning data may be a label image corresponding to the composite image including the various numbers of attention areas.

[0068] Next, images used as learning data will be described. In this embodiment, a group of pairs of a composite image 801 and a label image 802 constituting the learning data of the machine learning model is composed of pairs of rectangular area images of a certain image size that correspond in positional relationship. Hereinafter, the image pairs will be described with reference to Figs. 9(a) to 9(c).

[0069] First, a case will be described where one of the pairs constituting the learning data is a composite image 801 and a label image 802. In this case, as shown in Fig. 9(a), a pair is formed with a rectangular area image 901, which is the entire composite image 801, as input data, and a rectangular area image 902, which is the entire label image 802, as output data. Note that in the example shown in Fig. 9(a), a pair of input data and output data is formed by the entirety of each image, but the pairs are not limited to this.

[0070] 9(b), a pair may be formed by using a rectangular area image 911 in a composite image 801 as input data and a rectangular area image 913 at a corresponding pixel position in a label image 802 as output data. The rectangular area images 911 and 913 are based on A-scan units. The A-scan unit may be a single A-scan unit or a number of A-scan units.

[0071] In FIG. 9(b), the A-scan unit is used as a base, but the rectangular region may not cover the entire depth direction of the image, and may have portions outside the rectangular region above and below. That is, the horizontal size of the rectangular region may be set to be equal to several A-scans, and the depth size of the rectangular region may be set to be smaller than the depth size of the image. For example, as shown in FIG. 9(c), a pair may be formed by using a rectangular region image 921 of a composite image 801 as input data and a rectangular region image 923 of a corresponding pixel position in a label image 802 as output data. In this case, the size of the rectangular region may be set to include multiple labels within one rectangular region.

[0072] During learning, the scan range (photographing angle of view) and scan density (number of A-scans) are normalized to make the image sizes uniform, so that the rectangular region sizes during learning can be made constant. The rectangular region images shown in Figures 9(a) to 9(c) are examples of rectangular region sizes when learning separately.

[0073] The number of rectangular regions can be set to one in the example shown in FIG. 9(a), and to multiple in the examples shown in FIG. 9(b) and FIG. 9(c). For example, in the example shown in FIG. 9(b), a pair can be formed with a rectangular region image 912 in the composite image 801 as input data and a rectangular region image 914 at a corresponding pixel position in the label image 802 as output data. Also, for example, in the example shown in FIG. 9(c), a pair can be formed with a rectangular region image 922 in the composite image 801 as input data and a rectangular region image 924 at a corresponding pixel position in the label image 802 as output data. In this way, pairs of mutually different rectangular region images can be created from each pair of composite images and label images. Note that the group of pairs constituting the learning data can be enriched by creating a large number of pairs of rectangular region images while changing the positions of the regions in the original composite image and label image to different coordinates.

[0074] In the examples shown in FIG. 9(b) and FIG. 9(c), the rectangular regions are shown discretely, but in reality, the original composite image and the label image can be divided into a group of rectangular region images of a certain image size that are continuous without gaps. The original composite image and the label image may also be divided into a group of rectangular region images at random positions that correspond to each other. In this way, by selecting an image of a smaller region as a rectangular region (or a strip region) as a pair of input data and output data, a lot of pair data can be generated from the composite image 801 and the label image 802 that constitute the original pair. Therefore, the time required for training the machine learning model can be reduced. On the other hand, in a completed machine learning model that has been trained, the time required for the segmentation process to be performed tends to be long. Here, the segmentation process refers to a process of identifying and distinguishing regions and boundaries within an image.

[0075] As an example of training data of another machine learning model, a pair group of a synthetic image 801 and a label image to which only a label indicating each region is added to the synthetic image 801, such as the label images 707 to 710, may be used. Furthermore, training data may be created by combining such label-based training data with a method of creating training data in a region unit in the synthetic image as shown in FIG. 9(b) and FIG. 9(c), and used as the training data. For example, as a pair group of training data, training data in which an image of a rectangular region in the synthetic image 801 shown in FIG. 8 is used as input data, and images of a plurality of label units (not shown) corresponding to the image are used as output data may be used.

[0076] (Machine learning model configuration) Next, as an example of the machine learning model according to the present embodiment, a convolutional neural network (CNN) that performs a segmentation process on an input composite image will be described with reference to Fig. 10. Fig. 10 shows an example of a configuration 1001 of a machine learning model related to a trained model used by the processing unit 222.

[0077] The machine learning model shown in Fig. 10 is composed of a plurality of layers that process and output a group of input values. The types of layers included in the configuration 1001 of the machine learning model include a convolution layer, a downsampling layer, an upsampling layer, and a merger layer.

[0078] The convolution layer is a layer that performs convolution processing on a group of input values ​​according to parameters such as the set filter kernel size, the number of filters, the stride value, the dilation value, etc. The number of dimensions of the filter kernel size may also be changed according to the number of dimensions of the input image.

[0079] The downsampling layer is a layer that performs processing to reduce the number of output value groups to be less than the number of input value groups by thinning out or combining input value groups. Specifically, an example of such processing is Max Pooling processing.

[0080] The upsampling layer is a layer that performs a process to make the number of output values ​​larger than the number of input values ​​by duplicating the input values ​​or adding values ​​interpolated from the input values, for example, a linear interpolation process.

[0081] A synthesis layer is a layer that inputs a group of values, such as a group of output values ​​of a layer or a group of pixel values ​​that make up an image, from multiple sources and performs processing to synthesize them by concatenating or adding them.

[0082] In addition, as parameters set in the convolution layer group included in the configuration 1001 shown in Fig. 10, for example, a filter kernel size of 3 pixels wide and 3 pixels high, and the number of filters is 64, making it possible to perform segmentation processing with a certain degree of accuracy. However, it should be noted that if the parameter settings for the layers and nodes constituting the neural network are different, the degree to which the tendency trained from the learning data can be reproduced in the output data may differ. In other words, in many cases, appropriate parameters differ depending on the form of implementation, and can be changed to preferred values ​​as necessary.

[0083] In addition to the above-mentioned method of changing parameters, the configuration of the CNN may be changed to obtain better characteristics of the CNN. Better characteristics include, for example, high accuracy of the segmentation process, short time for the segmentation process, and short time required for training the machine learning model. Although not shown, as an example of changing the configuration of the CNN, for example, a batch normalization layer or an activation layer using a rectifier linear unit may be incorporated after the convolution layer.

[0084] When data is input to a trained model of such a machine learning model, data is output according to the design of the machine learning model, for example, output data that is likely to correspond to the input data according to the trends trained using the learning data.

[0085] (Area detection processing) In the trained model according to this embodiment, when a composite image 801 is input, a labeled image 802 is output according to a tendency trained using the training data. The processing unit 222 performs an unpacking process on the labeled image 802, thereby obtaining labeled images corresponding to the tomographic images used in generating the composite image as shown in FIG. 5. The processing unit 222 can detect the area of ​​the retinal layer in the tomographic image to be processed based on the obtained labeled images. Note that, when a boundary image showing a boundary or in which a label is attached to each boundary is used as output data of the training data, the processing unit 222 can detect the boundary of the retinal layer in the tomographic image to be processed.

[0086] In addition, as shown in Figures 9(b) and 9(c), when learning is performed by dividing an image region, the processing unit 222 divides a composite image obtained by packing a plurality of tomographic images into rectangular region images of an image size corresponding to the learning data, and inputs the rectangular region images to the learned model. The learned model outputs rectangular region images that are label images corresponding to each rectangular region. This allows the processing unit 222 to detect the region of the retinal layer in each rectangular region.

[0087] Furthermore, the processing unit 222 arranges each of the rectangular area images, which are the labeled images output from the trained model, in the same positional relationship as each of the rectangular area images input to the trained model, and combines them to obtain a labeled image corresponding to the composite image. The processing unit 222 performs an unpacking process on the obtained labeled images to obtain a labeled image corresponding to the input tomographic image. This allows the processing unit 222 to detect the area of ​​the retinal layer in the tomographic image to be processed based on the obtained labeled image. Note that, when a boundary image indicating a boundary or in which a label is attached to each boundary is used as output data of the training data, the processing unit 222 can detect the boundary of the retinal layer in the tomographic image to be processed.

[0088] In addition, when label images in units of labels are used as output data of the learning data, when a composite image is input to the learned model, a label image for each region corresponding to the composite image is output, such as label images 707 to 710. In this case, the processing unit 222 performs an unpacking process on the output multiple label images, thereby being able to detect the region of the retinal layer in the tomographic image.

[0089] (A series of operations) Next, a series of processes according to this embodiment will be described with reference to Fig. 11(a) and Fig. 11(b). Fig. 11(a) is a flowchart of a series of processes according to this embodiment. When a series of processes according to this embodiment is started, the process proceeds to step S1110.

[0090] In step S1110, the acquiring unit 21 acquires a subject identification number, which is an example of information for identifying the subject's eye, from outside the image processing device 20, such as the input unit 60. The acquiring unit 21 acquires information on the subject's eye held in the external storage device 40 based on the subject identification number, and stores the information in the storage unit 24.

[0091] In step S1120, the drive control unit 23 controls the OCT device 10 to scan the subject's eye to capture an image, and the acquisition unit 21 acquires an interference signal including tomographic information of the subject's eye from the OCT device 10. The scan of the subject's eye is performed by the drive control unit 23 controlling the OCT device 10 to operate the light source 11, the galvanometer mirror 12, etc. in response to an instruction from an operator to start scanning.

[0092] The galvanometer mirror 12 includes an X scanner for the horizontal direction and a Y scanner for the vertical direction. Therefore, the drive control unit 23 can scan the measurement light in each of the horizontal direction (X) and vertical direction (Y) in the device coordinate system by changing the orientation of each of these scanners. Note that the drive control unit 23 can also scan the measurement light in a direction that combines the horizontal direction and the vertical direction by changing the orientation of these scanners simultaneously. Therefore, the drive control unit 23 can scan the measurement light in any direction on the fundus plane.

[0093] The drive control unit 23 adjusts various shooting parameters when shooting. Specifically, the drive control unit 23 sets at least the position of the pattern displayed by the internal fixation light 16, the scan range and scan pattern by the galvanometer mirror 12, the coherence gate position, and the focus.

[0094] The drive control unit 23 controls the light emitting diodes of the display unit 161 to control the position of the pattern displayed by the internal fixation light 16 so as to capture the center of the macula and the optic disc of the subject's eye. The drive control unit 23 also sets a scan pattern such as a raster scan, radial scan, circle scan, or cross scan for capturing a three-dimensional volume as the scan pattern of the galvanometer mirror 12. Regardless of which scan pattern is selected, multiple images (repeated number of times is two or more) may be captured on one line. In this embodiment, the scan pattern is a raster scan for acquiring volume data (three-dimensional tomographic image). After adjustment of these imaging parameters is completed, in response to an instruction from the operator to start imaging, the drive control unit 23 controls the OCT device 10 to capture an image of the subject's eye and acquires an interference signal corresponding to the three-dimensional tomographic data.

[0095] Although detailed description is omitted in this disclosure, the OCT device 10 can track the subject's eye in order to capture the same location for averaging or for calculating motion contrast data related to OCTA (OCT Angiography). This allows the OCT device 10 to scan the subject's eye while reducing the influence of fixational eye movement.

[0096] In step S1130, the tomographic image generating unit 221 generates a three-dimensional tomographic image based on the interference signals acquired by the acquiring unit 21. The tomographic image generating unit 221 can generate a tomographic image by performing a general reconstruction process on each interference signal.

[0097] First, the tomographic image generating unit 221 removes fixed pattern noise from the interference signal. The fixed pattern noise is removed by averaging the acquired signals of multiple A-scans to extract the fixed pattern noise, and subtracting it from the input interference signal. After that, the tomographic image generating unit 221 performs a desired window function process to optimize the depth resolution and dynamic range, which are in a trade-off relationship when the interference signal is Fourier transformed in a finite section. The tomographic image generating unit 221 generates three-dimensional tomographic data by performing a fast Fourier transform (FFT) process on the interference signal that has been subjected to the window function process.

[0098] The tomographic image generating unit 221 obtains each pixel value of the tomographic image based on the generated three-dimensional tomographic data, and generates a three-dimensional tomographic image. Note that the method of generating the tomographic image is not limited to this, and any known method may be used.

[0099] In step S1140, the processing unit 222 of the image processing unit 22 performs a process of detecting the retinal layer. Details of step S1140 will be described later. When the image processing unit 22 performs the process of detecting the retinal layer in step S1140, the process proceeds to step S1150.

[0100] In step S1150, the display control unit 25 displays the layer region or boundary and the tomographic image detected by the processing unit 222 on the display unit 50. Here, FIG.

[0101] 12 shows a display screen 1200 including a front fundus image 1201, a thickness map 1202 superimposed on the front fundus image 1201, an En-Face brightness image 1203, a tomographic image 1211, and a retinal thickness graph 1212. Retinal boundaries 1215 and 1216 are superimposed on the tomographic image 1211.

[0102] In this embodiment, the range of the retina is from the boundary line L1 indicating the internal limiting membrane to the boundary line L4 indicating the retinal pigment epithelium, and the boundaries 1215 and 1216 correspond to the boundary lines L1 and L4, respectively. The range of the retina is not limited to this, and may be, for example, the range from the boundary line L1 indicating the internal limiting membrane to the boundary line L6 indicating the choroid, in which case the boundaries 1215 and 1216 can correspond to the boundary lines L1 and L6, respectively.

[0103] The retinal thickness graph 1212 is a graph showing the retinal thickness obtained from the boundaries 1215 and 1216. The thickness map 1202 expresses the retinal thickness obtained from the boundaries 1215 and 1216 as a color map. Note that, for the sake of explanation, color information corresponding to the thickness map 1202 is not shown in FIG. 12, but in reality, the thickness map 1202 can display the retinal thickness corresponding to each coordinate in the fundus front image 1201 according to the corresponding color map. The luminance En-Face image 1203 is a front image generated by projecting data in the range between the boundaries 1215 and 1216 of the three-dimensional tomographic data acquired about the subject's eye in the XY direction.

[0104] Here, the thickness map 1202, the luminance En-Face image 1203, the thickness graph 1212, and the boundaries 1215 and 1216 are examples of what can be generated by the image processing device 20 based on the retinal layers and boundaries detected by the processing unit 222. Any existing method may be adopted as a method for generating these. The image processing device 20 may also display an OCTA front image (OCTA En-Face image) or the like generated by determining a generation range of a front image based on the detected retinal layers and boundaries, and projecting motion contrast data within the generation range in the XY direction. Any known method may be used as a method for generating an OCTA front image or motion contrast data.

[0105] In the above, the range of the retinal layer used for generating the thickness map 1202, etc. is set to the boundary line L1 to boundary line L4 between the internal limiting membrane and the nerve fiber layer, but the range is not limited to this. The range may be any range that can be determined based on the boundary or retinal layer detected by the processing unit 222, and may be configured to be set or changed according to a desired configuration, such as boundary line L2 to boundary line L3, or boundary line L1 to boundary line L3, etc.

[0106] In addition to the above, the display screen 1200 of the display unit 50 may be provided with a patient tab, an imaging tab, a report tab, a setting tab, etc. In this case, the contents shown on the display screen 1200 of FIG. 12 will be displayed on the report tab. The display screen 1200 may also display a patient information display section, an examination sort tab, an examination list, etc. The examination list may display thumbnails of fundus images, tomographic images, and OCTA front images. When the process in step S1150 ends, the series of processes according to this embodiment ends.

[0107] (Retinal layer detection process) Next, a specific procedure for the retinal layer detection process in step S1140 according to this embodiment will be described with reference to Figures 4, 5, 11(b) and 13. Figure 11(b) is a flowchart of the retinal layer detection process according to this embodiment. When the retinal layer detection process is started in step S1140, the process proceeds to step S1141.

[0108] In step S1141, the processing unit 222 detects regions of interest from the tomographic image to be processed. Specifically, as described above, the processing unit 222 detects the regions of interest 406-409 from the two-dimensional tomographic images 401-404 included in the acquired three-dimensional tomographic image, respectively, and extracts images of the regions of interest 406-409. The processing unit 222 also stores in the storage unit 24 information on the tomographic images from which the extraction has been performed and position information on each of the extracted regions of interest in each tomographic image.

[0109] In step S1142, the processing unit 222 performs packing processing on the images of the regions of interest extracted from the multiple tomographic images in step S1141 to generate a composite image. Specifically, as described above, the processing unit 222 vertically arranges (combines) the images of the regions of interest 406-409 extracted from the multiple tomographic images 401-404 in a single image (depth direction of the subject's eye) to generate a composite image 405.

[0110] In step S1143, the processing unit 222 performs a segmentation process on the composite image generated in step S1142 by using the trained model. Specifically, the processing unit 222 inputs the composite image 405 to the trained model and obtains the output labeled image 505.

[0111] In step S1144, the processing unit 222 performs an unpacking process on the label image acquired in step S1143, and acquires the detection result images 501 to 504 corresponding to each of the regions of interest. At this time, the processing unit 222 returns the images of each of the regions of interest divided from the label image to the positions of the regions of interest in the corresponding tomographic images, using the information and position information of the tomographic images corresponding to each of the regions of interest stored in the storage unit 24 in step S1142.

[0112] In step S1145, the processing unit 222 performs post-processing on the detection result images 501-504 of step S1144. In this embodiment, the processing unit 222 performs binarization processing and hole filling processing using a dynamic threshold on the detection result images of each layer region of the retina as post-processing. In the detection result images 501-504, holes may occur in the detected regions due to noise, etc. Fig. 13(a) shows an example in which a hole 1303 occurs in a detection region 1302 of a detection result image 1301.

[0113] In this embodiment, the processing unit 222 can use an image expansion / contraction algorithm as the hole filling process to erase the holes. Fig. 13(b) shows an example of a detection result image 1304 having a detection area 1305 in which a hole 1303 has been erased by the hole filling process. Note that, although the image expansion / contraction algorithm is used as the hole filling process in this embodiment, the hole filling process is not limited to this. The processing unit 222 may perform the hole filling process by, for example, treating the entire inside of the binarized extraction area as the extraction area (feeling).

[0114] In step S1146, the processing unit 222 detects the area of ​​the retinal layer in each tomographic image based on the detection result image that has been subjected to post-processing. In the detection result image, the area of ​​the retinal layer is labeled by a segmentation process using the trained model, so the processing unit 222 can detect the area of ​​the retinal layer in the tomographic image using the label. The processing unit 222 may also detect each boundary based on each label of the detection result image in which the area of ​​the retinal layer is labeled. The processing unit 222 may detect, for example, the boundary between the labels in the detection result image as the boundary of the retinal layer.

[0115] In addition, when a boundary image showing a boundary or labeling each boundary is used as output data of the learning data, the processing unit 222 can obtain the boundary image as the detection result image. In this case, the processing unit 222 can detect each boundary in the retinal region based on the detection result image subjected to post-processing. In this case, the processing unit 222 can also detect the region of each retinal layer based on each detected boundary by rule-based processing. When the region of the retinal layer is detected in step S1146, the detection process of the retinal layer of this embodiment ends, and the process proceeds to step S1150.

[0116] As described above, the image processing device 20 according to this embodiment includes the acquisition unit 21 that acquires a tomographic image of the subject's eye, and the processing unit 222 that detects at least one of a layer and a boundary in the acquired tomographic image. The processing unit 222 generates a composite image by synthesizing images of a plurality of regions of interest, which are a plurality of partial regions in the acquired tomographic image. Furthermore, the processing unit 222 detects at least one of a layer and a boundary in the generated composite image using a trained model, and detects at least one of a layer and a boundary in the acquired tomographic image using a detection result obtained using the trained model.

[0117] Specifically, the processing unit 222 according to this embodiment generates a composite image by arranging images of a plurality of regions of interest (partial regions) in the depth direction of the subject's eye and synthesizing them. The acquisition unit 21 also acquires a three-dimensional tomographic image including a plurality of two-dimensional tomographic images of the subject's eye. Furthermore, the processing unit 222 detects a plurality of regions of interest in the plurality of two-dimensional tomographic images using the plurality of two-dimensional tomographic images included in the acquired three-dimensional tomographic image. In this embodiment, the processing unit 222 detects one region of interest from each of the plurality of two-dimensional tomographic images. Furthermore, the processing unit 222 detects a plurality of regions of interest using pixel values ​​and a threshold value in the acquired tomographic image. Here, the regions of interest include the retina.

[0118] Note that the processing unit 222 does not need to generate one composite image using all the regions of interest included in the three-dimensional tomographic image, but only needs to generate one composite image using at least two regions of interest. Therefore, the processing unit 222 only needs to detect at least two regions of interest included in the three-dimensional tomographic image as the multiple regions of interest used to generate one composite image.

[0119] The trained model used is a trained model obtained by using a composite image including a plurality of regions of interest in a tomographic image of the subject's eye and a label image indicating at least one of a region and a boundary in the composite image as training data. The processing unit 222 inputs the generated composite image into such a trained model to obtain a label image indicating at least one of a region and a boundary in the generated composite image. The processing unit 222 also obtains a detection result in each region of interest using the detection result obtained using the trained model, and detects at least one of a layer and a boundary in the tomographic image corresponding to each region of interest using the detection result in each region of interest obtained.

[0120] Since the image processing device according to the present embodiment has the above-mentioned configuration, when detecting the regions of the retinal layers from the tomographic image using the trained model, the image processing device detects a region of interest including the retinal region from the tomographic image. In addition, the image processing device uses a composite image that combines the images of the detected regions of interest as target data for the region detection process. This makes it possible to reduce the data to be transferred and the target data for the region detection process while performing region detection (segmentation) of all target retinal layers using the trained model, thereby improving the efficiency of the region detection process and speeding up the process.

[0121] In this embodiment, the OCT device 10 is used to capture a tomographic image to obtain an interference signal and a three-dimensional tomographic image. However, the method of obtaining a three-dimensional tomographic image of the subject's eye is not limited to this. For example, the image processing device 20 may obtain three-dimensional tomographic data or tomographic images that have been captured in the past and stored in an external storage device 40 or the like, and perform segmentation processing on the obtained three-dimensional tomographic data or tomographic images. This processing can also be applied to each of the modified examples and embodiments described below.

[0122] In this embodiment, the labeled image with no labeling for the choroid and sclera layers is used as the learning data, but the labeled image with the labeling for the choroid and sclera layers may be used as the learning data. In this case, the processing unit 222 can obtain the labeled image with the labeling for the choroid and sclera layers by processing using the trained model, and can detect the choroid and sclera layers. This processing can be appropriately applied to each of the modified examples and embodiments described below.

[0123] In this embodiment, a labeled image is obtained from the composite image and the retinal layer area is detected using a trained model trained using a labeled image with a region label attached as output data of the training data. In contrast, as described above, a trained model trained using a boundary image indicating a boundary or labeled for each boundary as output data of the training data may be used to obtain a boundary image from the composite image and detect the boundary of the retina. For a boundary image indicating a boundary, a rule-based process may be used to identify which boundary of the retinal layer the indicated boundary is. These processes may also be applied to each of the modified examples and embodiments described below.

[0124] The processing unit 222 may also be configured to correct the detected regions and boundaries of the retinal layers in response to an instruction from an operator via the input unit 60. The processing unit 222 may also be configured to correct the labeled image acquired from the trained model in response to an instruction from an operator via the input unit 60. In this case, the image processing device 20 may be configured to perform additional learning on the trained model using a synthetic image corresponding to the corrected labeled image as training data. This processing may also be applied to each of the modified examples and embodiments described below.

[0125] (Modification 1 of the first embodiment) In the first embodiment, in step S1143, the processing unit 222 uses a trained model to acquire a label image having labels of a plurality of regions, but the image acquired using the trained model is not limited to this. For example, a label image in units of labels as shown in FIG. 7 may be used as a label image of a composite image used as output data of the training data. When a trained model that has undergone such training is used, the image output from the trained model also becomes a label image in units of labels, and each image corresponds to the region detection result of each different label. The processing unit 222 can also perform unpacking processing or the like on such region detection results in the same manner as described above to detect each region and boundary of the retinal layer. However, there are cases where inconsistency occurs in the boundary lines of the detected regions of adjacent retinal layers (when the location of the common boundary line does not match). In this case, the center line (average line) of the corresponding boundary lines of each region may be set as the common boundary line.

[0126] (Modification 2 of the first embodiment) In step S1142 according to the first embodiment, the processing unit 222 packs the images of the extracted attention areas into one image to generate a composite image. Here, in the packing process, the processing unit 222 may provide a space between the attention areas to be packed into the composite image. Here, the space may be, for example, 10 pixels. However, the size of the space may be set to any value according to the desired configuration as long as it is large enough to distinguish the attention areas.

[0127] The processing unit 222 may also determine the size of the blank space by analyzing the region of interest. For example, the processing unit 222 may increase the blank space when the region of interest includes an enlarged retina. Furthermore, for example, when the region of interest includes a tilted retina, the blank space may be set to zero, or the blank space of adjacent regions of interest may be reduced in consideration of the blank space already included in the region of interest, thereby overlapping the regions of interest.

[0128] (Modification 3 of the first embodiment) In the first embodiment, a labeled image indicating the retinal layer area is output from the trained model by using a labeled image with a label of the retinal layer area as output data of the training data. Also, the processing unit 222 is configured to detect the retinal layer area from a detection result image obtained by unpacking the labeled image output from the trained model. In contrast, the processing unit 222 may detect the retinal layer area by detecting an image of the retinal area from an image of the attention area using the trained model and performing rule-based processing on the image of the retinal area.

[0129] It is known that in a tomographic image of a test eye, the boundaries between the retinal pigment epithelium and the choroid or sclera may not be properly detected by rule-based area detection processing. Therefore, as described above, it is expected that the accuracy of detecting the areas and boundaries of retinal layers can be improved by detecting the retinal area in the tomographic image using a trained model and performing rule-based area detection processing on the detected retinal area.

[0130] Therefore, in the third modification of the first embodiment, the processing unit 222 detects an image of the retina region from the image of the attention region using the trained model, and performs rule-based processing on the image of the retina region. In this case, a labeled image in which the retina region is labeled and corresponds to the composite image, which is the input data, is used as output data of the training data. Note that the method of creating the labeled image may be the same as the method of creating the labeled image used as output data of the training data according to the first embodiment.

[0131] Here, as the output data of the learning data according to this modification, the area from the NFL region to the RPE layer region is set as the range of the retina, and a label image in which the area is labeled as the retina region can be used. Note that in this modification, the area from the NFL region to the RPE layer region is set as the range of the retina, but the range of the retina is not limited thereto. For example, the range of the retina may be defined as the range from the boundary line L1 to the boundary line L3, the range from the boundary line L1 to the boundary line L5, or the range from the boundary line L1 to the boundary line L6, etc.

[0132] The processing unit 222 according to this modification can obtain a labeled image in which a label is added to the retinal region corresponding to the composite image by inputting the composite image described in the first embodiment to the trained model that has undergone such training. The processing unit 222 can perform an unpacking process on the obtained labeled image, similar to the unpacking process in the first embodiment, and overwrite each labeled image on the region of interest of the original tomographic image. The processing unit 222 can detect the retinal region based on the tomographic image that has undergone the unpacking process.

[0133] Furthermore, the processing unit 222 can perform boundary detection processing of the detected retinal region by extracting image features using edge detection (Sobel filter or gradient edge detector) of image processing. Here, the boundary of the retinal layer is extracted as the image feature. Furthermore, the processing unit 222 can perform region detection (segmentation) of the retinal layer by judging the result on a rule base.

[0134] (Boundary detection process) Here, the boundary detection process according to this modification will be described with reference to Fig. 14(a) to Fig. 14(d). Fig. 14(a) shows a tomographic image 1401, which is an example of an input tomographic image. Fig. 14(b) shows a label image 1402 corresponding to a detection result image acquired by performing an unpacking process on a label image acquired using a trained model, in which a label 1404 of the retina region and labels 1403 and 1405 corresponding to the rest are assigned. The processing unit 222 according to this modification sets the range of the retina region indicated by the label 1404 in the label image 1402 as the target region for layer detection.

[0135] The processing unit 222 can detect a target boundary by detecting a contour within the retinal region indicated by the label 1404 in the labeled image 1402. Fig. 14(c) shows an enhanced image 1406 obtained by performing noise removal processing and edge enhancement processing as processing for detecting a contour within the retinal region by the processing unit 222.

[0136] Here, the noise removal process and the edge enhancement process will be described. As shown in Fig. 14(c) and 14(d), the retinal layer is discontinued in the optic disc area, so the processing unit 222 according to this modification does not perform boundary detection.

[0137] The processing unit 222 performs noise removal and edge enhancement processing on the region corresponding to the label 1404 in the tomographic image 1401 to be processed. The processing unit 222 applies, for example, a median filter or a Gaussian filter as the noise removal processing. In addition, the processing unit 222 applies a Sobel filter or a Hessian filter as the edge enhancement processing.

[0138] Here, edge enhancement processing for two-dimensional tomographic images using a two-dimensional Hessian filter will be described. The Hessian filter uses two eigenvalues ​​(λ 1 , λ 2 ) relationship, the second-order local structure of the two-dimensional gray-scale distribution can be emphasized. Therefore, in this modification, the eigenvalues ​​and eigenvectors (e 1 , e 2 ) is used to emphasize two-dimensional line structures. Since the line structures in the two-dimensional tomographic image of the subject's eye correspond to the structure of the retinal layer, the application of the Hessian filter can emphasize the structure of the retinal layer.

[0139] In order to detect retinal layers with different thicknesses, the resolution of smoothing using a Gaussian function performed when calculating the Hessian matrix can be changed. In addition, when applying a two-dimensional Hessian filter, the data can be transformed to match the physical sizes of the XZ of the image before application. In the case of general OCT, the physical sizes in the XY and Z directions are different. Therefore, the filter is applied by matching (normalizing) the physical size of the retinal layer for each pixel. Note that the physical sizes in the XY and Z directions can be grasped from the design / configuration of the OCT device 10, so the data of the tomographic image can be transformed based on the physical sizes. In addition, if the physical size is not normalized, it can be approximately handled by changing the resolution of smoothing using a Gaussian function.

[0140] Although the processing on two-dimensional tomographic images has been described above, the application of the Hessian filter is not limited to this. If the data structure when the tomographic image is captured is a three-dimensional tomographic image obtained by raster scanning, it is also possible to apply a three-dimensional Hessian filter. In this case, after the image processing unit 22 performs position alignment processing in the XZ directions between adjacent tomographic images, the processing unit 222 calculates the three eigenvalues ​​(λ 1 , λ 2 , λ 3 ) relationship, it is possible to emphasize the second-order local structure of the three-dimensional gray distribution. Therefore, the eigenvalues ​​and eigenvectors of the Hessian matrix (e 1 , e 2 , e 3 ) to emphasize the three-dimensional layer structure, it is also possible to emphasize edges three-dimensionally.

[0141] In the emphasized image 1406, the portion where the edge is emphasized appears as a white line 1407. Note that a region in the tomographic image 1401 that does not correspond to the label 1404 can be treated as a region where edges are not detected. Also, although a configuration for performing edge emphasis processing using a Hessian filter has been described here, the processing method for edge emphasis processing is not limited to this, and any existing method may be used.

[0142] 14D shows a boundary image 1408 indicating the boundaries of retinal layers detected by the processing unit 222 using the labeled image 1402 and the highlighted image 1406. In the boundary image 1408, a black line 1409 indicates an example of the boundary.

[0143] Next, a process in which the processing unit 222 detects the boundaries of the retinal layers from the label image 1402 and the enhanced image 1406 will be described. The processing unit 222 detects the edge-enhanced boundaries from the enhanced image 1406. In this modification, since the boundaries between the ILM and the NFL and the RPE have already been detected, the processing unit 222 subsequently detects the boundaries of the ISOS, the NFL and the GCL. Although not shown, other boundaries may be detected, such as the boundary between the outer plexiform layer (OPL) and the outer nuclear layer (ONL), the boundary between the inner plexiform layer (IPL) and the inner nuclear layer (INL), the boundary between the INL and the OPL, or the boundary between the GCL and the IPL.

[0144] As a method of detecting the boundary, a plurality of points with strong edge strength are detected as boundary candidates in each A-scan, and points (points with strong edge strength) are connected as lines based on the continuity between the boundary candidates in adjacent A-scans. In addition, when points are connected as lines, the processing unit 222 can remove outliers by evaluating the smoothness of the lines. More specifically, for example, the Z-direction positions of the connected points are compared, and if the difference in the Z-direction positions is greater than a predetermined threshold, the newly connected point can be determined as an outlier and excluded from the connecting process. In addition, when the outliers are removed, boundary candidates in the A-scans adjacent to the A-scan positions of the excluded points may be connected as lines. Note that the method of removing outliers is not limited to this, and any existing method may be used.

[0145] The processing unit 222 determines the corresponding boundary for each line formed by connecting the points based on the vertical distance and positional relationship in the Z direction of the boundary of the retina layer. If no boundary is detected as a result of removing outliers in each A-scan, the boundary may be obtained by interpolation from the surrounding boundaries. Alternatively, the boundary candidate may be searched for in the horizontal direction (X or Y direction) based on the edge from the surrounding boundaries, and the boundary may be determined again based on the boundary candidate searched from the surrounding boundaries.

[0146] Thereafter, the processing unit 222 executes a process for correcting the detected boundary to make the boundary shape smooth. For example, the boundary shape may be smoothed using image features and shape features by an active contour model such as Snakes or Level Set method. In addition, the coordinate values ​​of the boundary shape may be regarded as time-series data based on a signal, and the shape may be smoothed by a smoothing process such as a Savitzky-Golay filter, a simple moving average, a weighted moving average, or an exponential moving average.

[0147] By such processing, the processing unit 222 can detect the retinal layer in the detected retinal region. Note that the above-mentioned detection process of the retinal layer by the processing unit 222 is an example, and the retinal layer can be detected by using any existing segmentation process.

[0148] In this modified example, the retinal region is detected using a trained model, and the retinal layer region and boundary are detected by performing rule-based segmentation processing on the retinal region. Even in such a case, the data to be transferred and the target data for the region detection processing can be reduced by using a synthetic image obtained by performing packing processing on the image of the attention region as the image to be input to the trained model. Therefore, the efficiency of the region detection processing can be improved and the processing speed can be increased.

[0149] In this modification, an example has been described in which the processing unit 222 performs boundary detection processing on a detection result image obtained by performing unpacking processing on a labeled image acquired from a trained model. Alternatively, the processing unit 222 may perform boundary detection processing on a labeled image acquired from a trained model and then perform unpacking processing.

[0150] In this modification, a labeled image with a label of a retinal area is used as the output data of the learning data. In contrast, a boundary image indicating the boundary of the retinal area corresponding to the composite image, which is the input data, may be used as the output data of the learning data. The method of creating the boundary image may be the same as the method of creating the boundary image used as the output data of the learning data according to the first embodiment.

[0151] In this modification, the processing unit 222 is configured to perform the retinal area detection process using the trained model and the boundary detection process using the detection result image, but components that perform these processes separately may be provided. In this case, the processing unit 222 can be configured to include components that perform these processes separately.

[0152] (Fourth Modification of the First Embodiment) In the first embodiment, the processing unit 222 may detect the boundary line based on each label in the detection result image in which the retinal layer area is labeled. However, the method of detecting the boundary of the retinal area is not limited to this. Here, with reference to Figs. 15(a) to 15(c), another example of the boundary detection process by the processing unit 222 will be described.

[0153] Fig. 15(a) shows a detection result image 1500 in which the retinal layer region is labeled, acquired by the processing unit 222. In the detection result image 1500, a retinal region 1501 shows a retinal region labeled with the retinal layer. The processing unit 222 performs a binarization process on the entire retinal region 1501 of the detection result image 1500. Fig. 15(b) shows a binarized image 1507 after the binarization process has been performed.

[0154] Next, the processing unit 222 calculates a center line 1502 of the binarized retinal region 1501. Specifically, the processing unit 222 calculates the center of the vertical direction (depth direction) of the retinal region 1501 at each pixel position in the horizontal direction (transverse direction) of the image for the binarized retinal region 1501, and connects these to calculate the center line 1502. Note that in FIG. 15(b), the center line 1502 is indicated by a dashed line.

[0155] Thereafter, the processing unit 222 determines that the boundary line above the center line 1502 in the binarized image 1507 is the ILM 1503, and determines that the region above the ILM 1503 is the vitreous region 1504. Furthermore, the processing unit 222 determines that the boundary line below the center line 1502 in the binarized image 1507 is Bruch's membrane 1505, and determines that the region below the Bruch's membrane 1505 is the choroid or sclera region 1506. Next, the processing unit 222 extracts the boundary line between the ILM 1503 and the Bruch's membrane 1505, for example, by the boundary detection process described in the third modification of the first embodiment, and obtains a boundary line image 1508 shown in FIG. 15(c). Even by such a process, the processing unit 222 can detect the boundary of the retina region using the detection result image in which the regions of the retinal layers are labeled.

[0156] In this modification, an example has been described in which the processing unit 222 performs the boundary detection process using a detection result image obtained by performing an unpacking process on a labeled image acquired from a trained model. Alternatively, the processing unit 222 may perform the boundary detection process using a labeled image acquired from a trained model, and then perform the unpacking process.

[0157] Second Embodiment In the first embodiment, regions of interest are extracted from multiple tomographic images, and a composite image obtained by packing the multiple regions of interest is used as input data for region detection processing, reducing the amount of data to be transferred and the data subject to region detection processing, improving the efficiency of region detection and speeding up processing. In contrast, an ophthalmologic apparatus according to a second embodiment of the present invention further reduces the granularity (size) of regions of interest, further improving detection efficiency.

[0158] Hereinafter, with reference to Fig. 16 to Fig. 19(b), the image processing device according to this embodiment will be described, focusing on the differences from the image processing device according to the first embodiment. Since the configuration of the image processing device according to this embodiment is similar to that of the image processing device according to the first embodiment, the same reference numerals are used for the respective components and the description thereof will be omitted. Furthermore, since the series of processes according to this embodiment are similar to the series of processes according to the first embodiment shown in Fig. 11(a) except for the process in step S1140, the description of the processes other than the process in step S1140 will be omitted.

[0159] First, a specific procedure of the retinal layer detection process in step S1140 according to this embodiment will be described with reference to Fig. 16 to Fig. 18(i). Fig. 16 is a flowchart of the retinal layer detection process according to this embodiment. In this embodiment, when the retinal layer detection process is started in step S1140, the process proceeds to step S1601.

[0160] In step S1601, the processing unit 222 divides each two-dimensional tomographic image included in the three-dimensional tomographic image acquired by the acquisition unit 21 into a plurality of stripes (rectangular regions) in the transverse direction of the tomographic image. Here, the transverse direction of the tomographic image may be any direction intersecting with the depth direction, for example, the main scanning direction in raster scanning. FIG. 17(a) shows a tomographic image 1700 to be processed, and FIG. 17(b) shows an example of a plurality of stripes 1701 to 1707 after the stripe division process in step S1601. The stripes 1701 to 1707 are stripes obtained by dividing the tomographic image 1700 in the transverse direction of the tomographic image 1700. In this embodiment, the width of the stripe is set to 128A scan, but the width of the stripe is not limited to this, and may be any width that the processing unit 222 can process, such as 256A scan. Furthermore, the width of the stripe does not need to be determined in advance, but may be determined dynamically, for example, as the width when the tomographic image to be processed is divided into a predetermined number (N) of parts in the transverse direction of the tomographic image.

[0161] In step S1602, the processing unit 222 detects a region of interest from the tomographic image to be processed. The processing in step S1602 is similar to the processing in step S1141 according to the first embodiment, and therefore a description thereof will be omitted. However, the detection of a region of interest according to this embodiment is performed in units of stripes. FIG. 17(c) shows an example of the result of the processing in step S1602. Specifically, the processing unit 222 detects regions of interest 1711-1717, which are partial regions, from each of the stripes 1701-1707 by the same processing as the detection processing of a region of interest according to the first embodiment, and extracts images of the regions of interest 1711-1717. Furthermore, when extracting images of the regions of interest from the stripes, the processing unit 222 stores information on the stripes from which the extraction was performed and position information of the extracted regions of interest in the stripes in the storage unit 24.

[0162] In step S1603, the processing unit 222 packs the multiple regions of interest extracted from the multiple stripes in step S1602 into one image to generate a composite image. The processing in step S1603 is similar to the processing in step S1142 according to the first embodiment. However, the packing processing in this embodiment is performed in stripe units. Note that the number of regions of interest that can be packed into one composite image depends on the size of the regions of interest and the size of the composite image, but the processing unit 222 performs the packing processing so as to pack as many regions of interest as possible. In relation to this, when there is a free area in the composite image into which the regions of interest in the tomographic images are packed, the processing unit 222 can pack images of the regions of interest in other tomographic images into the composite image.

[0163] Fig. 18(a) to Fig. 18(d) are diagrams for explaining the packing process. Fig. 18(a) shows examples of attention regions 1801 to 1807 detected in a two-dimensional tomographic image 1800 by the processing unit 222. Fig. 18(b) shows examples of attention regions 1811 to 1817 detected in another two-dimensional tomographic image 1810 in a three-dimensional tomographic image by the processing unit 222. Fig. 18(c) to Fig. 18(e) show examples of composite images 1820, 1830, and 1840 generated by performing packing processes on the attention regions 1801 to 1807 and 1811 to 1815, respectively.

[0164] In step S1603 according to this embodiment, the processing unit 222 performs packing processing on the images of the regions of interest 1801-1804 extracted from the tomographic image 1800 to be processed, and generates a composite image 1820. The processing unit 222 also performs packing processing on the images of the regions of interest 1805-1807 extracted from the tomographic image 1800 to be processed and the image of the region of interest 1811 extracted from another tomographic image 1810 to be processed, and generates a composite image 1830. The processing unit 222 also performs packing processing on the images of the regions of interest 1812-1815 extracted from the tomographic image 1810 to be processed, and generates a composite image 1840. Thereafter, the processing unit 222 performs similar packing processing on the subsequent regions of interest, for example, the regions of interest 1816-1817, and the regions of interest in the remaining two-dimensional tomographic images (not shown), and generates a composite image.

[0165] In step S1604, the processing unit 222 performs segmentation processing on the composite image generated in step S1603 using the trained model, and acquires a label image corresponding to the composite image. The processing in step S1604 is similar to the processing in step S1143 according to the first embodiment. Specifically, the processing unit 222 inputs the generated composite image 1820, etc., to the trained model, and acquires a label image output from the trained model. Here, FIG. 18(f) shows an example of a label image 1850 output from the trained model when the composite image 1820 shown in FIG. 18(c) is input to the trained model.

[0166] In addition, the learning data of the trained model according to this embodiment may be a composite image obtained by packing an image of a region of interest extracted from a stripe of a tomographic image as input data, and a label image corresponding to the composite image as output data. Note that the method of creating the composite image and the label image corresponding to the composite image may be the same as the method of creating the composite image and the label image according to the first embodiment, except that the image of the region of interest is an image extracted from the stripe of a tomographic image.

[0167] In step S1605, the processing unit 222 performs an unpacking process on the label image acquired in step S1604 to acquire detection result images corresponding to each of the regions of interest. The process in step S1605 is similar to the process in step S1144 according to the first embodiment. Specifically, the processing unit 222 divides the label image acquired in step S1604 into images of each of the regions of interest. In addition, the processing unit 222 returns each of the regions of interest in the label image to the position of the region of interest in the corresponding stripe, using the stripe information and position information corresponding to each of the regions of interest stored in the storage unit 24.

[0168] Here, Fig. 18(g) shows an example of a detection result image 1855 including an image of a region of interest 1851, and Fig. 18(h) shows an example of a detection result image 1856 including an image of a region of interest 1852. Fig. 18(i) shows an example of a detection result image 1857 including an image of a region of interest 1853, and Fig. 18(j) shows an example of a detection result image 1858 including an image of a region of interest 1854. According to this example, the processing unit 222 divides the label image 1850 into images of the regions of interest 1851-1854, returns the images of the regions of interest 1851-1854 to the positions of the regions of interest in the corresponding stripes, and can obtain detection result images 1855-1858.

[0169] In step S1606, the processing unit 222 performs stripe combining processing on the detection result image acquired in step S1605, and acquires a detection result image corresponding to the tomographic image to be processed. Fig. 19(a) shows an example of a detection result image 1910 corresponding to the tomographic image to be processed, acquired by performing stripe combining processing on the stripe detection result images 1855 to 1861.

[0170] In step S1607, the processing unit 222 performs post-processing on the detection result image acquired in step S1606. The process in step S1607 is similar to the process in step S1145 according to the first embodiment, and therefore a description thereof will be omitted.

[0171] In step S1608, the processing unit 222 detects the area of ​​the retinal layer in each tomographic image based on the detection result image post-processed in step S1607. Since the area of ​​the retinal layer is labeled in the detection result image by the segmentation process using the trained model, the processing unit 222 can detect the area of ​​the retinal layer in the tomographic image using the label.

[0172] Furthermore, the processing unit 222 may detect each boundary based on each label of the detection result image in which the area of ​​the retinal layer is labeled. For example, the processing unit 222 may detect the boundary between the labels in the detection result image as the boundary of the retinal layer. Furthermore, the processing unit 222 may detect the boundary of the retinal layer by the method described in the fourth modified example of the first embodiment. FIG. 19(b) shows an example of a boundary image 1920 in which a boundary line is extracted from the detection result image 1910. When the area of ​​the retinal layer is detected in step S1608, the detection process of the retinal layer of this embodiment ends, and the process proceeds to step S1150.

[0173] As described above, the processing unit 222 according to this embodiment divides the two-dimensional tomographic image into a plurality of stripes (regions) in a direction (transverse direction) intersecting the depth direction of the subject's eye, and detects a region of interest, which is a partial region, from each of the divided stripes. For this reason, the processing unit 222 detects a plurality of regions of interest from each of the acquired two-dimensional tomographic images. Note that the processing unit 222 does not need to generate one composite image using all the regions of interest included in the three-dimensional tomographic image, but only needs to be able to generate one composite image using at least two regions of interest. For this reason, the processing unit 222 only needs to detect at least two regions of interest included in the three-dimensional tomographic image as the plurality of regions of interest used to generate one composite image.

[0174] With the above configuration, the image processing device 20 according to the present embodiment can further reduce the granularity (size) of the target data for the region detection process by dividing the tomographic image to be processed into multiple stripes. This makes it possible to further improve detection efficiency while performing region detection (segmentation) of all target retinal layers using a trained model.

[0175] In this embodiment, the image processing device 20 is configured to perform retinal layer area detection processing for a three-dimensional tomographic image including a plurality of two-dimensional tomographic images. In contrast, the image processing device 20 may be configured to perform retinal layer area detection processing including stripe division and integration according to this embodiment for a single two-dimensional tomographic image.

[0176] (Modification 1 of the second embodiment) In the second embodiment, the tomographic image is divided into a plurality of stripes, and an area of ​​interest is detected from each of the plurality of stripes. In contrast, as described in the first embodiment, an area of ​​interest corresponding to a retinal layer may be detected from the tomographic image, and the detected area of ​​interest may be divided into a plurality of stripes as described in the second embodiment. In this case, the processing unit 222 may generate a composite image by arranging the divided plurality of stripes vertically (in the depth direction) with a predetermined space therebetween, and use the composite image for processing using the trained model. In such a case, a composite image to be input to the trained model can be generated from one tomographic image. Note that the space between the stripes when generating the composite image may be set as described in the second modification of the first embodiment. In addition, the processing according to this modification may be appropriately applied to the embodiments and modifications described later.

[0177] (Modification 2 of the second embodiment) In step S1601 according to the second embodiment, when the tomographic image to be processed is divided into a plurality of stripes in the transverse direction, the stripes are divided into stripes each having a width of 128 A scans. In contrast, when the tomographic image is divided into stripes, an overlapping area (also called a "gap") may be provided in the stripes to facilitate alignment of the stripes in a subsequent stripe combining process.

[0178] Hereinafter, an image processing device according to Modification 2 of the second embodiment will be described with reference to Fig. 20(a) to Fig. 20(c). The image processing device according to this modification will be described with a focus on the differences from the image processing device according to the second embodiment. Note that since the configuration of the image processing device according to this modification is similar to the configuration of the image processing device according to the second embodiment, the same reference numerals will be used for the respective components and the description will be omitted.

[0179] In this modification, in step S1601, when dividing the tomographic image into stripes, the processing unit 222 provides overlapping regions in the stripes. Here, the overlapping regions are regions (common regions) that overlap adjacent stripes. With this configuration, in the stripe concatenation process in step S1606, the overlapping regions are appropriately superimposed on the adjacent stripes, so that the alignment of adjacent stripes can be performed more smoothly.

[0180] FIG. 20(a) to FIG. 20(c) are diagrams for explaining an example of a method for creating a stripe having an overlap region. FIG. 20(a) shows an example of a tomographic image divided into stripes 1701 to 1707. When dividing a tomographic image into a plurality of stripes in step S1601, the processing unit 222 according to this modification provides an overlap region that overlaps adjacent stripes in the stripe. Note that, when there is no adjacent stripe in a stripe, that is, for the first or last stripe in the tomographic image, the processing unit 222 can provide an area where the end of the stripe is folded back as an overlap region. This allows an overlap region to be provided for stripes adjacent to a stripe, and the size of data to be processed in the subsequent processing to be the same. Note that the area where the end of the stripe is folded back includes a mirrored image of the tomographic image included in the stripe.

[0181] Here, with reference to Figures 20(b) and 20(c), an example of providing an overlapping region in a stripe 1703 will be described. Note that, although adjacent stripes 1702 exist for the stripe 1703 in the tomographic image shown in Figure 20(a), in Figures 20(b) and 20(c) the stripe 1703 is shown as if there were no adjacent stripes 1702 for the purpose of illustratively explaining the processing.

[0182] First, when there is no adjacent stripe on the left side of the paper for stripe 1703, processing unit 222 sets a region obtained by folding back a part of the left end of stripe 1703 as overlap region 2001 on the left side of stripe 1703 as shown in Fig. 20(b) . Note that when there is no adjacent stripe on the right side of the paper, processing unit 222 sets a region obtained by folding back a part of the right end of stripe 1703 on the right side of stripe 1703 as overlap region.

[0183] Furthermore, when there is an adjacent stripe on the right side of the paper, the processing unit 222 provides a part of the left end of the stripe 1704 as an overlap region 2002 to the right of the stripe 1703, as shown in Fig. 20(c). When there is an adjacent stripe on the left side of the paper, the processing unit 222 provides a part of the right end of the adjacent stripe as an overlap region to the left of the stripe 1703. By performing such processing on all stripes, the processing unit 222 can obtain a plurality of stripes having overlap regions that overlap adjacent stripes from the tomographic image to be processed.

[0184] As described above, the processing unit 222 according to this modification divides a two-dimensional tomographic image into a plurality of stripes (regions) in a direction (transverse direction) intersecting the depth direction of the subject's eye, and detects a region of interest from each of the divided stripes. Here, the divided regions have regions that overlap with adjacent regions in the divided regions. In addition, the processing unit 222 adds a region that is a partial inversion of a region that does not have an adjacent region in the divided regions as the overlapping region. With this configuration, the image processing device according to this modification can improve the efficiency of the combining process for the stripes of the label image that correspond to the stripes divided into the tomographic image.

[0185] In this modification, when the width of the stripe is 128A scan, the width of the overlap area is 32A scan. However, the width of the overlap area does not need to be limited to this size. If the overlap area is too small, smooth alignment may not be possible during stripe connection processing, and conversely, if the overlap area is too large, overhead occurs and the overall processing time increases. Therefore, the width of the overlap area may be determined based on the results of processing performed in advance. Also, the processing unit 222 may analyze the shape of the retina to determine the width of the overlap.

[0186] The processing unit 222 may thin out A-scan data (A-scan data) included in adjacent stripe regions used for the overlap region when generating the overlap region. In this case, the processing unit 222 thins out information on overlapping regions in multiple regions obtained by dividing the tomographic image, and detects a region of interest from each of the multiple regions from which the information on overlapping regions has been thinned out. With this configuration, it is possible to reduce information on overlapping regions, and to reduce the overall processing time. When overlapping regions are provided in stripes as in this modified example, the composite image and label image (or boundary image) used as learning data may also be images that include a region of interest including an overlap region.

[0187] In the stripe concatenation process in step S1606, an error may occur between the segmentation process result of the overlap region and the segmentation process result of the adjacent stripe. In this case, the processing unit 222 may average the segmentation result of the main stripe and the segmentation result of the overlap region. The averaging process may be a weighted averaging process. For example, the weight of the segmentation process result of the main stripe (e.g., stripe 1703) may be increased (e.g., 4) and the weight of the segmentation process result of the left overlap region of the adjacent stripe 1704 may be decreased (e.g., 1), and the averaging process may be performed. Other weight values ​​may be set arbitrarily according to a desired configuration.

[0188] (Third embodiment) In the modified example of the second embodiment, a method was described in which a tomographic image to be processed is divided into a plurality of stripes including overlapping regions, thereby further reducing the granularity (size) of the target data for the region detection process and further improving the detection efficiency. In contrast, in the third embodiment of the present invention, in order to reduce overlapping regions in the stripes, a two-dimensional tomographic image to be processed is divided into stripes in a direction with a small number of samples in a three-dimensional tomographic image to be processed, and an image of the stripes to be processed is obtained.

[0189] Hereinafter, the image processing device according to this embodiment will be described with reference to Fig. 21 and Fig. 22, focusing on the differences from the image processing device according to the modified example of the second embodiment. Note that the configuration of the image processing device according to this embodiment is similar to the configuration of the image processing device according to the modified example of the second embodiment, so the same reference numerals are used for the respective components and the description is omitted. Also, the retinal layer detection process according to this embodiment is similar to the retinal layer detection process according to the modified example of the second embodiment, so the description is omitted.

[0190] A series of processes according to this embodiment will be described with reference to Fig. 21 and Fig. 22. Fig. 21 is a flowchart of a series of processes according to this embodiment. When a series of processes according to this embodiment starts, the process proceeds to step S2110.

[0191] In step S2110, the acquisition unit 21 acquires a subject identification number, which is an example of information for identifying the subject's eye, from outside the image processing device 20, such as the input unit 60. The process in step S2110 is similar to the process in step S1110, and therefore will not be described.

[0192] In step S2120, the drive control unit 23 controls the OCT device 10 to scan the subject's eye to capture an image, and the acquisition unit 21 acquires an interference signal corresponding to three-dimensional tomographic data including tomographic information of the subject's eye from the OCT device 10. The process in step S2120 is similar to the process in step S1120, and therefore a description thereof will be omitted.

[0193] In step S2130, the tomographic image generating unit 221 generates a tomographic image based on the interference signal acquired by the acquiring unit 21. The process in step S2130 is similar to the process in step S1130, and therefore a description thereof will be omitted.

[0194] In step S2140, the tomographic image generating unit 221 reconstructs the tomographic image generated in step S2130. Here, the tomographic image generating unit 221 compares the number of samplings in the main scanning direction (number of A scans) and the number of samplings in the sub-scanning direction (number of B scans) when capturing a three-dimensional tomographic image of the retina. Based on the comparison result, the tomographic image generating unit 221 reconstructs a tomographic image in the direction with the fewer number of samplings.

[0195] For example, assume that the number of samples in the main scanning direction for the tomographic image acquired in step S2120 is 1024 (A scans) and the number of samples in the sub-scanning direction is 128 (B scans). In this case, in step S2140, the tomographic image generating unit 221 reconstructs the tomographic image with the sub-scanning direction having the fewer number of samples as the transverse direction. This allows the acquiring unit 21 to acquire 1024 tomographic images with the number of samples in the transverse direction being 128 (A scans).

[0196] An example of the processing in step S2120 will be described with reference to FIG. 22(a) and FIG. 22(b). In FIG. 22(a) and FIG. 22(b), retina 2201 indicates the retina of the imaging subject. FIG. 22(a) shows an outline of a three-dimensional tomographic image obtained by imaging using raster scanning in which the main scanning direction is the X direction and the sub-scanning direction is the Y direction. In this case, in step S2130, the tomographic image generating unit 221 generates a three-dimensional tomographic image in which the number of samples in the sub-scanning direction is 20 B scans (U01 to 20) and the number of samples in the main scanning direction is 29 A scans (T01 to 29). For such a three-dimensional tomographic image, the tomographic image generating unit 221 acquires 29 two-dimensional tomographic images in which the number of samples in the transverse direction is 20 A scans, as shown in FIG. 22(b), as a processing result in step S2140.

[0197] In step S2150, the processing unit 222 performs a retinal layer detection process. The process in step S2150 is similar to the process in step S1140, and therefore the description is omitted. However, the retinal layer detection process in this embodiment is performed on the multiple two-dimensional tomographic images reconstructed in step S2140.

[0198] In step S2160, the tomographic image generating unit 221 reconstructs the detection result image acquired in step S2150 as an image in the original transverse direction. Any method may be used for reconstructing the image. For example, the tomographic image generating unit 221 may arrange the multiple two-dimensional detection result images acquired in step S2150 in a three-dimensional direction to construct a three-dimensional detection result image, and reconstruct a two-dimensional image in the original transverse direction from the three-dimensional detection result image. According to the process in step S2160, the tomographic image generating unit 221 can acquire a detection result image in the same transverse direction as the tomographic image generated in step S2130.

[0199] In step S2170, the display control unit 25 displays the retinal layer area or boundary detected in step S2160, the tomographic image, and the like on the display unit 50. The process in step S2170 is similar to the process in step S1150, and therefore a description thereof will be omitted. Note that when the process in step S2170 ends, a series of processes according to this embodiment ends.

[0200] As described above, the acquisition unit 21 according to the present embodiment acquires a two-dimensional tomographic image in which the transverse direction corresponding to the direction intersecting the depth direction of the subject's eye is set as the scanning direction with a small number of samples in the three-dimensional tomographic image of the subject's eye. This makes it possible to reduce the overlapping area in the stripes, improve the efficiency of striping and the efficiency of packing processing, and reduce the overall processing time.

[0201] (Modification of the third embodiment) In step S2140 according to the third embodiment, the number of samplings in the main scanning (the number of A scans) is compared with the number of samplings in the sub-scanning (the number of B scans), and a tomographic image is reconstructed with the direction with the smaller number of samplings as the transverse direction. However, the direction in which the tomographic image is reconstructed may be determined by other criteria. For example, the tomographic image generating unit 221 may reconstruct the tomographic image with the transverse direction being the direction in which the size of the imaging area on the retina is smaller than the size in the X direction and the Y direction. In this case, the acquiring unit 21 acquires a two-dimensional tomographic image with the scanning direction in which the imaging area size in the three-dimensional tomographic image of the subject's eye is smaller as the transverse direction.

[0202] Furthermore, the tomographic image generating unit 221 may analyze the curvature of the retina using the three-dimensional tomographic image acquired in step S2130, for example, and reconstruct a B-scan with the direction with the smaller curvature in the XY direction as the transverse direction. In this case, the acquiring unit 21 acquires a two-dimensional tomographic image with the transverse direction as the scanning direction with the smaller curvature of the retina. Note that the image used for analyzing the curvature of the retina is not limited to a three-dimensional tomographic image, and may be a frontal image of the fundus acquired using the fundus image capturing device 30, or the like. The curvature of the retina may be analyzed using any known method.

[0203] Even when these processes are performed, it is possible to reduce overlapping areas in stripes, improve the efficiency of striping, and also improve the efficiency of packing processes, thereby reducing the overall processing time.

[0204] (Fourth embodiment) In the second embodiment, a method was described in which a tomographic image to be processed is divided into a plurality of stripes to further reduce the granularity (size) of the target data for the region detection process and further improve the detection efficiency. In contrast, in the fourth embodiment of the present invention, the region of interest is flattened in order to reduce the effect of the curvature and tilt of the retina on the process and further reduce the target data for the segmentation process.

[0205] Hereinafter, with reference to FIG. 23 and FIG. 24, the image processing device according to this embodiment will be described, focusing on the differences from the image processing device according to the second embodiment. Since the configuration of the image processing device according to this embodiment is similar to the configuration of the image processing device 20 according to the second embodiment, the same reference numerals are used for each component and the description will be omitted. Furthermore, since the series of processes according to this embodiment are similar to the series of processes according to the second embodiment except for the process in step S1140, the description of the processes other than the process in step S1140 will be omitted. In this embodiment, as an example, the process in the case where a boundary image is used as the output data of the learning data will be described. However, as in the first to third embodiments, the process may be performed using a label image as the output data of the learning data.

[0206] A specific procedure of the retinal layer detection process in step S1140 according to this embodiment will be described with reference to Fig. 23 and Fig. 24. Fig. 23 is a flowchart of the retinal layer detection process according to this embodiment. Fig. 24(a) to (f) are diagrams for explaining the retinal layer detection process according to this embodiment. In this embodiment, when the retinal layer detection process is started in step S1140, the process proceeds to step S2301.

[0207] In step S2301, the processing unit 222 performs flattening processing on the tomographic image to be processed. Specifically, the processing unit 222 moves the tomographic image in the Z direction in units of A scans constituting the two-dimensional tomographic image (B-scan image) to align the position of the retina in the two-dimensional tomographic image. Note that in this embodiment, the processing unit 222 performs flattening processing based on the ILM in the tomographic image as a reference, but is not limited thereto, and flattening processing may be performed based on the RPE (retinal pigment epithelial cell) layer, the center of the retina, or other references.

[0208] In the flattening process according to this embodiment, the processing unit 222 checks pixel values ​​from the top in the Z direction for each A-scan data in the tomographic image, and determines the first pixel having a pixel value that is not a background value as the pixel corresponding to the top position of the retina. The processing unit 222 moves the A-scan data in the tomographic image so as to align the top positions of the retina in all A-scan data in the transverse direction of the tomographic image. The processing unit 222 may also check pixel values ​​from the top in the Z direction for the A-scan data of the tomographic image, and consider a pixel with a large change in pixel value between adjacent pixels as an ILM, and align the A-scan data based on the position of that pixel. The processing unit 222 also stores information on the amount of movement of each A-scan data in step S2301 in the storage unit 24.

[0209] Here, Fig. 24(a) shows an example of a tomographic image 2401 to be processed and generated in step S1130. Also, a tomographic image 2402 shown in Fig. 24(b) is an example of a tomographic image obtained by performing a flattening process on the tomographic image 2401. Referring to Fig. 24(b), it can be seen that the position of the upper retinal layer (ILM) is aligned in each A-scan data.

[0210] In step S2302, the processing unit 222 divides the flattened tomographic image 2402 into a plurality of stripes in the transverse direction. The process in step S2302 is similar to the process in step S1601, and therefore the description will be omitted. Here, each of the stripes 2421 to 2427 in the tomographic image 2402 shown in FIG. 24(b) shows an example of the stripes divided in step S2302.

[0211] In step S2303, the processing unit 222 detects an area of ​​interest from each stripe. The process in step S2303 is similar to the process in step S1602, and therefore the description will be omitted. Here, Fig. 24(c) shows examples of areas of interest 2431 to 2437 in a tomographic image 2403 in which area of ​​interest detection is performed, which are detected in step S2303.

[0212] In step S2304, the processing unit 222 packs the images of the multiple regions of interest extracted from the multiple stripes in step S2303 into one image to generate a composite image. The process in step S2304 is similar to the process in step S1603, so a description thereof will be omitted. Figure 24(d) shows an example of the composite image 2404 generated in step S2304.

[0213] In step S2305, the processing unit 222 performs segmentation processing on the composite image generated in step S2404 using the learned model, and obtains a boundary image corresponding to the composite image. The processing in step S2305 is similar to the processing in step S1604, and therefore a description thereof will be omitted. However, in this embodiment, since the boundary image is used as output data of the learning data, the processing unit 222 obtains a boundary image corresponding to the composite image.

[0214] In step S2306, the processing unit 222 unpacks the boundary image of each region of interest from the boundary image acquired in step S2305, and overwrites the boundary image of the region of interest at the position of the region of interest in the stripe corresponding to the region of interest. This allows the processing unit 222 to acquire a detection result image of each stripe. The process in step S2306 is similar to the process in step S1605, and therefore a description thereof will be omitted.

[0215] In step S2307, the processing unit 222 performs stripe combining processing on the stripe detection result image obtained in step S2306 to obtain a detection result image related to the flattened tomographic image. The processing in step S2307 is similar to the processing in step S1606, and therefore a description thereof will be omitted. Fig. 24(e) shows an example of a detection result image 2405 related to the flattened tomographic image.

[0216] In step S2308, the processing unit 222 performs an inverse flattening process on the region detection result tomographic image obtained in step S2307. Here, the inverse flattening process according to this embodiment is a process of returning the region of interest (retinal region) to its original position in the depth direction of the tomographic image in A-scan units. Therefore, the processing unit 222 moves each A-scan data in the detection result image related to the flattened tomographic image using information on the movement amount of each A-scan data stored in the storage unit 24 in step S2301. FIG. 24(f) shows an example of a detection result image 2406 generated by performing an inverse flattening process on the detection result image 2405.

[0217] In step S2309, the processing unit 222 performs post-processing of the detection result image of step S1606. The processing in step S2309 is similar to the processing in step S1607, and therefore a description thereof will be omitted. However, in this embodiment, since a boundary image is acquired as the detection result image, the post-processing in step S2309 may be processing such as connecting broken boundary lines instead of filling holes. Such processing may be performed using any known method.

[0218] In step S2310, the processing unit 222 detects the area of ​​the retina layer in each tomographic image based on the detection result image post-processed in step S2309. The processing in step S2310 may be the same as the processing in step S1608. However, in this embodiment, since a boundary image is acquired as the detection result image, the processing unit 222 may perform rule-based processing on the boundary image that is the detection result image to detect the area between each boundary.

[0219] In addition, the processing unit 222 may detect each boundary of the retinal layer from the detection result image, which is the boundary image, at this stage. At this time, if the boundary shown in the detection result image is not labeled, the processing unit 222 may perform rule-based processing to detect which boundary of the retinal area each boundary indicates. When the area of ​​the retinal layer is detected in step S2310, the detection process of the retinal layer in this embodiment ends, and the process proceeds to step S1150.

[0220] As described above, the processing unit 222 according to the present embodiment aligns the depth-direction position of the retina or the region of interest in a two-dimensional tomographic image, or in each of a plurality of regions obtained by dividing the two-dimensional tomographic image in a direction intersecting the depth direction of the subject's eye. The processing unit 222 also detects at least one of a layer and a boundary in the tomographic image acquired by the acquisition unit 21, using the detection result obtained using the trained model and the amount of movement of the position when the depth-direction position of the retina or the region of interest is aligned.

[0221] With the above configuration, the image processing device according to the present embodiment can reduce the influence of the curvature and inclination of the retina by performing flattening processing of the regions of interest on the tomographic image or the stripe image. Therefore, more regions of interest can be packed into the stripe that is the target of the region detection (segmentation) processing, and the data to be subjected to the region detection (segmentation) processing can be further reduced, and the detection efficiency (speed) can be further improved.

[0222] The flattening process according to the present embodiment is performed on the tomographic image to be processed. Alternatively, the processing unit 222 may perform the flattening process on the stripe image after the stripe division process in step S2302. In this case, the inverse flattening process can be performed after the unpacking process in step S2306.

[0223] Fifth embodiment In the second embodiment, a method for further improving detection efficiency by dividing a tomographic image to be processed into a plurality of stripes and further reducing the granularity (size) of data to be subjected to region detection processing has been described. In contrast, in the fifth embodiment of the present invention, an image of a region of interest is analyzed, and when it is determined that segmentation processing using a trained model is difficult, the region of interest is excluded from segmentation processing using a trained model.

[0224] The image processing device according to this embodiment will be described below, focusing on the differences from the image processing device according to the second embodiment. Since the configuration of the image processing device according to this embodiment is similar to the configuration of the image processing device 20 according to the second embodiment, the same reference symbols are used for the respective components and the description will be omitted. Furthermore, since the processing according to this embodiment is similar to the processing according to the second embodiment shown in FIG. 16 except for the processing in step S1602, the same reference symbols are used and the description will be omitted.

[0225] In this embodiment, in step S1602, the processing unit 222 detects a region of interest from a tomographic image, and determines whether or not to exclude the detected region of interest from segmentation processing using a trained model. The detection processing of the region of interest in step S1602 is the same as the processing in step S1601 according to the second embodiment, and therefore a description thereof will be omitted. Below, the determination in step S1602 according to this embodiment as to whether or not to exclude the detected region of interest from segmentation processing using a trained model will be described.

[0226] The processing unit 222 first determines whether the pixel values ​​are low or there is a lot of noise in the detected region of interest. Specifically, the processing unit 222 calculates a signal-to-noise ratio (SNR) for pixel values ​​(signal values) in the region of interest. If the calculated SNR is equal to or less than a threshold, the processing unit 222 determines that the region of interest is to be excluded from segmentation processing using the trained model. The processing unit 222 may also calculate the maximum value of pixel values ​​(signal values) in the region of interest, and if the maximum value is equal to or less than a threshold, determine that the region of interest is to be excluded from segmentation processing using the trained model.

[0227] Furthermore, the processing unit 222 determines whether or not a virtual image (aliased image) of a tomographic image is included in the region of interest. A known method may be used as a specific determination method. When a virtual image of a tomographic image is included in the region of interest, the processing unit 222 determines that the region of interest is to be excluded from the segmentation process using the trained model. Note that the processing unit 222 may analyze the entire tomographic image to be processed in order to determine the presence of a virtual image in the region of interest.

[0228] In step S1602, the region of interest determined to be excluded from the segmentation process is not used in the packing process in step S1603. However, the processing unit 222 may perform rule-based segmentation process for the region of interest instead of the segmentation process using the trained model. Such segmentation process may be performed by any known method.

[0229] The display control unit 25 may present that the attention region excluded from the segmentation process using the learned model in step S1602 is a region that has not been segmented in the display process in step S1150. In this case, the processing unit 222 may attach a label indicating that the attention region is an unreliable region (region without segmentation process) in the detection result image.

[0230] As described above, the processing unit 222 according to this embodiment does not use the image of interest for generating a synthetic image when the signal-to-noise ratio or maximum value of pixel values ​​in the detected region of interest is smaller than a threshold value or when aliasing images are included in the region of interest. With such a configuration, the image processing device according to this embodiment determines whether or not segmentation processing using a trained model is difficult based on the image of the region of interest. The processing unit 222 can avoid unnecessary processing by excluding the region of interest for which it is determined that segmentation processing using a trained model is difficult from segmentation processing using a trained model, thereby improving detection efficiency (speed).

[0231] Sixth embodiment In the second embodiment, a method was described in which the tomographic image to be processed is divided into multiple stripes to further reduce the granularity (size) of the data to be subjected to the region detection process, thereby further improving the detection efficiency. However, when the retina is enlarged due to disease or the like, the region of interest becomes large, and the number of regions of interest that can be packed into one composite image in the region of interest packing process decreases, which may result in a decrease in processing efficiency. In contrast, in the sixth embodiment of the present invention, the region of interest is analyzed, and if the region of interest is large, the process switches to rule-based segmentation processing.

[0232] Hereinafter, the image processing device according to the sixth embodiment will be described with reference to Fig. 25, focusing on the differences from the image processing device according to the second embodiment. Since the configuration of the image processing device according to this embodiment is similar to that of the image processing device according to the second embodiment, the same reference numerals are used for the respective components and the description thereof will be omitted. Furthermore, since the series of processes according to this embodiment are similar to the series of processes according to the second embodiment except for the process in step S1140, the description of the processes other than the process in step S1140 will be omitted.

[0233] A specific procedure of the retinal layer detection process in step S1140 according to this embodiment will be described with reference to Fig. 25. Fig. 25 is a flowchart of the retinal layer detection process according to this embodiment. In this embodiment, when the retinal layer detection process is started in step S1140, the process proceeds to step S2501.

[0234] In step S2501, the processing unit 222 divides the tomographic image into a plurality of stripes in the transverse direction. The process in step S2501 is similar to the process in step S1601, and therefore the description will be omitted.

[0235] In step S2502, the processing unit 222 detects an area of ​​interest from the tomographic image. The process in step S2502 is similar to the process in step S1602, and therefore a description thereof will be omitted.

[0236] In step S2503, the processing unit 222 analyzes the region of interest detected in step S2502. In this embodiment, the processing unit 222 analyzes the image of the region of interest and calculates the size of the region of interest in the Z direction (depth direction).

[0237] In step S2504, the processing unit 222 selects whether to perform segmentation using a trained model or rule-based segmentation (second segmentation) using the analysis result of the region of interest in step S2503. The selection criterion here may be, for example, whether the size of the region of interest in the Z direction is smaller than one tenth of the size of the stripe (or tomographic image) in the depth direction.

[0238] When the size of the region of interest in the Z direction is smaller than one tenth of the size of the stripe in the depth direction, the processing unit 222 selects the segmentation process using the trained model. When the size of the region of interest in the Z direction is one tenth or more of the size of the stripe in the depth direction, the processing unit 222 selects the rule-based segmentation process. When the processing unit 222 selects the segmentation process using the trained model in step S2504, the process proceeds to step S2505.

[0239] In step S2505, the processing unit 222 performs packing processing on the multiple regions of interest extracted from the multiple stripes in step S2502 to generate a composite image. The processing in step S2505 is similar to the processing in step S1603, and therefore a description thereof will be omitted.

[0240] In step S2506, the processing unit 222 performs segmentation processing on the composite image generated in step S2505 using the learned model. The processing in step S2506 is similar to the processing in step S1604, and therefore the description will be omitted.

[0241] In step S2507, the processing unit 222 performs an unpacking process on the label image acquired in step S2506 to acquire a detection result image corresponding to each of the regions of interest. The process in step S2507 is similar to the process in step S1605, and therefore a description thereof will be omitted.

[0242] On the other hand, if the processing unit 222 selects the rule-based segmentation process in step S2504, the process proceeds to step S2508. In step S2508, the processing unit 222 performs the rule-based segmentation process (second segmentation process) as an alternative process based on the analysis result of the region of interest in step S2504, and obtains a label image in the region of interest.

[0243] Furthermore, in step S2508, the processing unit 222 performs processing on the acquired label image of the region of interest similar to the processing of returning the label image of the region of interest to the position of the region of interest in the corresponding stripe in the unpacking processing in step S2507. This allows the processing unit 222 to acquire a detection result image for the stripe corresponding to the region of interest by using the label image of the region of interest acquired by the rule-based segmentation processing.

[0244] In step S2509, the processing unit 222 performs stripe joining processing on the detection result images acquired in steps S2507 and S2508 to acquire a detection result image of the tomographic image to be processed. The processing in step S2509 is similar to the processing in step S1606, and therefore a description thereof will be omitted.

[0245] In step S2510, the processing unit 222 performs post-processing on the region extraction result image acquired in step S2509. The process in step S2510 is similar to the process in step S1607, and therefore a description thereof will be omitted.

[0246] In step S2511, the processing unit 222 detects the area of ​​the retina layer in each tomographic image based on the detection result image post-processed in step S2510. The process in step S2511 is similar to the process in step S1608, so the description is omitted. When the area of ​​the retina layer is detected in step S2511, the detection process of the retina layer in this embodiment ends, and the process proceeds to step S1150.

[0247] As described above, the processing unit 222 according to this embodiment uses the pixel values ​​in the detected region of interest and a threshold value to select the processing to be performed on the detected region of interest from processing using a trained model and rule-based processing. In other words, the processing unit 222 analyzes the detected region of interest, and selects the processing to be performed on the detected region of interest from processing using a trained model and rule-based processing based on the analysis result. In particular, in this embodiment, the processing unit 222 selects the rule-based processing when the region of interest corresponding to the analysis result is larger than the threshold value.

[0248] With the above configuration, the image processing device according to this embodiment analyzes the region of interest, and when the region of interest is large and the number of regions of interest that can be packed into one composite image is small in the region of interest packing process, performs rule-based segmentation processing. This makes it possible to efficiently and quickly perform the entire segmentation process even when the region of interest is large, such as when the retina is enlarged due to disease.

[0249] In this embodiment, the selection criterion in step S2504 is whether the size of the region of interest in the Z direction is smaller than one tenth of the depth of the stripe. However, the selection criterion in step S2504 is not limited to this. The selection criterion may be, for example, whether the size of the region of interest in the Z direction is smaller than one fifth of the depth of the stripe, and the size criterion may be set arbitrarily according to a desired configuration.

[0250] In addition, in this embodiment, a labeled image is obtained by performing rule-based segmentation processing. However, the result of retinal layer area detection by rule-based segmentation processing does not need to be an image, and may be information that can identify each area or boundary. In this case, the processing unit 222 may ultimately integrate information for identifying areas and boundaries obtained by rule-based segmentation processing with the result of segmentation processing using a trained model.

[0251] Seventh embodiment In the sixth embodiment, a method for selecting a rule-based segmentation process when it is determined that a region of interest is large by analyzing the region of interest has been described. In contrast, in the seventh embodiment of the present invention, a region of interest is analyzed, and a segmentation process is selected based on the part of the retina.

[0252] Hereinafter, with reference to FIG. 25, the image processing device according to the seventh embodiment will be described, focusing on the differences from the image processing device according to the modified example of the sixth embodiment. Since the configuration of the image processing device according to this embodiment is similar to that of the image processing device according to the sixth embodiment, the same reference numerals are used for the respective components and the description thereof will be omitted. Furthermore, since the processing according to this embodiment is similar to the processing according to the sixth embodiment except for steps S2503, S2504, and S2508, the description thereof will be omitted. In the retinal layer detection processing according to this embodiment, when the region of interest is detected in step S2502, the processing proceeds to step S2503.

[0253] In step S2503, the processing unit 222 analyzes the region of interest detected in step S2502. Specifically, the processing unit 222 analyzes the part of the region of interest and judges whether the part of the region of interest is the optic disc. Here, the processing unit 222 uses the front fundus image of the subject's eye acquired by the acquisition unit 21 to judge whether the part of the region of interest is the optic disc. Note that the front fundus image may be acquired by photographing the subject's eye using the fundus image photographing device 30 when photographing the tomographic image in step S1120. Note that, when the tomographic image to be processed is acquired from the external storage device 40 or the like, the front fundus image acquired when photographing the tomographic image may be used.

[0254] The processing unit 222 aligns the front fundus image and the tomographic image according to the photographing conditions of the tomographic image and the front fundus image, and recognizes the position of the optic disc in the tomographic image. For example, the processing unit 222 may recognize the optic disc on the front fundus image based on a luminance value specific to the optic disc on the front fundus image. The processing unit 222 may also recognize, for example, the location where the optic nerve passes as the position of the optic disc while checking the shape of the optic disc in the tomographic image. These processes may be performed using any known process.

[0255] In step S2504, the processing unit 222 selects a segmentation process using the analysis result of the region of interest in step S2503. The selection criterion can be whether or not the region of interest includes the optic disc. If the processing unit 222 determines that the region of interest includes the optic disc, it selects the segmentation process using the trained model. On the other hand, if the processing unit 222 determines that the region of interest does not include the optic disc, it selects the rule-based segmentation process. If the processing unit 222 selects the segmentation process using the trained model in step S2504, the process proceeds to step S2505. The processes in steps S2505 to S2507 are similar to the processes in the same steps according to the sixth embodiment, and therefore will not be described.

[0256] In step S2504, if the processing unit 222 selects the rule-based segmentation processing, the process proceeds to step S2508. In step S2508, the processing unit 222 performs a rule-based segmentation processing (second segmentation processing) on ​​the optic disc based on the analysis result of the region of interest in step S2504, and obtains a label image in the region of interest. The rule-based segmentation processing on the optic disc may be performed using a known method.

[0257] In step S2508, the processing unit 222 performs the same process for the label image of the acquired region of interest as the process for returning the label image of the region of interest to the position of the region of interest in the corresponding stripe in the unpacking process in step S2507. This allows the processing unit 222 to use the label image of the region of interest acquired by the rule-based segmentation process to acquire a detection result image for the stripe corresponding to the region of interest. The subsequent processes are the same as those in the sixth embodiment, and therefore will not be described.

[0258] In this embodiment, when the region of interest is the optic disc, the rule-based segmentation process is performed on the region of interest. However, the region of interest on which the rule-based segmentation process is performed is not limited to the optic disc. For example, when the region of interest is another region such as the fovea, the rule-based segmentation process may be performed. In addition, for example, the processing unit 222 may analyze whether the region of interest includes a diseased area, and select the rule-based segmentation process when the region of interest includes a diseased area.

[0259] As described above, the processing unit 222 according to the present embodiment selects the rule-based processing based on the part of the region of interest to be analyzed. In addition, the processing unit 222 may select the rule-based processing when diseased parts exist in multiple regions of interest.

[0260] With such a configuration, the image processing device 20 according to the present embodiment can analyze the region of interest and appropriately select the segmentation process based on the retinal part or disease state, thereby enabling the entire segmentation process to be performed efficiently and quickly.

[0261] In this embodiment, a labeled image is obtained by performing rule-based segmentation processing. However, the result of retinal layer area detection by rule-based segmentation processing does not need to be an image, and may be information that can identify each area or boundary. In this case, the processing unit 222 may finally integrate the information for identifying the area or boundary obtained by the rule-based segmentation processing with the result of the segmentation processing using the trained model.

[0262] In the various embodiments and modifications described above, the processing unit 222 is configured to perform stripe division / integration processing, attention area detection processing, segmentation selection processing, packing / unpacking processing, segmentation processing using a trained model, and the like. However, components that perform these processes may be provided separately. In this case, the processing unit 222 may be configured to include such separately provided components.

[0263] In the above-described various embodiments and modified examples, the size of the image input to the trained model and the size of the image used as the input data of the training data are made to match. However, depending on the configuration of the machine learning model, the sizes of these images do not need to match. For example, in CNN and the like, since processing is performed according to the kernel size of the filter, it is expected that processing is possible even if the size of the image input to the trained model and the image size of the input data of the training data do not match. For this reason, the size of the image input to the trained model may be arbitrary, and the image size of the input data of the training data may also be arbitrary.

[0264] For this reason, for example, when multiple attention areas to be processed are packed into a composite image and one attention area is ultimately left, the processing unit 222 may input the remaining one attention area to the trained model. In relation to this, the training data may include a pair group in which an image of one attention area is used as input data and a label image corresponding to the image of the attention area is used as output data, so that the one attention area can be processed by the trained model.

[0265] In the various embodiments and modifications described above, the acquisition unit 21 acquires the interference signal acquired by the OCT device 10, the tomographic data generated by the tomographic image generating unit 221, and the like. However, the configuration in which the acquisition unit 21 acquires these signals and images is not limited to this. For example, the acquisition unit 21 may acquire these signals and data from an external storage device 40, a server, an imaging device, and the like connected to the image processing device 20 via a LAN, a WAN, the Internet, or the like.

[0266] The trained models according to the various embodiments and modifications described above can be provided in the image processing device 20. The trained models may be configured, for example, by a software module executed by a processor such as a CPU, MPU, GPU, or FPGA, or may be configured by a circuit that performs a specific function such as an ASIC. These trained models may also be provided in another server device connected to the image processing device 20. In this case, the image processing device 20 can use the trained model by connecting to a server or the like that has the trained model via any network such as the Internet. Here, the server that has the trained model may be, for example, a cloud server, a fog server, or an edge server. In addition, the training data of the trained model is not limited to data obtained using the OCT device 10 itself that performs actual shooting, and may be data obtained using the same type of OCT device or data obtained using the same type of OCT device depending on the desired configuration.

[0267] In addition, the GPU can perform efficient calculations by processing more data in parallel. Therefore, when learning is performed multiple times using a learning model such as deep learning, it is effective to perform the processing with the GPU. Therefore, the processing by the processing unit 222, which is an example of a learning unit (not shown), may use a GPU in addition to a CPU. In this case, when a learning program including a learning model is executed, the CPU and the GPU cooperate to perform calculations to perform learning. In addition, the processing by the learning unit may be performed by only the CPU or the GPU. In addition, the processing unit (estimation unit) that executes the processing using the above-mentioned various learned models may also use a GPU like the learning unit. In addition, the learning unit may include an error detection unit and an update unit (not shown). The error detection unit obtains an error between the output data output from the output layer of the neural network according to the input data input to the input layer and the correct answer data. The error detection unit may calculate the error between the output data from the neural network and the correct answer data using a loss function. In addition, the update unit updates the coupling weighting coefficient between the nodes of the neural network, etc., based on the error obtained by the error detection unit so that the error becomes smaller. The update unit updates the connection weighting coefficients, etc., using, for example, the backpropagation method. The backpropagation method is a method for adjusting the connection weighting coefficients, etc., between the nodes of each neural network so as to reduce the above-mentioned error.

[0268] In the above-mentioned various embodiments and modifications, the motion contrast data is data showing a change between a plurality of volume data obtained by controlling the measurement light to scan the same region (same position) of the subject's eye a plurality of times. At this time, the volume data is composed of a plurality of tomographic images obtained at different positions. Then, by obtaining data showing a change between a plurality of tomographic images obtained at approximately the same position at each of the different positions, the motion contrast data can be obtained as volume data. The motion contrast front image is also called an OCTA front image (OCTA En-Face image) related to OCT angiography (OCTA) that measures the movement of blood flow, and the motion contrast data is also called OCTA data. The motion contrast data can be obtained, for example, as a decorrelation value, a variance value, or a value obtained by dividing the maximum value by the minimum value (maximum value / minimum value) between two tomographic images or the interference signals corresponding thereto, and may be obtained by any known method. At this time, the two tomographic images can be obtained, for example, by controlling the measurement light to scan the same region (same position) of the subject's eye a plurality of times.

[0269] Moreover, the En-Face image is, for example, a front image generated by projecting data in a range between two layer boundaries in the XY direction. At this time, the front image is generated by projecting or integrating data corresponding to a depth range determined based on two reference planes, which is at least a part of the depth range of volume data (three-dimensional tomographic image) obtained using optical interference, onto a two-dimensional plane. The En-Face image is a front image generated by projecting data corresponding to a depth range determined based on a detected retinal layer, out of the volume data, onto a two-dimensional plane. Note that, as a method for projecting data corresponding to a depth range determined based on two reference planes onto a two-dimensional plane, for example, a method of setting a representative value of data within the depth range as a pixel value on the two-dimensional plane can be used. Here, the representative value can include values ​​such as an average value, a median value, or a maximum value of pixel values ​​within a range in the depth direction of the region surrounded by the two reference planes. Furthermore, the depth range related to the En-Face image may be, for example, a range including a predetermined number of pixels in a deeper or shallower direction based on one of two layer boundaries related to the detected retinal layer. In addition, the depth range for the En-Face image may be, for example, a range that is changed (offset) from the range between two layer boundaries for the detected retinal layers in response to an operator's instruction.

[0270] In addition, any known OCT device may be used as the OCT device according to the above-mentioned various embodiments and modified examples. For example, the OCT device may be any other type of OCT device, such as a Spectral Domain OCT (SD-OCT) device using an SLD as a light source, or a Wavelength Swept OCT (SS-OCT) device using a wavelength swept light source capable of sweeping the wavelength of emitted light. In addition, the OCT device may be a Line-OCT device (or an SS-Line-OCT device) using a line light, or a Full Field-OCT device (or an SS-Full Field-OCT device) using an area light. Furthermore, the OCT device may be a Wavefront Compensated OCT (AO-OCT) device using a wavefront adaptive optics system, or a Polarized OCT (PS-OCT) device for visualizing information related to polarization phase difference and depolarization.

[0271] (Other Examples) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments and modifications is provided to a system or device via a network or a storage medium, and the computer of the system or device reads and executes the program. The computer has one or more processors or circuits, and may include multiple separate computers or a network of multiple separate processors or circuits for reading and executing computer-executable instructions.

[0272] The processor or circuitry may include a central processing unit (CPU), a microprocessing unit (MPU), a graphics processing unit (GPU), an application specific integrated circuit (ASIC), or a field programmable gateway (FPGA), and the processor or circuitry may include a digital signal processor (DSP), a data flow processor (DFP), or a neural processing unit (NPU).

[0273] Although the present invention has been described above with reference to the embodiments and modifications, the present invention is not limited to the above embodiments and modifications. The present invention also includes inventions that have been modified within the scope of the present invention and inventions equivalent to the present invention. In addition, the above-mentioned embodiments and modifications can be appropriately combined within the scope of the present invention. [Explanation of symbols]

[0274] 20: medical image processing device, 21: acquisition unit, 222: processing unit

Claims

1. An acquisition unit for acquiring a tomographic image of the subject's eye; a processing unit for detecting at least one of a layer and a boundary in the acquired tomographic image; Equipped with The processing unit includes: generating a composite image by arranging and synthesizing images of a plurality of partial regions in the tomographic image acquired by the acquisition unit; Detecting at least one of layers and boundaries in the synthetic image using a trained model; Detecting at least one of a layer and a boundary in the acquired tomographic image using a detection result obtained using the trained model; A medical image processing device, wherein each of the plurality of partial regions is a region of interest including a retina region and a choroid.

2. The medical image processing apparatus according to claim 1 , wherein the processing unit generates a composite image by arranging the images of the plurality of partial regions in a depth direction of the subject's eye and combining them.

3. The medical image processing apparatus according to claim 1 , wherein the tomographic image is a two-dimensional tomographic image or a three-dimensional tomographic image including a plurality of two-dimensional tomographic images.

4. The processing unit includes: Dividing the two-dimensional tomographic image into a plurality of regions in a direction intersecting a depth direction of the subject's eye, and detecting a partial region from each of the divided plurality of regions; or The medical image processing apparatus according to claim 3 , further comprising: detecting a partial region from the two-dimensional tomographic image; and dividing an image of the detected partial region into a plurality of partial regions in a direction intersecting the depth direction.

5. The medical image processing apparatus according to claim 4 , wherein the divided regions or the divided partial regions have regions that overlap with adjacent regions in the divided regions or the divided partial regions.

6. The medical image processing apparatus according to claim 5 , wherein the processing unit thins out information about the overlapping regions in the divided regions or the divided partial regions.

7. 7. The medical image processing device according to claim 5, wherein the processing unit adds an inverted region of a region that has no adjacent regions in the divided regions or the divided partial regions as the overlapping region.

8. The medical image processing device according to claim 4 , wherein the acquisition unit acquires a two-dimensional tomographic image in which a direction intersecting the depth direction is a scanning direction in which a number of samples is small in a three-dimensional tomographic image of the subject's eye.

9. 8. The medical image processing device according to claim 4, wherein the acquisition unit acquires a two-dimensional tomographic image in which a direction intersecting the depth direction is a scanning direction in which the size of an imaging area in a three-dimensional tomographic image of the test eye or the curvature of the retina is small.

10. The processing unit includes: Aligning a position in a depth direction of a retina or a partial region in the two-dimensional tomographic image or in each of a plurality of regions obtained by dividing the two-dimensional tomographic image in a direction intersecting a depth direction of the subject's eye; The medical image processing device according to any one of claims 3 to 9, wherein at least one of a layer and a boundary in the acquired tomographic image is detected using a detection result obtained using the trained model and a positional movement amount when the depth direction position of the retina or a partial region is aligned.

11. The medical image processing apparatus according to claim 3 , wherein the processing unit detects a plurality of partial regions in the plurality of two-dimensional tomographic images by using the plurality of two-dimensional tomographic images.

12. The medical image processing apparatus according to claim 11 , wherein the processing unit detects one partial region from each of the plurality of two-dimensional tomographic images.

13. The medical image processing apparatus according to claim 11 , wherein the processing unit detects a plurality of partial regions from each of the plurality of two-dimensional tomographic images.

14. The medical image processing device according to any one of claims 1 to 13, wherein the processing unit inputs the generated synthetic image to a trained model obtained by using a synthetic image obtained by synthesizing images of multiple partial regions of a tomographic image of the test eye and a label image indicating at least one of a region and a boundary in the synthetic image as training data, and obtains a label image indicating at least one of a region and a boundary in the generated synthetic image.

15. The medical image processing apparatus according to claim 1 , wherein the processing unit detects the plurality of partial regions by using pixel values ​​in the acquired tomographic image and a threshold value.

16. The medical image processing apparatus according to claim 1 , wherein the processing unit detects the plurality of partial regions by rule-based processing of the acquired tomographic image.

17. The medical image processing apparatus according to claim 1 , wherein the subregion includes a retina.

18. The processing unit includes: Obtaining detection results in each sub-region using the detection results obtained using the trained model; The medical image processing apparatus according to claim 1 , further comprising: detecting at least one of a layer and a boundary in a tomographic image corresponding to each partial region by using the obtained detection result in each partial region.

19. 10. The medical image processing device according to claim 4, wherein the processing unit selects a segmentation process for the detected partial region from among a segmentation process using the trained model and a rule-based segmentation process, using the detected partial region and a threshold value.

20. 10. The medical image processing device according to claim 4, wherein the processing unit analyzes the detected partial region, and selects a segmentation process for the detected partial region from a segmentation process using the trained model and a rule-based segmentation process based on an analysis result.

21. The medical image processing apparatus according to claim 20 , wherein the processing unit selects the rule-based processing when a partial region corresponding to the analysis result is larger than a threshold value, or when a diseased part is present in the partial region corresponding to the analysis result.

22. The medical image processing apparatus according to claim 20 , wherein the processing unit selects the rule-based processing based on a part of a partial region corresponding to the analysis result.

23. 19. The medical image processing device according to claim 1, wherein the processing unit does not use an image of a detected partial region for generating a composite image when a signal-to-noise ratio or a maximum value of pixel values ​​in the detected partial region is smaller than a threshold value or when the partial region contains an aliasing image.

24. A medical image processing device as described in any one of claims 1 to 23, wherein the trained model is obtained by training using as training data a synthetic image obtained by arranging and synthesizing images of multiple partial regions including the retinal region and choroid in a tomographic image of the test eye, and a label image indicating at least one of the regions and boundaries in the synthetic image.

25. Obtaining a tomographic image of the subject's eye; generating a composite image by arranging and combining images of a plurality of partial regions in the acquired tomographic image; detecting at least one of layers and boundaries in the synthetic image using a trained model; Detecting at least one of a layer and a boundary in the acquired tomographic image using a detection result obtained using the trained model; Including, A medical image processing method, wherein each of the plurality of partial regions is a region of interest including a retina region and a choroid region.

26. A program that, when executed by a computer, causes the computer to execute each step of the medical image processing method according to claim 25.

Images