IMAGE PROCESSING APPARATUS, TRAINED MODEL, IMAGE PROCESSING METHOD, AND PROGRAM

Abstract

To provide information more appropriate for assisting eye diagnostic by an examiner than that in a prior art.SOLUTION: An image processing device includes acquisition means which acquires data related to tomography containing a plurality of types of analysis values of an optometry object eye and being related to first tomography of the optometry object eye associated to a first time point, and data of second tomography of the optometry object eye associated to a second time point after the first time point; and generation means which generates, from the required data related to the first tomography and the required data related to the second tomography, at least one of the plurality of types of analysis values of the optometry object eye at a third time point after the second time point using a learned model.SELECTED DRAWING: Figure 7

Description

[Technical field]

[0001] The present invention relates to an image processing device, a trained model, an image processing method, and a program. [Background technology]

[0002] Tomographic imaging devices for the eye, such as devices using optical coherence tomography (OCT devices), are capable of three-dimensionally observing the state of the inside of the retina. These tomographic imaging devices have been attracting attention in recent years because they are useful for more accurate diagnosis of diseases.

[0003] One form of OCT is TD-OCT (Time domain OCT), which combines a broadband light source with a Michelson interferometer. This is configured to obtain information in the depth direction by scanning the delay of the reference arm to measure the interference light with the backscattered light of the signal arm. However, it is difficult to obtain images at high speed with this type of TD-OCT.

[0004] On the other hand, as a method for acquiring images at higher speeds, Spectral domain OCT (SD-OCT) is known, which uses a broadband light source and acquires an interferogram with a spectrometer. Also, Swept Source OCT (SS-OCT) is known, which uses a high-speed wavelength swept light source as a light source and measures the spectral interference with a single channel photodetector.

[0005] By using a tomographic image taken by OCT to measure the thickness of the nerve fiber layer, it is possible to quantitatively diagnose the progression of diseases such as glaucoma and the state of recovery after treatment. Patent Document 1 discloses an ophthalmic analysis device that creates a time series graph of analytical values ​​such as film thickness obtained from a tomographic image of the fundus. This ophthalmic analysis device predicts future analytical values ​​from a regression line obtained from the time series graph. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2017-170193 A Summary of the Invention [Problem to be solved by the invention]

[0007] However, depending on the type of lesion or abnormality occurring in the eye, the eye may exhibit a combination of characteristics rather than a single characteristic, and predictions based on specific analytical values ​​may not be sufficient to aid in diagnosis.

[0008] An object of the present invention is to provide more suitable information than ever before when assisting an examiner in making an eye diagnosis, in view of the above-mentioned problems. [Means for solving the problem]

[0009] In order to achieve the above object, an image processing device according to one aspect of the present invention comprises: Regarding tomography including multiple types of analysis values ​​of the subject eye The imaging conditions, including the imaging area, are associated with the an acquiring means for acquiring data relating to a first cross-section of the subject's eye associated with a first time and data relating to a second cross-section of the subject's eye associated with a second time after the first time; A plurality of devices for acquiring tomographic data of the subject's eye photograph conditions Each of Among the trained models corresponding to At least the imaging region of the plurality of imaging conditions is the same. The acquired data on the first fault and the acquired data on the second fault photograph A selection means for selecting a trained model corresponding to the condition; A generating means for generating at least one of a plurality of types of analysis values ​​of the subject's eye at a third time after the second time from the acquired data on the first cross-section and the acquired data on the second cross-section by using the selected trained model; The present invention is characterized by comprising: Effect of the Invention

[0010] According to one aspect of the present invention, when assisting an examiner in making an eye diagnosis, more suitable information than in the past can be provided. [Brief description of the drawings]

[0011] [Figure 1] 1 illustrates an example of a schematic configuration of an image processing system according to a first embodiment. [Diagram 2] 1A and 1B are diagrams for explaining the structure of an eye, a tomographic image, and a fundus image. [Diagram 3] 2 is a flowchart showing a series of processes according to the first embodiment. [Figure 4] FIG. 11 is a diagram for explaining an example of learning data. [Diagram 5] FIG. 1 is a diagram for explaining an example of a machine learning model. [Figure 6] FIG. 1 is a diagram for explaining an example of a machine learning model. [Figure 7] FIG. 11 is a diagram for explaining a display screen of a processing result according to the first embodiment. [Figure 8] 13 illustrates an example of a schematic configuration of an image processing system according to a second embodiment. [Figure 9] 13 illustrates an example of a schematic configuration of an image processing system according to a third embodiment. [Figure 10] FIG. 11 is a diagram for explaining a display screen of a processing result according to the third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, exemplary embodiments for carrying out the present invention will be described in detail with reference to the drawings. However, the dimensions, materials, shapes, and relative positions of components described in the following embodiments are arbitrary and can be changed according to the configuration of the device to which the present invention is applied or various conditions. In addition, the same reference numerals are used between the drawings to indicate elements that are the same or functionally similar.

[0013] Example 1 Hereinafter, an image processing system for presenting information that aids in diagnosis of a subject's eye according to a first embodiment of the present invention will be described with reference to Figs. 1 to 7. In this embodiment, a trained model related to a machine learning model is used to generate a future analysis value of the subject's eye. Specifically, a trained model obtained by learning using multiple types of analysis values ​​obtained from a tomographic image of the subject's eye is used to predict a future analysis value and present the result.

[0014] Here, a machine learning model refers to a learning model based on a machine learning algorithm such as deep learning. A trained model is a model that has been obtained (learned) by training a machine learning model based on an arbitrary machine learning algorithm using appropriate learning data in advance. However, although a trained model has been obtained in advance using appropriate learning data, this does not mean that no further learning is performed, and additional learning can also be performed.

[0015] Fig. 1 shows an example of a schematic configuration of an image processing system 1 including an image processing device 300 according to the present embodiment. As shown in Fig. 1, the image processing system 1 includes an OCT device (hereinafter, OCT device 200) as an example of a tomographic image capturing device, the image processing device 300, a fundus image capturing device 400, an external storage device 500, a display unit 600, and an input unit 700.

[0016] The OCT device 200 is used as a device for capturing a tomographic image of a subject's eye. Any type of OCT device can be used, for example, SD-OCT or SS-OCT.

[0017] The image processing device 300 is connected to the OCT device 200, the fundus image capturing device 400, the external storage device 500, the display unit 600, and the input unit 700 via an interface, and can control these. The image processing device 300 can generate various images such as a tomographic image and an En-Face image (front image) of the subject's eye based on various signals acquired from the OCT device 200, the fundus image capturing device 400, and the external storage device 500. The image processing device 300 can also perform image processing on these images. The image processing device 300 may be configured by a general-purpose computer, or may be configured by a computer dedicated to the image processing system 1.

[0018] The fundus image capturing device 400 is a device for capturing a fundus image of the subject's eye, and as the device, for example, a fundus camera, a scanning laser ophothalmoscope (SLO), etc. may be used. The OCT device 200 and the fundus image capturing device 400 may be configured as an integrated device or as separate devices.

[0019] The external storage device 500 stores information about the subject's eye (such as the patient's name, age, and sex), various captured image data, shooting parameters, image analysis parameters, and parameters set by the user, each associated with each other. The external storage device 500 may be configured by any storage device, and may be configured by a storage medium such as an optical disk or a memory.

[0020] The display unit 600 is configured by any display, and can display information about the subject's eye and various images under the control of the image processing device 300.

[0021] The input unit 700 is composed of, for example, a mouse, a keyboard, a touch operation screen, etc. A user can input instructions to the image processing device 300, the OCT device 200, and the fundus image capturing device 400 to the image processing device 300 via the input unit 700. When the input unit 700 is a touch operation screen, the input unit 700 can be configured integrally with the display unit 600.

[0022] Although these components are shown as separate entities in FIG. 1, some or all of these components may be configured as an integrated unit.

[0023] Next, the OCT device 200 will be described. The OCT device 200 includes a galvanometer mirror 201, a drive control unit 202, a focus lens stage 203, an internal fixation light 204, a coherence gate stage 205, a light source 206, and a detector 207. Note that since the OCT device 200 is a known device, detailed description thereof will be omitted, and here, the capturing of a tomographic image performed in response to an instruction from an image processing device 300 will be described.

[0024] When an instruction to capture an image is transmitted from the image processing device 300, the light source 206 emits light. The light from the light source 206 is split into measurement light and reference light by a splitting unit (not shown). The OCT device 200 can generate an interference signal including tomographic information of the subject's eye by irradiating the measurement light onto the subject's eye and detecting interference light between the return light from the subject and the reference light.

[0025] The galvanometer mirror 201 is used to scan the fundus of the subject's eye with the measurement light, and the imaging range of the fundus during tomographic imaging can be determined by the scanning range of the measurement light by the galvanometer mirror 201. The image processing device 300 can determine the imaging range in the planar direction of the fundus and the number of scanning lines (scanning speed in the planar direction) by controlling the driving range and speed of the galvanometer mirror 201. In FIG. 1, the galvanometer mirror 201 is shown as one unit for the sake of simplicity of explanation, but the galvanometer mirror 201 is actually composed of a mirror for X-scanning and two mirrors for Y-scanning, and can scan a desired range on the fundus with the measurement light. Note that the configuration of the scanning unit for scanning the measurement light is not limited to the galvanometer mirror, and any other deflection mirror can be used. In addition, as the scanning unit, for example, a deflection mirror capable of scanning the measurement light in a two-dimensional direction with a single MEMS mirror or the like may be used.

[0026] The focus lens stage 203 is provided with a focus lens (not shown). By moving the focus lens stage 203, the focus lens can be moved along the optical axis of the measurement light. Therefore, the measurement light can be focused on the retinal layer of the fundus via the anterior segment of the subject's eye by the focus lens. The measurement light that has irradiated the fundus is reflected and scattered at each retinal layer, and travels back along the optical path as return light.

[0027] The coherence gate stage 205 is used to adjust the length of the optical path of the reference light or the measurement light to accommodate differences in axial length of the subject's eye. In this embodiment, the coherence gate stage 205 is configured by a stage provided with a mirror, and can correspond to the optical path length of the reference light to the optical path length of the measurement light by moving in the optical axis direction in the optical path of the reference light. Here, the coherence gate represents a position where the optical distances of the measurement light and the reference light in OCT are equal. The coherence gate stage 205 can be controlled by the image processing device 300. The image processing device 300 can control the imaging range in the depth direction of the subject's eye by controlling the position of the coherence gate by the coherence gate stage 205, and can control imaging of the retinal layer side or imaging of a deeper side than the retinal layer.

[0028] The detector 207 detects interference light between the return light of the measurement light from the subject's eye and the reference light, which occurs in an interference unit (not shown), and generates an interference signal. The image processing device 300 acquires the interference signal from the detector 207 and performs Fourier transform or the like on the interference signal to generate a tomographic image of the subject's eye.

[0029] The internal fixation lamp 204 is provided with a display unit 241 and a lens 242. In this embodiment, a display unit 241 in which a plurality of light-emitting diodes (LD) are arranged in a matrix is ​​used as an example of the display unit 241. The lighting positions of the light-emitting diodes are changed according to the part to be photographed under the control of the image processing device 300. The light from the display unit 241 is guided to the subject's eye via the lens 242. The light emitted from the display unit 241 has a wavelength of, for example, 520 nm, and is displayed in a desired pattern under the control of the image processing device 300.

[0030] The drive control unit 202 is connected to the above-mentioned galvanometer mirror 201, focus lens stage 203, internal fixation light 204, coherence gate stage 205, light source 206, and detector 207. The drive control unit 202 controls the drive of each of these components based on the control by the image processing device 300 via an instruction unit 304 described later. Note that the drive control unit 202 may not be provided, and the image processing device 300 may directly control the drive of these components using the instruction unit 304.

[0031] Next, the structure of the eye and images acquired by the image processing system 1 will be described with reference to Figs. 2(a) to (c). Fig. 2(a) is a schematic diagram of an eyeball. Fig. 2(a) shows the cornea C, the crystalline lens CL, the vitreous body V, the macular region M (the center of the macula represents the fovea), and the optic disc D. In this embodiment, the case of photographing the posterior pole of the retina including the vitreous body V, the macular region M, and the optic disc D will be mainly described. Although not described below, the OCT device 200 can also photograph the anterior eye segments such as the cornea and the crystalline lens.

[0032] FIG. 2(b) shows an example of a tomographic image obtained by photographing a retina using the OCT device 200. In FIG. 2(b), AS indicates an image unit obtained by one A-scan. Here, the A-scan refers to obtaining tomographic information in the depth direction at one point of the subject's eye by a series of operations of the OCT device 200. In addition, performing A-scans multiple times in any transverse direction (main scanning direction) to obtain two-dimensional tomographic information in the transverse direction and depth direction of the subject's eye is called a B-scan. A single B-scan image can be constructed by collecting multiple A-scan images obtained by A-scans. Hereinafter, this B-scan image will be referred to as a tomographic image.

[0033] Figure 2(b) shows the vitreous body V, macula M, optic disc D, and lamina cribrosa La. Boundary L1 represents the boundary between the internal limiting membrane (ILM) and nerve fiber layer (NFL), boundary L2 represents the boundary between the nerve fiber layer and ganglion cell layer (GCL), and boundary L3 represents the photoreceptor inner-outer segment junction (ISOS). Boundary L4 represents the retinal pigment epithelium layer (RPE), boundary L5 represents Bruch's membrane (BM), and boundary L6 represents the choroid. In the tomographic image, the horizontal axis (main scanning direction of OCT) is the x-axis, and the vertical axis (depth direction) is the z-axis.

[0034] Fig. 2(c) shows an example of a fundus image acquired by photographing the fundus of a test eye using the fundus image photographing device 400. Fig. 2(c) shows the macular region M and optic disc D, and retinal blood vessels are shown by thick curves. In the fundus image, the horizontal axis (main scanning direction of the OCT device) is the x-axis, and the vertical axis (sub-scanning direction of the OCT device) is the y-axis.

[0035] Next, a description will be given of the image processing device 300. The image processing device 300 includes an acquisition unit 301, a storage unit 302, a processing unit 303, an instruction unit 304, and a display control unit 305.

[0036] The acquiring unit 301 can acquire data of the interference signal of the subject's eye from the OCT device 200. The data of the interference signal acquired by the acquiring unit 301 may be an analog signal or a digital signal. When the acquiring unit 301 acquires an analog signal, the image processing device 300 can convert the analog signal into a digital signal. The acquiring unit 301 also has a tomographic image generating unit 311, and can acquire various images such as tomographic data, tomographic images, and En-Face images generated by the tomographic image generating unit 311. Here, the tomographic data is data including information about the tomography of the subject, and includes data based on the interference signal by OCT, and data obtained by performing a fast Fourier transform (FFT) or any signal processing on the data.

[0037] The tomographic image generating unit 311 can generate tomographic data by performing processing such as Fourier transform on the interference signal acquired by the acquiring unit 301, and generate a tomographic image based on the tomographic data. Note that any known method may be adopted as a method for generating a tomographic image, and detailed description thereof will be omitted.

[0038] Furthermore, the acquisition unit 301 acquires a group of imaging conditions of the tomographic image to be image-processed (for example, imaging date and time, imaging part name, imaging area, imaging angle of view, imaging method, image resolution and gradation, image pixel size, image filter, information on the image data format, etc.). The group of imaging conditions is not limited to the exemplified ones. The group of imaging conditions does not need to include all of the exemplified ones, and may include some of them.

[0039] The acquiring unit 301 can acquire data including fundus information acquired by the fundus image capturing device 400. The acquiring unit 301 can acquire information for identifying the subject's eye, such as a subject identification number, from the input unit 700. The acquiring unit 301 can store the acquired various data and images in the storage unit 302.

[0040] Next, a description will be given of the processing unit 303. The processing unit 303 includes an image processing unit 331, an arithmetic processing unit 322, an imaging condition acquisition unit 333, and a selection unit 334.

[0041] The image processing unit 331 generates a tomographic image, an En-Face image, and the like from the data acquired by the acquisition unit 301 and the data stored in the storage unit 302, and calculates a plurality of types of analytical values ​​related to the subject's eye (described later) from the generated or acquired images. Note that the plurality of types of analytical values ​​may be acquired from values ​​calculated in advance and stored in the storage unit 302 or the external storage device 500.

[0042] The arithmetic processing unit 332 includes a trained model created by providing training data to an arbitrary machine learning model such as deep learning and having it train. Specific learning content will be described later. Using this trained model, the arithmetic processing unit 332 generates a tomographic image captured at a third time after the second time from, for example, tomographic data obtained by photographing the subject's eye at a first time and tomographic data obtained by photographing the subject's eye at a second time after the first time. The generated tomographic image is subjected to an analysis process in the image processing unit 331, and multiple types of analytical values ​​in the tomographic image are calculated. The calculation of the multiple types of analytical values ​​may be performed by the arithmetic processing unit 332, or may be directly generated from two pieces of tomographic data using the trained model instead of calculation.

[0043] The imaging condition acquisition unit 333 acquires imaging conditions when a tomographic image to be subjected to image processing was captured from the storage unit 302. Note that the imaging conditions to be acquired may be the above-mentioned imaging conditions acquired by the acquisition unit 301 at the time of capturing a tomographic image, or may be imaging conditions stored in the storage unit 302 or the like in association with a previously captured tomographic image.

[0044] The selection unit 334 selects an appropriate trained model according to the imaging conditions acquired by the imaging condition acquisition unit 333, or the imaging site such as the anterior eye segment, the posterior eye segment, etc. in addition to the above-mentioned conditions. A plurality of trained models are generated in advance according to the imaging conditions and stored in the storage unit 302, and the calculation processing unit 332 generates a tomographic image and its analysis value using the trained model selected by the selection unit 334 according to the imaging conditions.

[0045] The instruction unit 304 can control the driving of each component of the OCT device 200 and the fundus image capturing device 400. The storage unit 302 can store various images and data, such as the tomographic data acquired by the acquisition unit 301 and the tomographic images generated and processed by the processing unit 303. The storage unit 302 can also store programs, etc., which are executed by the processor to perform the functions of each component of the image processing device 300.

[0046] The display control unit 305 can control the display on the display unit 600 of various information acquired by the acquisition unit 301, tomographic images generated and processed by the processing unit 303, information input by the user, and the like.

[0047] Each component of the image processing device 300 other than the storage unit 302 may be configured by a software module executed by a processor such as a CPU (Central Processing Unit) or an MPU (Micro Processing Unit). Each component may be configured by a circuit that performs a specific function such as an ASIC. The storage unit 302 may be configured by any storage medium such as an optical disk or a memory.

[0048] Next, a series of processes according to this embodiment will be described with reference to Fig. 3. Fig. 3 is a flowchart of a series of processes according to this embodiment. When a series of processes according to this embodiment is started, the process proceeds to step S301.

[0049] In step S301, the acquiring unit 301 acquires a subject identification number, which is an example of information for identifying a subject's eye, from outside the image processing device 300 via the input unit 700 or the like. The acquiring unit 301 acquires information on the subject's eye held in the external storage device 500 based on the subject identification number, and stores the information in the storage unit 302.

[0050] In step S302, the drive control unit 202 controls the OCT device 200 to scan the subject's eye with the measurement light to capture an image, and the acquisition unit 301 acquires an interference signal including tomographic information of the subject's eye from the OCT device 200. The scanning with the measurement light is performed by the instruction unit 304 controlling the OCT device 200 to operate the light source 206, the galvanometer mirror 201, etc. in response to a user's instruction to start scanning.

[0051] The galvanometer mirror 201 includes an X scanner for the horizontal direction and a Y scanner for the vertical direction. The drive control unit 202 can scan the measurement light in each of the horizontal direction (X) and vertical direction (Y) in the device coordinate system by changing the orientation of each of these scanners. The drive control unit 202 can also scan the measurement light in a direction that combines the horizontal direction and the vertical direction by changing the orientation of these scanners simultaneously. Therefore, the drive control unit 202 can scan the measurement light in any direction on the fundus plane.

[0052] The drive control unit 202 adjusts various shooting parameters when shooting. Specifically, the drive control unit 202 sets at least the position of the pattern displayed by the internal fixation light 204, the scan range and scan pattern by the galvanometer mirror 201, the coherence gate position, and the focus lens position.

[0053] The drive control unit 202 controls the light emitting diodes of the display unit 241 to control the position of the pattern displayed by the internal fixation lamp 204 so that, for example, the center of the macula or the optic disc of the subject's eye is photographed. The drive control unit 202 also sets a scanning pattern such as a raster scan, radial scan, or cross scan for photographing a three-dimensional volume as a scanning pattern by the galvanometer mirror 201. Note that, regardless of which scanning pattern is selected, scanning is repeated multiple times on one line, and multiple photographs (the number of repetitions is two or more) are taken. In this embodiment, the scanning pattern is a cross scan, and 150 photographs of the same location are taken repeatedly. After these photographing parameters are adjusted, the instruction unit 304 controls the OCT device 200 to photograph the subject's eye in response to an instruction from the user to start photographing.

[0054] Although not described in this disclosure, the OCT device 200 can perform tracking of the subject's eye in order to capture the same location for averaging. This allows the OCT device 200 to scan the subject's eye while reducing the influence of fixational eye movement.

[0055] In step S303, the tomographic image generating unit 311 generates a tomographic image based on the interference signals acquired by the acquiring unit 301. The tomographic image generating unit 311 can generate a tomographic image by performing a general reconstruction process on each interference signal.

[0056] First, the tomographic image generating unit 311 removes fixed pattern noise from the interference signal. Fixed pattern noise removal is performed by extracting fixed pattern noise by averaging multiple acquired A-scan signals and subtracting this from the input interference signal. After that, the tomographic image generating unit 311 performs a desired window function processing to optimize the depth resolution and dynamic range, which are in a trade-off relationship when Fourier transform is performed in a finite section. The tomographic image generating unit 311 generates tomographic data by performing fast Fourier transform (FFT) processing on the interference signal that has been subjected to the window function processing.

[0057] The tomographic image generating unit 311 obtains pixel values ​​of the tomographic image based on the generated tomographic data, and generates the tomographic image. Note that the method of generating the tomographic image is not limited to this, and any known method may be used.

[0058] In this embodiment, the tomographic image generating unit 311 generates a tomographic image, which is a so-called intensity image, based on the intensity of the interference light. However, the image generated by the tomographic image generating unit 311 is not limited to an intensity image, and may be, for example, a motion contrast image. The motion contrast image or the motion contrast data used in generating the motion contrast image is also included in one aspect of the above-mentioned tomographic data. In addition, information on blood vessels, such as blood vessel diameter and blood vessel density, can be obtained from the motion contrast data, and this information is included in the data on the tomography, which will be described later in this embodiment.

[0059] In the subject's eye, blood cells flowing through blood vessels are objects that cause fluctuations in the intensity of interference light for a very short period of time. By obtaining motion contrast data based on the movement of such blood cells, blood vessels can be imaged without using contrast agents. By integrating three-dimensional motion contrast data obtained based on the depth information of the subject's eye in the depth direction and projecting it onto a two-dimensional plane, a frontal blood vessel image called an OCTA image can be generated.

[0060] Here, a specific example will be described in which motion contrast data is generated by the tomographic image generating unit 311. A plurality of tomographic images are obtained by photographing substantially the same location at successive photographing times, and are aligned. Various known methods can be used for the alignment. For example, one of the plurality of tomographic images is selected as a reference image, and while changing the position and angle of the reference image, the similarity with the other tomographic images is calculated, and the amount of positional deviation of each tomographic image from the reference image is calculated. Each tomographic image is corrected based on the calculation result, thereby aligning the plurality of tomographic images. The method of alignment is not limited to this, and may be performed by any known method.

[0061] The tomographic image generating unit 311 calculates a decorrelation value for every two tomographic images captured at consecutive times among the plurality of aligned tomographic images, using the following Equation 1.

number

[0062] Here, A(x,z) indicates the amplitude at position (x,z) of tomographic image A, and B(x,z) indicates the amplitude at the same position (x,z) of tomographic image B. The resulting decorrelation value M(x,z) ranges from 0 to 1, and the greater the difference between the two amplitude values, the closer the value is to 1. Note that in this embodiment, a two-dimensional tomographic image in the XZ plane is used, but a two-dimensional tomographic image in the YZ plane, for example, may also be used. In this case, the position (x,z) may be replaced with the position (y,z), etc. Note that the decorrelation value may be calculated based on the luminance value of the tomographic image, or may be calculated based on the value of the tomographic signal corresponding to the tomographic image.

[0063] The tomographic image generating unit 311 determines pixel values ​​of a motion contrast image based on the decorrelation value M(x,z) at each position (pixel position) to generate a motion contrast image. In this embodiment, the tomographic image generating unit 311 calculates decorrelation values ​​for tomographic images whose imaging times are consecutive, but the calculation method of motion contrast data is not limited to this. The two tomographic images for which the decorrelation value M is calculated may have corresponding imaging times within a predetermined time interval, and may not be consecutive imaging times. Therefore, for example, for the purpose of extracting an object with little change over time, two tomographic images whose imaging interval is longer than a normal specified time may be extracted from the acquired multiple tomographic images to calculate the decorrelation value. In addition, instead of the decorrelation value, a variance value, a value obtained by dividing a maximum value by a minimum value (maximum value / minimum value), or the like may be calculated.

[0064] The method of generating a motion contrast image is not limited to the above-mentioned method, and any other known method may be used. As described above, the motion contrast data is obtained by calculating the difference, ratio, correlation, or the like of the phase, vector, and temporal change in intensity of a complex OCT signal between a plurality of tomographic images obtained by repeatedly capturing images of, for example, approximately the same location. In this case, the approximately same location refers to a position that is the same to an extent that is permissible for generating motion contrast data, and also refers to a location that is slightly shifted from the strictly same location.

[0065] In step S304, the image processor 331 detects the retinal layer and obtains multiple analytical values ​​such as the thickness, volume, curvature, and shape of the retinal layer. First, the detection of the retinal layer will be described. The image processor 331 detects the boundaries of the retinal layer in multiple tomographic images captured by the OCT device 200.

[0066] Here, the details of the process for detecting the retinal layers will be described. The image processing unit 331 performs noise removal and edge enhancement processes on the tomographic image to be processed. For example, a median filter or a Gaussian filter is applied as the noise removal process. For example, a Sobel filter or a Hessian filter is applied as the edge enhancement process.

[0067] The edge enhancement process for a two-dimensional tomographic image using a two-dimensional Hessian filter will be described. The Hessian filter can enhance the secondary local structure of a two-dimensional gray distribution based on the relationship between two eigenvalues ​​(λ1, λ2) of the Hessian matrix. The two-dimensional line structure is enhanced using the relationship between the eigenvalues ​​and eigenvectors (e1, e2) of the Hessian matrix. Since the line structure in the two-dimensional tomographic image of the subject's eye corresponds to the retinal layer, the structure of the retinal layer can be enhanced by applying the Hessian filter.

[0068] In order to detect retinal layers with different thicknesses, the resolution of smoothing using a Gaussian function performed when calculating the Hessian matrix can be changed. In addition, when applying a two-dimensional Hessian filter, the data can be transformed to match the physical sizes of the XZ of the image before application. In the case of general OCT, the physical sizes in the XY and Z directions are different. Therefore, the filter is applied by matching the physical size of the retinal layer for each pixel. Note that the physical sizes in the XY and Z directions can be grasped from the design / configuration of the OCT device 200, so the data of the tomographic image can be transformed based on the physical sizes. In addition, when the physical size is not normalized, it is possible to approximately deal with this by changing the resolution of smoothing using a Gaussian function.

[0069] Although the processing in a two-dimensional tomographic image has been described above, the subject to which the Hessian filter is applied is not limited to this. When the data structure when the tomographic image is captured is a three-dimensional tomographic image by raster scanning, it is also possible to apply a three-dimensional Hessian filter. In this case, the image processing unit 331 performs a positioning process in the XZ direction between adjacent tomographic images, and then the secondary local structure of the three-dimensional gray distribution can be emphasized based on the relationship between the three eigenvalues ​​(λ1, λ2, λ3) of the Hessian matrix. Therefore, it is also possible to emphasize edges three-dimensionally by emphasizing the three-dimensional layer structure using the relationship between the eigenvalues ​​and eigenvectors (e1, e2, e3) of the Hessian matrix. Note that, although the configuration for performing edge enhancement processing using a Hessian filter has been described here, the processing method for edge enhancement processing is not limited to this, and any existing method may be used.

[0070] The image processor 331 detects edge-enhanced boundaries from the tomographic image that has been subjected to edge enhancement processing. In this embodiment, the boundaries between the ILM and NFL, and the boundaries between the RPE, ISOS, NFL and GCL are detected. Although not shown, other boundaries such as the boundary between the outer plexiform layer (OPL) and the outer nuclear layer (ONL), the boundary between the inner plexiform layer (IPL) and the inner nuclear layer (INL), the boundary between the INL and OPL, and the boundary between the GCL and IPL can also be detected.

[0071] As a method of detecting the boundary, a plurality of points with strong edge strength are detected as boundary candidates in each A-scan, and points (points with strong edge strength) are connected as lines based on the continuity between the boundary candidates in adjacent A-scans. When points are connected as lines, the image processing unit 331 can remove outliers whose positions are shifted in the depth direction by evaluating the smoothness of the lines. More specifically, for example, the positions of the connected points in the Z direction are compared, and if the difference in the positions in the Z direction is greater than a predetermined threshold, the newly connected point can be determined as an outlier and excluded from the connecting process. When the outlier is removed, boundary candidates in the A-scan adjacent to the A-scan position of the excluded point may be connected as lines. Note that the method of removing the outliers is not limited to this, and any existing method may be used.

[0072] The image processor 331 determines the corresponding boundary for each line formed by connecting the points based on the vertical distance and positional relationship in the Z direction of the boundary of the retina layer. If no boundary is detected as a result of removing outliers in each A-scan, the boundary may be obtained by interpolation from the surrounding boundaries. Alternatively, the boundary candidate may be searched for in the horizontal direction (X or Y direction) based on the edge from the surrounding boundaries, and the boundary may be determined again based on the boundary candidate searched from the surrounding boundaries.

[0073] After that, the image processing unit 331 executes a process for correcting the detected boundary to make the boundary shape smooth. For example, the boundary shape may be smoothed by using image features and shape features by an active contour model such as Snakes or Level Set method. In addition, the coordinate values ​​of the boundary shape may be regarded as time-series data based on a signal, and the shape may be smoothed by a smoothing process such as a Savitzky-Golay filter, a simple moving average, a weighted moving average, or an exponential moving average.

[0074] In addition, a dedicated trained model may also be used for the detection of retinal layers. In this case, a trained model for layer detection is generated by training a pair of a label image created in advance as training data and a tomographic image used to generate the label image as an input image. The label image refers to an image in which a label is given to each pixel in the tomographic image, the image appearing (photographed) at that pixel. For example, the label given may be a label on the superficial layer side (vitreous side) of the retina, a label on the inner retina layer, and a label on the deeper layer side (choroid side) of the retina.

[0075] Next, the image processing unit 331 performs measurement processing to obtain a plurality of analysis values. An example of the analysis value is the thickness of the retina defined by any two boundaries obtained by boundary detection. For example, in glaucoma, the retina tends to thin and the layer thickness becomes asymmetric between the top and bottom. In addition, when the subject's eye has an abnormality accompanied by bleeding such as retinal vein molecular occlusion, the entire retina becomes thick due to the bleeding. Therefore, by measuring the thickness of the retina and obtaining the analysis value, the characteristics of the layer thickness change caused by the disease can be known.

[0076] In addition, the standard thickness of the retina layer varies depending on whether the measurement site is the macula or the optic disc. For this reason, it is advisable to measure the thickness together with the position information of the measurement site. In addition, when the imaging site is the optic disc, it is advisable to measure the C / D ratio, which is the ratio of Cup to Disc. In addition, it is possible to measure the shape characteristics of any boundary obtained by boundary detection. For example, in severe myopia, the retina is deformed into a shape that is stretched backward (downward). In addition, in age-related macular degeneration, the RPE is deformed into a shape that protrudes upward. Therefore, it is advisable to measure the local curvature as an analytical value using the boundaries of the RPE, Bruch's membrane, etc. as a shape characteristic. From such analytical values, it is possible to grasp the shape characteristics that appear due to the disease.

[0077] As mentioned above, the layer thickness and the shape of the retina are characterized by the disease. Furthermore, in one eye, not only one of these characteristics may appear, but multiple characteristics may appear. For example, as the thinning of a specific layer progresses due to a certain disease, the shape of the retina, such as the curvature, may also change, and as the disease progresses, the thinning may become rapid at a certain point due to the influence of the curvature. In such a case, if only the progression of thinning is assumed from the analysis value, a sudden change cannot be assumed. Therefore, it is a good idea to obtain multiple types of analysis values, such as layer thickness and curvature, that are expected to be measured.

[0078] Also, as described above, the image processor 331 may generate motion contrast data. In this case, the image processor 331 may obtain analytical values ​​using an OCTA image, which is a front image of a motion contrast image obtained by performing average projection (AIP) or maximum intensity projection (MIP) on the motion contrast data. For example, the OCTA image may be binarized, or the binarized image may be thinned to obtain characteristics such as the density of a portion corresponding to a blood vessel or the shape (length and shape) of the blood vessel. These characteristics may also be used as analytical values.

[0079] In step S304, when the image processing unit 331 performs a measurement process of the analytical value, the process proceeds to step S305. In step S305, the detected boundary and the tomographic image are displayed on the display unit 600. Here, the process of displaying a screen on the display unit 600 will be described with reference to FIG. 3(b) and FIGS. 4 to 6. Note that FIG. 3(b) is a flowchart of a series of processes executed in the display process of step S305. When the display process according to this embodiment is started, the process proceeds to step S351.

[0080] In step S351, the photographing condition acquisition unit 333 acquires the photographing conditions under which the interference signal was acquired in step S302 from the storage unit 302. The photographing condition acquisition unit 333 also acquires from the storage unit 302 the subject's eye information acquired in step S301.

[0081] In step S352, the selection unit 334 judges whether the storage unit 302 or the external storage device 500 stores information obtained by previously photographing the same subject's eye with the same scanning pattern, based on the information acquired in step S351. If there is data previously photographed with the same scanning pattern, the process proceeds to step S353. If there is no such data, the process proceeds to step S356.

[0082] In addition, the flow may be divided according to the number of past examinations, not just the presence or absence of captured data. For example, if the number of captured data in the past is only one, there is only two data including the current data, and there is not enough information to use the trained model described later. Therefore, a threshold value Th (for example, Th>2) may be set as the number of past examinations, and the process may proceed to step S353 if the number of past examinations is equal to or greater than a certain number.

[0083] In step S353, the selection unit 334 selects the learned model to be used based on the shooting conditions acquired by the shooting condition acquisition unit 333. For example, the shooting conditions related to the shooting part include whether the shooting part is the anterior segment or the posterior segment of the subject's eye. Since the analysis values ​​output from the image data are different between the anterior segment and the posterior segment, separate learned models are obtained for each. In this embodiment, an example has been described in which a learned model to be used is selected depending on whether the anterior segment or the posterior segment is to be shot. However, the shooting part to be referenced is not limited to this, and a learned model may also be selected for the posterior segment depending on whether it is the macula or the optic disc.

[0084] For example, in the case where a trained model for the posterior segment is present but a trained model for the anterior segment is not present, if the input data is an image of the anterior segment, the flow may proceed to step S356 at the time of step S352. With this processing, even if the trained model is not compatible initially, a trained model that can be compatible can be added by updating the software, and it becomes possible to handle cases where the anterior segment is photographed.

[0085] In step S354, the arithmetic processing unit 332 uses the trained model to generate a tomographic image that is likely to be obtained when the subject's eye is photographed under the same photographing conditions in the future. The image processing unit 331 also uses the generated tomographic image to calculate the above-mentioned multiple types of analytical values. In this embodiment, the arithmetic processing unit 332 generates a tomographic image using the trained model. However, the arithmetic processing unit 332 may directly generate each analytical value that is likely to be obtained in the future from the multiple types of analytical values ​​obtained by the above-mentioned image processing unit 331.

[0086] As described above, the trained model in the description of the present invention is a model trained in advance using appropriate training data for a machine learning model according to an arbitrary machine learning algorithm such as deep learning. Here, the training data used for training when obtaining the trained model according to the present embodiment will be described. The training data is a pair of input data corresponding to data actually input to the trained model and teacher data corresponding to data generated by the trained model. In the present invention, as input data, two tomographic images obtained by photographing the same subject's eye under the same shooting conditions at at least two times, a first time and a second time after the first time, or two inspection results consisting of multiple types of analysis values ​​obtained from the tomographic images are used. In addition, as teacher data, a tomographic image obtained by photographing the same subject's eye under the same shooting conditions at a third time further after the second time, or an inspection result consisting of multiple types of analysis values ​​obtained from the tomographic images is used.

[0087] When obtaining a trained model, as described herein, training is performed using tomographic images and test results obtained in advance at least three points in time in the past as training data. When using this trained model, two test results obtained at the same time interval as the time interval between the first time and the second time in the past corresponding to the input data used when generating the trained model are input. The teacher data corresponds to the analysis value, etc. generated from the trained model. In other words, by using the trained model, it is possible to obtain an analysis value that is likely to be obtained from the subject's eye several months to several years in the future, which corresponds to the time interval between the second time and the third time described above.

[0088] Next, a machine learning algorithm using multiple types of analytical values ​​will be described with reference to FIG. 4. Here, learning data used for training when obtaining a trained model will be described. As described above, the learning data includes, for example, numerical data consisting of multiple types of analytical values ​​obtained from the same subject's eye photographed at different times in the past. An example of the learning data is shown in FIG. 4. FIG. 4 shows an example of analytical values ​​(numerical values) on a time series (date and time T) of N measurement parameters (Params1 to ParamsN). Here, the multiple types of analytical values ​​include, for example, the thickness of the entire retina, the thickness between any layers, the thickness of the choroid, and the shape characteristics of the retina (for example, the curvature of the RPE, Bruch's membrane, etc.) when the photographed part is a posterior eye image. Also, when the photographed part is the optic disc, in addition to the thickness of the retina, the multiple types of analytical values ​​include the C / D ratio, which is the ratio of Cup to Disc. The C / D ratio may be obtained not from a tomographic image but from a planar image such as a fundus photograph or SLO obtained from the fundus image photographing device 400. Furthermore, when the imaging site is the anterior segment, the corneal thickness, corneal curvature, anterior chamber depth, total lens curvature, posterior lens curvature, and the like are included in the multiple types of analysis values.

[0089] FIG. 4 is a time series graph consisting of a horizontal axis consisting of a time axis T indicating the date and time when the image was taken, and a vertical axis consisting of an analysis value of a specific measurement parameter. The numerical value of the analysis value may be used as it is as the learning data. However, in order to make it easier to handle, the difference from the analysis value at the previous time may be calculated and used as the learning data. Note that the above describes an example in which two pieces of data on a time series are used as input data used when generating the above-mentioned trained model, and two pieces of data are also used correspondingly to the data input to the trained model. However, the number of input data and input data is not limited to two, and may be a larger number of data. Using a large amount of data on a time series when generating a trained model increases the time required for learning. However, it becomes possible to match the interval of the input data with the interval of any data on this time series, which creates the possibility of improving versatility.

[0090] Furthermore, during learning, information on the degree of progression of the disease, such as whether the disease has progressed, has not progressed, or has stopped, may be added to the learning data as an evaluation value by adding up the data consisting of these multiple analysis values. In this case, the degree of progression of the disease is defined as a numerical value. For example, a doctor or the like judges the condition of the disease by referring to the time series data of the analysis values ​​and tomographic images, and if the disease has progressed, the evaluation value is set to 100, if there is no change, the evaluation value is set to 50, and if the disease has improved, the evaluation value is set to 0, etc. Note that this numerical value is merely an example, and is not limited to this value, and the evaluation value is appropriately determined depending on the type of disease and the item to be judged. Note that, although the evaluation value is treated as a numerical value as an example here, for example, the progress of the disease or the condition of the eyes may be ranked as good or bad. In other words, the evaluation value described here is an example of evaluation information that a doctor or the like evaluates the condition of the eyes and determines it, and inputs it as data.

[0091] Next, as an example of a machine learning model according to the present embodiment, a Recurrent Neural Network (hereinafter, RNN), which is a neural network that handles time-series information, will be described with reference to Fig. 5. In addition, a Long short-term memory (hereinafter, LSTM), which is a type of RNN and is used by the arithmetic processing unit 332, will be described with reference to Fig. 6.

[0092] 5(a) shows the structure of an RNN, which is a machine learning model in the arithmetic processing unit 332. The RNN 520 shown in FIG. 5 has a loop structure in the network, and at time t, data x t Enter 510 and enter data h t The RNN 520 outputs 530. Since the RNN 520 has a loop function in the network, it is possible to pass the current state to the next state, so it can handle time-series information. Figure 5(b) shows an example of input and output of a parameter vector at time t. Data x t 510 includes analytical values ​​for N measurement parameters (Params1 to ParamsN). In addition, the data output from the RNN 520 is t 530 includes N analytical values ​​(Params1 to ParamsN) corresponding to the input data.

[0093] However, RNN cannot handle long-term information during backpropagation. For this reason, LSTM is used in this embodiment. LSTM can learn long-term information by equipping it with a forget gate, an input gate, and an output gate. Here, the structure of LSTM is shown in FIG. 6(a). In LSTM 620, the information that the network takes over at the next time t is the internal state c of the network called a cell. t-1 and output data h t-1 In the figure, the lowercase letters (c, h, x) represent vectors. In this embodiment, the vectors are composed of the analytical values ​​shown in FIG.

[0094] Next, the details of LSTM620 are shown in Figure 6(b). In Figure 6(b), FG is the forget gate network, IG is the input gate network, and OG is the output gate network, each of which is a sigmoid layer. Therefore, it outputs a vector in which each element has a value between 0 and 1. The forget gate network FG decides how much past information to retain, and the input gate network IG decides which value to update. CU is the cell update candidate network, which is an activation function tanh layer. It creates a vector of new candidate values ​​to be added to the cell. The output gate network OG selects the cell candidate elements and selects how much information to convey at the next time.

[0095] Note that the above-mentioned LSTM model is a basic form, and is not limited to the network shown here. The connections between networks may be changed. A QRNN (Quasi Recurrent Neural Network) may be used instead of an LSTM. Furthermore, the machine learning model is not limited to a neural network, and machine learning models such as boosting and support vector machines may be used. In addition, the RNN and LSTM exemplified here are described as cases in which numerical values ​​exemplified as analytical values ​​are input as data. However, as described above, the RNN and LSTM may be configured to input tomographic data such as a tomographic image and output tomographic data.

[0096] When data is input to the trained model, data according to the design of the trained model is output. For example, output data that is likely to correspond to the input data according to the tendency trained from the training data is output. For example, when a tomographic image captured by OCT is input to the trained model trained by the above-mentioned training data, a tomographic image predicted to be likely to be captured in the future is output. In addition, when a trained model using an analytical value instead of a tomographic image is used, when an analytical value obtained from a tomographic image captured by OCT is input to the trained model, an analytical value predicted to be likely to be measured from a tomographic image captured in the future is output.

[0097] In the trained model of the calculation processing unit 332 according to this embodiment, data x t When a tomographic image is input as 510, data h is processed according to the tendency learned using the learning data. t The tomographic image is output as data x t When the analysis values ​​of the multiple measurement parameters are input as 510, the data h is calculated according to the trend learned using the learning data. t A plurality of analysis values ​​of the corresponding measurement parameters are output as 530 .

[0098] In step S355 or S356, display unit 600 displays the analysis results in response to an instruction from display control unit 305. Note that the only difference between step S355 and step S356 is whether the results generated by calculation processing unit 332 are displayed or the analytical values ​​etc. acquired in step S351 are displayed as is, and the other display information is the same. Therefore, the screens displayed on display unit 600 will be explained collectively below.

[0099] FIG. 7 shows an example of a screen displayed on the display unit 600. A patient tab 701, an imaging tab 702, a report tab 703, and a setting tab 704 are displayed on the display screen 710. Also, on the left side of the display screen, a group of tomographic images previously captured of the same subject's eye are displayed as thumbnail images 705. In the figure, the report tab 703 is shaded, and this shaded line indicates that the report screen is in an active state. In the following, a case where the report screen is displayed in this embodiment will be described. Note that, although shaded lines are used here to indicate an active state, the color of the display area can also be changed to indicate an active state by, for example, changing the color of the displayed characters.

[0100] The report screen displays a tomographic image 711 obtained by scanning the subject's eye in the horizontal direction, and a tomographic image 712 obtained by scanning the subject's eye in the vertical direction. The report screen also displays an SLO image 713, which is a front image of the fundus. A color map 714 representing, for example, the thickness of the retina is superimposed on this SLO image 713. Also displayed are an ETDRS grid 715 displaying the retinal thickness of each region of the sector superimposed on the SLO image 713, and a table 716 showing the retinal thickness within the ETDRS grid.

[0101] Graph 720 is obtained by arranging the retinal thickness obtained from the tomographic images 711 and 712 in a time series. When there is a predicted analysis value for the retinal thickness generated by the calculation processing unit 332, it is displayed here. Selector 721 is used to select the measurement parameter that has obtained the analysis value shown in graph 720. Here, Params1, which has the retinal thickness as a measurement parameter, is selected.

[0102] In the graph 720, a number of black circles 722 indicate time-series analysis values ​​of retinal thickness obtained from previously captured tomographic images, and white circles 723 indicate the latest analysis values. Furthermore, stars 724 indicate future predicted values ​​of retinal thickness generated by the calculation processing unit 332. Probability 725 corresponds to the evaluation value input by the doctor or the like as described above. When an evaluation value is input together with the past analysis value, the calculation processing unit 332 generates this evaluation value, and this is displayed as probability 725 on the report screen.

[0103] In this embodiment, the analytical values ​​of a plurality of measurement parameters are input, and the calculation processing unit 332 outputs the corresponding progress prediction value as the analytical value, and the result is displayed as the graph 720. Therefore, the time-series analytical value displayed on the graph 720 changes by selecting the measurement parameter displayed by the selector 721, and the future prediction value indicated by the star 724 changes accordingly. However, since the probability 725 is generated based on the evaluation value comprehensively input by the doctor or the like using the overall parameters, the result does not change even if the measurement parameter displayed by the selector 721 is changed.

[0104] Here, an example is shown in which the predicted value of the progress of the probability 725 is simply displayed as a numerical value. However, as another display example, the numerical value may be displayed in different colors to be easily understood visually, such as blue or green if the symptom improves, black if there is little change, and orange or red if the symptom is likely to worsen. Also, instead of displaying as the probability 752, each point displayed on the graph 720 may be displayed in the above-mentioned color to show the progress of the symptom to the examiner. Furthermore, here, a doctor or the like evaluates the subject's eye, and the evaluation result is quantified as a probability and input, but the evaluation result is not limited to a numerical value and may be, for example, ranked. That is, the object displayed here may be evaluation information obtained by a doctor or the like by evaluating the condition of the subject's eye.

[0105] In step S355, a report screen including the graph 720 including the results generated by the calculation processing unit 332 and the probability 725 is displayed on the display screen 710. After display, the flow proceeds to step S306. In step S356, since the calculation processing unit 332 cannot generate an analysis value or the like, the result obtained in step S351 is displayed as is on the display unit 600. In this case, the star 724 is not displayed on the graph 720, and the probability 725 is not displayed or is displayed as a blank field. After display, the flow proceeds to step S306. In the display in step S336, a doctor or the like can also input an evaluation value as the probability 725 via the input unit 700.

[0106] As described above, in this embodiment, the arithmetic processing unit 332 generates future tomographic images and analytical values ​​in the process of displaying the report screen. However, the timing at which the arithmetic processing unit 332 generates future tomographic images and analytical values ​​is not limited to this. When the measurement process in step S304 is completed, if there is a past examination, the sectional images or analytical values ​​may be generated and stored in the external storage device 500. In this case, in the display process in step S305, the stored tomographic images or analytical values ​​may be read out and displayed.

[0107] In step S306, the acquisition unit 301 acquires from the outside an instruction as to whether or not to end a series of processes related to the capturing of a tomographic image by the image processing system 1. This instruction can be input by the user using the input unit 700. When an instruction to end the processes is acquired, the image processing system 1 ends a series of processes related to this embodiment. On the other hand, when the acquisition unit 301 acquires an instruction not to end the processes, the flow returns to step S302 and continues the capture.

[0108] As described above, the image processing device 300 according to this embodiment includes an acquisition means (acquisition unit 301) and a generation means (arithmetic processing unit 332). The acquisition means acquires data on a first tomography of the subject's eye associated with a first time and data on a second tomography of the subject's eye associated with a second time after the first time. The data on the first tomography includes at least one of a tomography image obtained by photographing the subject's eye at the first time and a plurality of types of analytic values ​​(first analytic values) obtained from the tomography image. The data on the second tomography includes at least one of a tomography image obtained by photographing the subject's eye at the second time and a plurality of types of analytic values ​​(second analytic values) obtained from the tomography image.

[0109] The calculation processing unit 332 uses the learned model to generate at least one of a plurality of types of analytical values ​​of the subject's eye at a third time after the second time from the data on the first tomography and the data on the second tomography acquired by the acquisition unit 301. At this time, the data on the tomography includes a plurality of types of analytical values. Note that the data on the tomography includes not only a case where a plurality of types of analytical values ​​are treated as one data set, but also a case where each data of the plurality of types of analytical values ​​is treated separately. In addition, at least one analytical value can be calculated and generated from a tomography image of the subject's eye (data on the third tomography) generated using the learned model. That is, the data on the tomography may be a tomography image or tomography data. In this case, a tomography image associated with the above-mentioned first time and a tomography image associated with the second time are acquired, a tomography image is generated from these tomography images using the learned model, and a specific analytical value is obtained from the generated tomography image. In addition, when comparing tomographic images of the same test eye at different times, for example, due to the progression of a disease, the characteristic layer may thin and the curvature may also change in the layer or the surrounding layers. In other words, by using the tomographic images acquired at different times from the same test eye as data to be input to the trained model, a future tomographic image in which changes in multiple types of analytical values ​​for the test eye are taken into consideration can be obtained. As described above, the present invention aims to provide diagnostic assistance using the complex characteristics of the eye, and a specific analytical value is obtained using a trained model from two data related to a tomogram containing multiple types of analytical values ​​of the test eye and associated with different times. Therefore, obtaining a specific analytical value by analyzing a tomogram generated using a trained model from two tomograms obtained by photographing the test eye at different times can achieve the same effect as obtaining a specific analytical value from the above-mentioned data related to two tomograms associated with different times. Note that an example has been described here in which a tomographic image associated with a first time and a tomographic image associated with a second time are acquired, a tomographic image is generated from these tomographic images using a trained model, and a specific analytical value is obtained from the generated tomographic image.However, instead of using a learning model that generates a tomographic image from two tomographic images, a trained model that generates an analytic value directly from two tomographic images may be used. In this case, a tomographic image associated with a first time and a tomographic image associated with a second time are acquired, and a specific analytic value is obtained directly from these tomographic images using the trained model.

[0110] The trained model according to the present embodiment is obtained using training data consisting of input data and teacher data. For the input data, for example, a tomographic image obtained by photographing the subject's eye at a fourth time before the first time and a tomographic image obtained by photographing the subject's eye at a fifth time before the first time and after the fourth time are used. For the teacher data, a tomographic image obtained by photographing the subject's eye at a sixth time before the first time and after the fifth time is used.

[0111] In addition, in the arithmetic processing unit 332, the data input to the trained model may not be a tomographic image, but may be each of a plurality of types of analytical values ​​obtained from a tomographic image of the test eye. In this case, one of the input data in the trained model is a plurality of types of fourth analytical values ​​obtained from a tomographic image obtained by photographing the test eye at a fourth time before the first time. In addition, the other input data is a plurality of types of fifth analytical values ​​obtained from a tomographic image obtained by photographing the test eye at a fifth time before the first time and after the fourth time. And the teacher data is a plurality of types of sixth analytical values ​​obtained from a tomographic image obtained by photographing the test eye at a sixth time before the first time and after the fifth time. By learning using these input data and teacher data, the trained model described above is obtained.

[0112] In addition, in the data input to the trained model and the input data used to obtain the trained model, the elapsed time between the second time and the third time and the elapsed time between the fifth time and the sixth time are approximately equal. In this case, the elapsed times do not need to be exactly the same, but the user can appropriately determine the amount of difference that is acceptable depending on, for example, a pathology or abnormality of interest in the subject's eye.

[0113] The acquisition unit 301 of the image processing device 300 according to this embodiment can acquire the above-mentioned evaluation information regarding the state of the subject's eye at the first time and the second time. In this case, the calculation processing unit 332 can generate the evaluation information at the third time together with at least one of the multiple types of analysis values ​​at the third time. Furthermore, the display control means (display control unit 305) can display the generated evaluation information on the display unit 600.

[0114] Here, consider a case where the data on the first fault and the data on the second fault are fault images (tomographic data). In this case, a first analytic value associated with a first time and a second analytic value associated with a second time are obtained from the data on these faults. The display control unit 305 generates a graph 720 using the first and second analytic values ​​of the same type as the at least one analytic value at a third time, and the at least one analytic value at the third time. The display control unit 305 causes the display unit 600 to display the generated graph 720.

[0115] Moreover, the image processing device 300 according to this embodiment acquires data on a first slice and data on a second slice from the subject's eye. photograph The system further includes a selection unit (selection unit 334) for selecting a trained model based on a condition. photograph The conditions include, for example, at least one of the following: imaging date and time, name of imaging part, imaging area, imaging angle of view, and imaging method.

[0116] According to this embodiment, from the tomographic data obtained by photographing the subject's eye at at least two different times and the analysis values ​​obtained from the tomographic data, it is possible to obtain tomographic data or analysis values ​​that are likely to be obtained when the subject's eye is photographed in the future using the trained model. Therefore, when diagnosing the subject's eye, it is possible to present information that is effective for assisting diagnosis in follow-up observation, treatment policy, etc.

[0117] Example 2 In the first embodiment, an example is shown in which a plurality of analytical values ​​for each measurement parameter of the subject eye is generated using a tomographic image of the subject eye and various analytical values ​​obtained from the tomographic image and the planar image of the fundus of the eye. However, the numerical values ​​related to the subject eye used when inspecting the subject eye are not limited to the analytical values ​​measured from the tomographic image described above. In addition, the progress of the condition of the subject eye, such as the pathology, can be known from the measurement values ​​measured on the subject eye. In the second embodiment, a trained model is used that can also input the results of other tests (e.g., axial length, intraocular pressure, etc.), thereby generating various analytical values, measurement values, and the above-mentioned evaluation value.

[0118] Hereinafter, an image processing system 8 according to this embodiment will be described with reference to Fig. 8. In the following description, the differences from the image processing system 1 according to the embodiment will be mainly described. Note that the configuration and processing of the image processing system according to this embodiment that are similar to those of the image processing system 1 according to the embodiment will be indicated by the same reference numerals, and the description thereof will be omitted here.

[0119] 8 shows an example of a schematic configuration of an image processing system 8 according to this embodiment. In the image processing system 8 according to this embodiment, a processing unit 803 of an image processing device 800 is provided with a calculation processing unit 832 according to this embodiment, which replaces the calculation processing unit 332, and a newly provided measurement value acquisition unit 835.

[0120] The measurement value acquiring unit 835 acquires other examination information related to the subject's eye. Specifically, the measurement value acquiring unit 835 may acquire a measurement value input by a user through, for example, a user interface. In addition, the measurement value acquiring unit 835 may cooperate with an electronic medical record system via a network to acquire numerical data registered in the electronic medical record system. Alternatively, information may be acquired from an external reader such as a barcode reader or an RFID reader.

[0121] In this embodiment, the contents of the input / output parameter vectors (c, h, x) shown in Fig. 5 and Fig. 6 in the first embodiment are different. That is, the data input / output to / from the arithmetic processing unit 832 is different. In the first embodiment, the contents of the vectors used during learning and generation of analytical values ​​used tomographic images or analytical values ​​obtained from the tomographic images as Params. In contrast, in this embodiment, not only tomographic images or analytical values ​​obtained from the tomographic images but also test data related to the eye that does not use images, such as axial length tests and intraocular pressure tests, are added to the input / output parameter vectors to generate analytical values, etc.

[0122] For example, in the first embodiment, an example was shown in which N analytical values ​​were used as Params1 to ParamsN. In contrast, in the present embodiment, if the measurement value acquisition unit 835 acquires M measurement values, learning is performed using the analytical values ​​Params1 to ParamsN and the measurement values ​​Params1' to ParamsM. The calculation processing unit 832 acquires learning data from these N analytical values ​​and M measurement values, and uses a learned model obtained using this learning data.

[0123] The arithmetic processing unit 832 according to this embodiment includes a trained model created by providing training data to an arbitrary machine learning model such as deep learning. The arithmetic processing unit 832 inputs tomographic data and measurement values ​​obtained by photographing the subject's eye at a first time, and tomographic data and measurement values ​​obtained by photographing the subject's eye at a second time after the first time, to this trained model. The arithmetic processing unit 832 generates a tomographic image and measurement values ​​from these tomographic data and measurement values ​​using the trained model. The image processing unit 331 performs an analysis process on the generated tomographic image, and multiple types of analysis values ​​are calculated for the tomographic image. The calculation of the multiple types of analysis values ​​may be performed by the arithmetic processing unit 832 using the above-mentioned edge enhancement process, or may be directly generated from the analysis values ​​using the trained model instead of calculation.

[0124] Here, the learning data used for training when obtaining the trained model in this embodiment will be described. The learning data consists of input data corresponding to data actually input to the trained model and teacher data corresponding to data generated by the trained model. In the first embodiment, as input data, two tomographic images obtained by photographing the same test eye under the same shooting conditions at at least two times, a first time and a second time after the first time, or two test results consisting of multiple types of analysis values ​​obtained from the tomographic images are used. In this embodiment, the input data includes a measurement value obtained by measuring the same test eye at the first time and a measurement value obtained by measuring at the second time. In addition, as teacher data, a tomographic image obtained by photographing the same test eye under the same shooting conditions at a third time after the second time or a test result consisting of multiple types of analysis values ​​obtained from the tomographic images, and a third measurement value obtained by measuring at the third time are used.

[0125] When obtaining a trained model, as described herein, the test results and measurement results obtained in advance at least three times in the past are used as training data for training. When using this trained model, two test results and measurement results obtained at a time interval between a first time and a second time in the past corresponding to the input data used when generating the trained model are input. The teacher data corresponds to the analysis values ​​and measurement results generated from the trained model. In other words, by using the trained model, it is possible to obtain analysis values ​​and measurement values ​​that are likely to be obtained from the subject's eye several months to several years in the future corresponding to the time interval between the second time and the third time described above.

[0126] In this embodiment, the time when the tomographic image is acquired and the time when the measurement value is acquired are the same, for example, at the first time. However, in the case of a subject's eye with a slowly progressing disease such as glaucoma, the time when the tomographic image is acquired and the measurement value do not have to be exactly the same. For example, there are cases where an interval of several days is acceptable in consideration of the progression of the disease.

[0127] For example, in an examination for glaucoma, in addition to information obtained from an analysis value from a tomographic image, an intraocular pressure value and an axial length in a severely myopic eye are effective. According to this embodiment, it is possible to generate future analysis values ​​and measurement values ​​for the subject eye using not only the analysis values ​​obtained from the tomographic image but also the measurement values ​​of other types of measurement parameters related to the subject eye. In addition, a doctor or the like determines the above-mentioned evaluation value by referring to these measurement values, and the arithmetic processing unit 832 generates an evaluation value expected in the future using the trained model. Therefore, in the case of glaucoma as an example, by referring to the measurement values ​​that can more appropriately judge the symptoms, it is possible to provide more accurate diagnostic auxiliary data when a doctor or the like makes a diagnosis.

[0128] In this embodiment, the axial length and intraocular pressure are used as the measurement values. However, the measurement parameters for obtaining the measurement values ​​used in this embodiment are not limited to these, and visual field data, visual acuity data, etc. can also be used. In addition, an example in which both the analysis value obtained from the tomographic image and the measurement value obtained by a means other than analyzing the tomographic image are used as input data has been described. However, the input data or the data to be input is not limited to these. For example, the arithmetic processing unit 832 may generate a measurement value that can be measured in the future using only the measurement value, without using the analysis value obtained from the tomographic image.

[0129] As described above, the acquisition unit 301 according to this embodiment further acquires a first measurement value of the subject's eye associated with a first time and a second measurement value of the subject's eye associated with a second time in addition to the analysis values ​​and the like described in the first embodiment. The calculation processing unit 832 generates at least one of a plurality of types of analysis values ​​at a third time from the data on the first tomography, the first measurement value, the data on the second tomography, and the second measurement value. The calculation processing unit 832 may generate at least one measurement value. Also, similar to the first embodiment, evaluation information may be added to the data to be input to the trained model, and the calculation processing unit 832 may also generate the evaluation information. In this case, only the evaluation information may be generated and displayed.

[0130] Alternatively, the acquisition unit 301 according to the present embodiment may acquire a first measurement value of the subject's eye associated with a first time and a second measurement value of the subject's eye associated with a second time. In this case, the calculation processing unit 832 may generate at least one of a plurality of types of measurement values ​​at a third time by previously obtaining a trained model using only the measurement values ​​and inputting the first measurement value and the second measurement value to the trained model. In this case, evaluation information may further be included as the data to be input and the data to be generated. In addition, although a case where tomographic data (tomographic images) is used as the data to be input and the input data for generating the trained model is described here, a plurality of analysis values ​​may also be used.

[0131] According to this embodiment, not only the tomographic data and its analysis values ​​used in the first embodiment, but also other measurement values ​​of the subject's eye such as axial length and intraocular pressure can be used to obtain tomographic data, its analysis values, and measurement values ​​that are likely to be obtained from the subject's eye in the future using a trained model. Therefore, when diagnosing this subject's eye, it is possible to present information that is more effective as a diagnostic aid for follow-up observation, treatment policy, etc.

[0132] Example 3 In the first and second embodiments, examples were shown in which information that aids in the diagnosis of the test eye is presented using a tomographic image obtained by photographing the test eye, an analysis value obtained from the tomographic image, and a measurement value obtained by measuring the test eye. However, when a lesion or abnormality actually exists in the test eye, it is rare for a doctor or the like to leave it as it is and observe the progress, and the test eye is treated with medication, surgery, or the like depending on the situation. In the third embodiment, a trained model is used, including the presence or absence of treatment, and various analysis values ​​are generated with reference to the treatment. In addition, an evaluation value regarding the future condition of the test eye, such as the condition of the disease, is generated, for example, for the purpose of confirming the effect of the treatment.

[0133] 9 and 10, an image processing system 9 according to this embodiment and a report screen displayed on the display unit will be described below. In the following description, the differences from the image processing systems according to embodiments 1 and 2 will be mainly described. Note that the configurations and processes of the image processing system 9 according to this embodiment that are similar to those of the image processing systems 1 and 8 according to embodiments 1 and 2 will be indicated by the same reference numerals, and description thereof will be omitted here.

[0134] An example of a schematic configuration of an image processing system 9 according to this embodiment is shown in Fig. 9. In the image processing system 9 according to this embodiment, a processor 903 of an image processing device 900 is provided with a processor 932 according to this embodiment, which replaces the processor 332, and a newly provided treatment information acquisition unit 935.

[0135] The treatment information acquisition unit 935 acquires information related to the treatment of the subject's eye. Specifically, the treatment information acquisition unit 935 may acquire measurement values ​​input by a user through a user interface, for example. The treatment information acquisition unit 935 may also cooperate with an electronic medical record system via a network to acquire numerical data registered in the electronic medical record system. Alternatively, the treatment information acquisition unit 935 may acquire information from an external reader, such as a barcode reader or an RFID reader.

[0136] In this embodiment, the contents of the input / output parameter vectors (c, h, x) shown in Fig. 5 and Fig. 6 in the first embodiment are different. That is, the data input / output to the arithmetic processing unit 932 is different. In the first embodiment, the contents of the vectors used during learning and generation of the analytical values ​​used the tomographic images or the analytical values ​​obtained from the tomographic images as Params. In contrast, in this embodiment, not only the tomographic images or the analytical values ​​obtained from the tomographic images but also the evaluation values ​​regarding the condition of the subject's eye specified by a doctor or the like and the treatment method are added to the input / output parameter vectors to generate the analytical values.

[0137] For example, in the first embodiment, an example was shown in which N analytical values ​​were used as Params1 to ParamsN. In contrast, in the present embodiment, if the information acquired by the treatment information acquisition unit 935 is information S on the contents of treatment such as the surgical technique and the type of medication, learning is performed using the analytical values ​​of Params1 to ParamsN and ParamsS. The calculation processing unit 932 acquires learning data from these N analytical values ​​and the information S on the treatment, and uses a trained model obtained using this learning data.

[0138] Next, referring to Fig. 10, an example of a screen displayed on the display unit 600 according to this embodiment is shown. In this embodiment, a second selector 1021 is displayed on the display screen 1010 together with the selector 721. Here, an example is shown in which treatment 1 selected by the second selector 1021 at the present time is performed on the examinee's eye in a time series graph 1020 for Params1. The graph 720 shows that if treatment 1 is performed on the examinee's eye at the time shown by the white circle 1023, the predicted value of future retinal thickness will be the star 1024.

[0139] In this embodiment, for example, when a certain treatment (e.g., medication) is performed on the subject's eye, the future analysis value of the selected measurement parameter can be obtained based on the analysis value obtained by imaging. Furthermore, the type of treatment can be selected by switching the second selector 1021, and the analysis value obtained as the star 1024 and the probability 1025, which is the progression prediction value, change depending on the selected treatment.

[0140] In this embodiment, the arithmetic processing unit uses the learned model to generate a tomographic image captured at a third time after the second time from tomographic data obtained by photographing the subject's eye at a first time and tomographic data obtained by photographing the subject's eye at a second time after the first time. At that time, if treatment is administered to the subject's eye at least at either the first time or the second time, the treatment is also input to the learned model together with the tomographic data. The generated tomographic image is subjected to an analysis process in the image processing unit 331, and multiple types of analytical values ​​in the tomographic image are calculated. The calculation of the multiple types of analytical values ​​may be performed by the arithmetic processing unit 332, or may be generated using the learned model instead of calculation.

[0141] Furthermore, the evaluation value of the subject's eye at least at the first time and the second time may also be input to the trained model. This allows the effect of the treatment applied to the subject's eye to be numerically understood from the generated evaluation value. Note that data indicating the presence or absence of treatment may be input to the trained model as, for example, 0 when no treatment was performed and 1 when treatment was performed. Furthermore, in this case, the input value may be set to 2 or 3 depending on the type of treatment (e.g., drug treatment, surgery, etc.) in addition to whether or not treatment was performed.

[0142] Here, the learning data used for training when obtaining the trained model in this embodiment will be described. In the first embodiment, as input data, two tomographic images obtained by photographing the same subject's eye under the same photographing conditions at at least two times, a first time and a second time after the first time, and their evaluation information can be used. Alternatively, instead of the two tomographic images, two examination results consisting of a plurality of types of analysis values ​​obtained from the tomographic images and their evaluation information can be used. In this embodiment, the contents of the treatment administered to the subject's eye at at least one of the first time and the second time are also used as input data. In addition, as teacher data, a tomographic image obtained by photographing the same subject's eye under the same photographing conditions at a third time after the second time, or an examination result consisting of a plurality of types of analysis values ​​obtained from the tomographic image, and the evaluation information of the subject's eye at the third time are used.

[0143] When obtaining a trained model, as described herein, training is performed using test results and their evaluation information obtained in advance at least three times in the past as training data. When using this trained model, two test results obtained at a first time and a second time in the past corresponding to the input data used when generating the trained model, evaluation information of the test eye at either time, and information on treatment are input. The teacher data corresponds to the analysis value and the evaluation information generated from the trained model. In other words, by using the trained model, it is possible to obtain an analysis value that is likely to be obtained from the test eye several months to several years in the future corresponding to the time interval between the second time and the third time described above, and the evaluation information of the test eye at that time.

[0144] More specifically, for example, when a specific drug is administered to a subject eye having glaucoma at the first time described above, one of the input data is an analysis value obtained from a tomographic image of the subject eye at that time, evaluation information of the subject eye at that time, and a flag indicating administration of the drug. When no drug is administered at the second time, another of the input data is an analysis value obtained from a tomographic image of the subject eye at that time, and evaluation information of the subject eye at that time. In contrast, as a result of using the trained model, an analysis value of a tomographic image taken at a third time after the second time for the same subject eye and evaluation information of the subject eye at that time are generated.

[0145] In this case, the learning data used to generate the trained model is data on the subject's eye that has been given the above-mentioned specific drug and for which a time-series analysis value has been obtained in advance. As the actual input data, the analysis value and evaluation information at the time of drug administration are one of the input data. Another of the input data is the analysis value obtained by photographing the subject's eye at a time interval substantially equal to the time interval between the first time and the second time after drug administration, and the evaluation information at that time. Furthermore, the teacher data is the analysis value obtained by photographing the subject's eye at a time interval substantially equal to the time interval between the second time and the third time, and the evaluation information at that time. Note that this time interval does not need to be strictly the same, and an error of several days to several weeks may be acceptable depending on, for example, the progression speed of the disease.

[0146] Here, the learning data used to obtain the trained model according to this embodiment, the data input to the trained model, and the generated results are described using an example in which medication is administered at the first time. However, the timing of treatment, such as medication, is not limited to the example described here, and may be administered at the second time, or may be administered at both the first time and the second time. In this case, by obtaining the corresponding trained model in advance, it is possible to obtain analysis values ​​and evaluation information that are likely to be obtained as a result of treatment.

[0147] As described above, in this embodiment, the treatment information acquisition unit 935 acquires information on the treatment performed on the subject's eye at least at the first time and at the second time. In this case, the treatment is associated with at least one of the data on the first tomography and the data on the second tomography. In this embodiment, the calculation processing unit 932 generates at least one of the multiple types of analysis values ​​at the third time from the data on the first tomography, the data on the second tomography, and the information on the treatment associated with at least one of these data. Note that the data on the tomography described here may be the above-mentioned multiple analysis values ​​obtained from a tomography image, or may be tomography data (tomography image).

[0148] The image processing device 900 according to this embodiment is not limited to the aspect in which the tomographic image and its analysis value described in the embodiment 1, and the measurement value described in the embodiment 2 are input to the arithmetic processing unit. For example, the image processing device 900 may be configured to input evaluation information on the subject's eye, at least one piece of subject's eye information indicating the state of the subject's eye, and information on the treatment of the subject's eye to the arithmetic processing unit 932.

[0149] In this case, the evaluation target is the subject's eye that has been treated at least at the first time and at the second time after the first time. The treatment information acquisition unit 935 acquires information about this treatment. The calculation processing unit 932 calculates the first measurement data of the subject's eye associated with the first time and the second measurement data of the subject's eye associated with the second time, for example, from a tomographic image. Note that here, the measurement data includes the analysis value obtained by image analysis of the tomographic image of the subject's eye described in Example 1 and the measurement value obtained by measuring the subject's eye described in Example 2, and the data actually obtained from the subject's eye is considered to be the measurement data. The calculation processing unit 932 inputs the evaluation information associated with at least one of the first time and the second time, the first and second measurement data, and information about the treatment, into the trained model. The trained model can also generate evaluation information of the subject's eye at a third time after the second time using the input measurement data and information.

[0150] In this case, the display control unit 305 generates third measurement data at the third time, which is measurement data predicted to be obtained at the third time of the subject's eye generated using the learned model. The display control unit 305 generates a graph 1020 using the generated third measurement data, the first measurement data, and the second measurement data. The display control unit 305 can display this graph 1020 on the display screen 1010 of the display unit 600. In addition, at this time, the display control unit 305 can display the generated evaluation information (probability 1025) at the third time and the type of treatment (second selector 1021) on the display screen 1010. At this time, the type of treatment can include at least one of surgery and medication. In addition, the evaluation information on the condition of the subject's eye is information presented by a user such as a doctor as a diagnosis result of the subject's eye based on a tomographic image, its analysis value, the above-mentioned measurement value, or other observation images, and is input by the user.

[0151] According to this embodiment, not only the tomographic data and its analysis value used in the first embodiment but also the presence or absence of treatment performed on the subject eye can be used to obtain tomographic data that is likely to be obtained from the subject eye in the future, its analysis value, and evaluation information indicating the state of the subject eye by the trained model. Therefore, when diagnosing this subject eye, it is possible to present to a doctor or the like the possibility of how the state of the subject eye will change due to treatment, and it is possible to present information that is more effective in assisting diagnosis for follow-up observation, treatment policy, etc.

[0152] (Variation 1) In the first to third embodiments, the acquisition unit 301 acquires tomographic data obtained by photographing the subject's eye with the OCT device 200, and the image processing device uses the tomographic data to generate the above-mentioned analysis values, etc. However, the configuration in which the acquisition unit 301 acquires these tomographic data is not limited to this. For example, the acquisition unit 301 may acquire tomographic data that has already been photographed and stored in a server such as the external storage device 500 connected to the image processing device via a LAN, a WAN, or the Internet, and perform the above-mentioned series of processes using the tomographic data. In this case, the process related to the photographing is omitted, and instead, a photographed tomographic image is acquired. Then, in step S304, the retinal layer is detected and the retinal thickness is measured. This allows the detection of the retinal layer to be performed when necessary without processing during the photographing, and allows the user to focus only on the photographing during the photographing.

[0153] (Variation 2) In the first embodiment, an image processing device was described that obtains an analysis value and evaluation information that are likely to be obtained in the future by inputting tomographic data or an analysis value obtained from the tomographic data, and furthermore, evaluation information of the subject's eye into a trained model. In the second embodiment, an image processing device was described that obtains an analysis value and measurement value that are likely to be obtained in the future by inputting tomographic data or a measurement value of the subject's eye together with the analysis value obtained from the tomographic data into a trained model. However, the aspects of the present invention are not limited to these embodiments, and for example, a measurement value of the subject's eye and evaluation information obtained by referring to the measurement value may be input into a trained model to obtain a measurement value and evaluation information that are likely to be obtained in the future.

[0154] Furthermore, in the third embodiment, only the flag related to the treatment and the measurement value of the subject eye may be input to the trained model to obtain a measurement value that is likely to be obtained in the future. Furthermore, by further inputting evaluation information, evaluation information that is likely to be obtained in the future may also be obtained. Furthermore, in the above-mentioned embodiment, the input data is data obtained at two times, one associated with the first time and the other associated with the second time, and data that is likely to be obtained at the third time is generated from these data. However, the number of input data to be input or input data used in generating the trained model is not limited to the two exemplified, and can be further increased. By increasing the input data and input data, the accuracy of data predicted to be obtained (output data) can be improved. Furthermore, when the number of actually input data is less than the number of input data used in generating the trained model, for example, appropriate output data can be obtained by complementing missing data using an appropriate method. For example, in the second embodiment, when a trained model obtained by using five analysis values ​​and measurement values ​​arranged in a chronological order as input data is used, even if there are three or four analysis values, it is possible to output data by complementing the missing analysis values. For example, it is possible to estimate the missing analytic value using at least two analytic values ​​before and after the missing analytic value. In this case, the missing analytic value may be estimated as an average value, a median value, or the like obtained from at least two analytic values. The missing analytic value may also be estimated using an approximate line or an approximate curve obtained from at least two analytic values. The multiple types of analytic values ​​associated with the first time and the multiple types of analytic values ​​associated with the second time do not need to be exactly the same type, and some types may be different, or some types may be missing. The above content can also be applied during learning to the extent that there is no contradiction.

[0155] The trained model can be provided in the arithmetic processing unit 332, 832 as described above. In this case, the trained model can be configured, for example, as a software module executed by a processor such as a CPU. The trained model may also be provided in another server connected to the image processing device 300, 800. In this case, by connecting to a server equipped with the trained model via any network such as the Internet that can be connected to the arithmetic processing unit 332, 832, it is possible to generate analytical values, measurement values, evaluation information, and the like using the trained model.

[0156] (Other embodiments) Each of the above-mentioned embodiments is an embodiment of the present invention realized as an image processing device. However, the embodiments of the present invention are not limited to the image processing device or the image processing method described in the embodiments. The present invention can also be realized as software that runs on a computer. The CPU of the image processing device controls the entire computer using computer programs and data stored in the RAM or ROM. It can also control the execution of software corresponding to each part of the image processing device to realize the functions of each part and execute each process performed during image processing. The user interface such as buttons and the display layout are not limited to the above-mentioned aspects.

[0157] Furthermore, the present invention can be realized by supplying a program for implementing one or more of the functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in a computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0158] Although the present invention has been described above with reference to the examples and modifications, the present invention is not limited to the above examples and modifications. The present invention also includes inventions that have been modified within the scope of the present invention and inventions equivalent to the present invention. In addition, the above-mentioned examples and modifications can be appropriately combined within the scope of the present invention. [Explanation of symbols]

[0159] 1, 8: image processing system, 200: tomographic imaging apparatus, 300, 800: image processing apparatus, 301: image acquisition section, 303: processing section, 305: display control section, 311: tomographic image generating section, 332, 832: calculation processing section, 333: imaging condition acquisition section, 334: selection section, 835: measurement value acquisition section

Claims

1. an acquisition means for acquiring data relating to a slice including a plurality of types of analysis values ​​of a subject's eye and associated with imaging conditions including an imaging site, the data including a first slice of the subject's eye associated with a first time and a second slice of the subject's eye associated with a second time after the first time; A selection means for selecting a trained model corresponding to an imaging condition of the acquired first tomographic data and the acquired second tomographic data, in which at least the imaging site of the acquired first tomographic data and the acquired second tomographic data are the same, from among trained models corresponding to each of a plurality of imaging conditions for acquiring data related to a tomographic section of the subject's eye; A generating means for generating at least one of a plurality of types of analysis values ​​of the subject's eye at a third time after the second time from the acquired data on the first cross-section and the acquired data on the second cross-section by using the selected trained model; An image processing device comprising:

2. the data on the first tomography is a plurality of types of analysis values ​​obtained by analyzing the tomography image captured at the first time, 2. The image processing apparatus according to claim 1, wherein the data relating to the second tomography is a plurality of types of analysis values ​​obtained by analyzing the tomographic image captured at the second time.

3. The image processing device described in claim 1 or 2, characterized in that the trained model is obtained using learning data in which input data include multiple types of analytical values ​​obtained by analyzing a tomographic image obtained by photographing the test eye at a fourth time before the first time and multiple types of analytical values ​​obtained by analyzing a tomographic image obtained by photographing the test eye at a fifth time before the first time and after the fourth time, and teacher data includes multiple types of analytical values ​​obtained by analyzing a tomographic image obtained by photographing the test eye at a sixth time before the first time and after the fifth time.

4. an acquisition means for acquiring data relating to a section including a tomographic image obtained by photographing a subject's eye, the data being associated with photographing conditions including a photographing site, the data relating to a first section of the subject's eye associated with a first time, and data relating to a second section of the subject's eye associated with a second time after the first time; A selection means for selecting a trained model corresponding to an imaging condition of the acquired first tomographic data and the acquired second tomographic data, in which at least the imaging site of the plurality of imaging conditions is the same, from among trained models corresponding to each of a plurality of imaging conditions for acquiring data on a tomographic section of the subject's eye; A generating means for generating at least one of a plurality of types of analysis values ​​of the subject's eye at a third time after the second time from the acquired data on the first cross-section and the acquired data on the second cross-section by using the selected trained model; An image processing device comprising:

5. The image processing device according to claim 4, characterized in that the generating means generates data relating to a third tomography of the test eye using the trained model, the data relating to a tomography including a tomographic image obtained by photographing the test eye, and generates at least one of the multiple types of analytical values ​​using the generated data relating to the third tomography.

6. The image processing device described in claim 4 or 5, characterized in that the trained model is obtained using learning data in which a tomographic image obtained by photographing the test eye at a fourth time before the first time and a tomographic image obtained by photographing the test eye at a fifth time before the first time and after the fourth time are used as input data, and a tomographic image obtained by photographing the test eye at a sixth time before the first time and after the fifth time is used as teacher data.

7. 7. The image processing apparatus according to claim 3, wherein the elapsed time between the second time and the third time is substantially equal to the elapsed time between the fifth time and the sixth time.

8. The acquiring means further acquires a first measurement value of the subject's eye associated with the first time and a second measurement value of the subject's eye associated with the second time, The image processing device according to any one of claims 1 to 7, characterized in that the generation means generates at least one of the multiple types of analysis values ​​at the third time from data regarding the first fault, the first measurement value, data regarding the second fault, and the second measurement value.

9. the acquiring means acquires information on a treatment associated with at least one of the data on the first cross-section and the data on the second cross-section, the treatment being performed on the subject's eye at the first time or the second time; 9. The image processing device according to claim 1, wherein the generating means generates at least one of the plurality of types of analysis values ​​at the third time from data on the first slice, data on the second slice, and information on the treatment.

10. the acquiring means acquires evaluation information regarding a state of the subject's eye at the first time and the second time, 10. The image processing apparatus according to claim 1, wherein the generating means generates evaluation information at the third time in addition to at least one of the plurality of types of analysis values ​​at the third time.

11. The image processing apparatus according to claim 10 , further comprising a display control unit for causing the generated evaluation information to be displayed on a display unit.

12. The image processing device according to any one of claims 1 to 11, further comprising a display control means for causing a display unit to display a first analytical value and a second analytical value obtained from the data on the first fault and the data on the second fault, the first analytical value and the second analytical value being of the same type as at least one of the first analytical value and the second analytical value at the third time, and a graph obtained using the at least one analytical value at the third time.

13. 2. The image processing apparatus according to claim 1, wherein the imaging conditions include at least one of an imaging date and time, a name of an imaging part, an imaging area, an imaging angle of view, and an imaging method.

14. 14. The image processing device according to claim 1, wherein the multiple types of analysis values ​​include the overall retinal thickness, thickness between any layers, choroidal thickness, and retinal shape characteristics in the posterior segment of the test eye, the retinal thickness and Cup and Disc in the optic disc, and the corneal thickness, corneal curvature, anterior chamber depth, total lens curvature, and posterior lens curvature in the anterior segment.

15. an acquisition means for acquiring data including an imaging site of a subject's eye that has been treated at least at a first time and / or a second time subsequent to the first time, the acquisition means acquiring first measurement data of the imaging site of the subject's eye associated with the first time, second measurement data of the imaging site of the subject's eye associated with the second time, and evaluation information on a state of the imaging site of the subject's eye associated with at least one of the first time and the second time; A selection means for selecting a learned model corresponding to the imaging conditions of the acquired first measurement data and the acquired second measurement data, in which at least the imaging site of the imaging conditions is the same, from among learned models corresponding to each of a plurality of imaging conditions for acquiring measurement data of the subject's eye; a generating means for generating evaluation information regarding a state of the imaging site of the subject's eye at a third time after the second time from information regarding the treatment, the first measurement data, the second measurement data, and the evaluation information by using the selected trained model; and An image processing device comprising:

16. The generating means generates third measurement data of the imaging site of the subject's eye at the third time using the selected trained model, The image processing device described in claim 15, further comprising a display control means for causing a display unit to display the generated third measurement data, a graph obtained using the first measurement data and the second measurement data, the generated evaluation information at the third time, and the type of treatment.

17. 17. The image processing apparatus according to claim 15, wherein the evaluation information on the state of the imaging portion of the subject's eye is information input by a user as a diagnosis result of the subject's eye.

18. 18. The image processing device according to claim 15, wherein the treatment includes at least one of surgery and medication.

19. the first measurement data is obtained from a tomographic image obtained by photographing the subject's eye at the first time, 19. The image processing apparatus according to claim 15, wherein the second measurement data is obtained from a tomographic image obtained by photographing the subject's eye at the second time.

20. an acquiring step of acquiring data relating to a slice including a plurality of types of analysis values ​​of the subject's eye and associated with imaging conditions including an imaging site, the data including: data relating to a first slice of the imaging site of the subject's eye associated with a first time; and data relating to a second slice of the imaging site of the subject's eye associated with a second time after the first time; A selection step of selecting a trained model corresponding to an imaging site included in imaging conditions at the time of imaging the data on the first tomography and the data on the second tomography from trained models trained using respective analysis values ​​of the data on the multiple tomography of the subject's eye; A generation process of generating at least one of a plurality of types of analysis values ​​of the imaging site of the subject's eye at a third time after the second time from the acquired data on the first tomography and the acquired data on the second tomography using the selected trained model; 13. An image processing method comprising:

21. an acquiring step of acquiring data relating to a section including a tomographic image obtained by photographing a subject's eye, the data being associated with photographing conditions including a photographing site, the data relating to a first section of the photographing site of the subject's eye associated with a first time, and data relating to a second section of the photographing site of the subject's eye associated with a second time after the first time; A selection step of selecting a trained model corresponding to an imaging site included in the imaging conditions at the time of imaging the first tomographic data and the second tomographic data from among trained models corresponding to each of a plurality of imaging conditions for acquiring data related to a tomography of the subject's eye; A generation process of generating at least one of a plurality of types of analysis values ​​of the imaging site of the subject's eye at a third time after the second time from the acquired data on the first tomography and the acquired data on the second tomography using the selected trained model; 13. An image processing method comprising:

22. A method of operating an image processing apparatus, comprising the steps of: A step of acquiring, by the image processing device, first measurement data of an imaging site of the test eye associated with the first time, second measurement data of the imaging site of the test eye associated with the second time, and evaluation information on a state of the imaging site of the test eye associated with at least one of the first time and the second time, for a test eye that has been treated at at least one of a first time and a second time after the first time; A step in which the image processing device selects, from among trained models corresponding to each of a plurality of shooting conditions for acquiring measurement data of the subject's eye, a trained model corresponding to an imaging site included in the shooting conditions of the acquired first measurement data and the acquired second measurement data; A step of generating evaluation information regarding a state of the imaging site of the subject's eye at a third time after the second time from information regarding the treatment, the first measurement data, the second measurement data, and the evaluation information by the image processing device using the selected trained model; 13. A method for operating an image processing device, comprising:

23. A program which, when executed by a processor, causes the processor to carry out each step of the image processing method according to claim 20 or 21, or the method of operating an image processing device according to claim 22.

Images