Information processing device, information processing method, and program

Abstract

To improve accuracy of diagnosis.SOLUTION: An image generation device comprises: a generation unit which generates a first analytic result having two-dimensional thickness data of the retinal layer of a subject eye by using a tomographic image of the subject eye and generates a second analytic result different from the first analytic result by using a machine learning model obtained by learning with a plurality of thickness maps and the first analytic result; and a display control unit which controls a display unit to display information on the second analytic result.SELECTED DRAWING: Figure 5

Description

Technical Field

[0001] The present disclosure relates to an information processing apparatus, an information processing method, and a program.

Background Art

[0002] As ophthalmic devices, there are devices for acquiring two-dimensional fundus images of an eye to be examined (hereinafter referred to as a fundus camera device), and devices for acquiring tomographic images of an eye to be examined using optical coherence tomography (OCT) with low coherence light (hereinafter referred to as an OCT device), which have been put into practical use.

[0003] If the thickness of each layer can be measured from the tomographic image of the retina taken by these devices, it becomes possible to quantitatively diagnose the progression of diseases such as glaucoma and the recovery status after treatment. In order to quantitatively measure the thickness of these layers, by comparing the statistical value of the thickness of the retinal layer in a normal eye (normal eye database) stored with the thickness of the retinal layer obtained from the eye to be examined, a portion deviating from the statistical value of the thickness of the normal eye is displayed. Note that when the axial length of the eye becomes long, the entire retina becomes thinner as the eyeball elongates, and the difference may become large when compared with the statistical value of the thickness of a general normal eye.

[0004] Here, in Patent Document 1, in addition to a database (first database) storing fundus layer thickness information regarding a plurality of eyes having a long axial length, a normal eye database (second database) is provided, and a technique for acquiring analysis information by referring to either one of them is disclosed.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] Here, the statistical value of the thickness stored in the normal eye database is the average value of multiple eyes. On the other hand, there are individual differences in the normal values of thickness. For example, even when there is a difference from the statistical value of the thickness stored in the normal eye database, it may be a normal thickness for that individual. In that case, if a diagnosis is made with reference to the normal eye database, even a normal test eye may be diagnosed as abnormal.

[0007] In addition to thickness, there are also individual differences in the running shape of nerve fibers. For example, although the running shape of nerve fibers is considered normal when it is arcuate, even when it is not arcuate, it may be a normal running shape for that individual. In that case, for example, if a diagnosis is made with reference to the normal eye database, even a normal test eye may be diagnosed as abnormal.

[0008] Therefore, the object of the present disclosure is to improve the accuracy of diagnosis.

Means for Solving the Problems

[0009] The image generation device of the present disclosure generates a first analysis result having two-dimensional thickness data of the retinal layer of the test eye using a tomographic image of the test eye, and uses a machine learning model learned using a plurality of thickness maps and the first analysis result to generate a second analysis result different from the first analysis result, and includes a display control unit that controls a display unit to display information regarding the second analysis result.

Advantages of the Invention

[0010] According to the present disclosure, the accuracy of diagnosis can be improved.

Brief Description of the Drawings

[0011]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Embodiments for Carrying Out the Invention

[0012] Hereinafter, exemplary embodiments for carrying out the disclosed technology will be described in detail with reference to the drawings. However, the dimensions, materials, shapes, relative positions of components, etc. described in the following embodiments are arbitrary and can be changed according to the configuration of the apparatus to which the disclosed technology is applied or various conditions. Also, in the drawings, the same reference numerals are used between the drawings to indicate elements that are the same or functionally similar.

[0013] [Embodiment 1] In this embodiment, as an example of an ophthalmic device according to the disclosed technology, an ophthalmic device that performs both OCT imaging and fundus imaging using visible light will be described. The ophthalmic device of this embodiment detects retinal layers from the captured OCT data, and generates two-dimensional thickness data of the retinal layers from the detected layer information. Then, two-dimensional thickness data inferred by the image generation model is generated from the two-dimensional thickness data.

[0014] Note that the image processing system (information processing system) in this embodiment may be configured such that an ophthalmic device (composite examination unit) according to the disclosed technology is connected to a network, receives an order (examination instruction) from a doctor's personal computer, performs an examination of the eye to be examined according to the order, and transmits the result to the doctor's personal computer.

[0015] FIG. 1 is a diagram showing the configuration of an image processing system 100 (information processing system) including an image processing device 300 according to this embodiment. As shown in FIG. 1, the image processing system 100 is configured such that the image processing device 300 is communicably connected to an optical coherence tomography (OCT) device 200 (also referred to as an OCT imaging device), a fundus imaging device 400, an external storage unit 500, a display unit 600, and an input unit 700 via an interface. Here, communicably connected includes not only direct communication between devices but also communication via other devices. Note that the image processing device 300 is an example of an information processing device.

[0016] The OCT device 200 is a device that captures a tomographic image of the eye. Devices used for the OCT device include, for example, spectral domain optical coherence tomography (SD-OCT) and swept source optical coherence tomography (SS-OCT). Since the OCT device 200 is a known device, a detailed description thereof is omitted, and here, the capture of the tomographic image performed according to an instruction from the image processing device 300 will be described.

[0017] In FIG. 1, the galvanometer mirror 201 is for scanning the fundus with the measurement light and defines the imaging range of the fundus by OCT. Also, the drive control unit 202 defines the imaging range and the number of scanning lines (scanning speed in the planar direction) in the planar direction of the fundus by controlling the drive range and speed of the galvanometer mirror 201. Here, for simplicity, the galvanometer mirror is shown as one unit, but actually it is composed of two mirrors, one for X-scanning and one for Y-scanning, and a desired range on the fundus can be scanned with the measurement light.

[0018] The focus 203 is for focusing on the retinal layer of the fundus through the anterior segment of the eye which is the subject. The measurement light is focused on the retinal layer of the fundus through the anterior segment of the eye which is the subject by a focus lens (not shown). The measurement light that irradiates the fundus is reflected and scattered back by each retinal layer.

[0019] The internal fixation lamp 204 is composed of a display unit 241 and a lens 242. As the display unit 241, one in which a plurality of light-emitting diodes (LDs) are arranged in a matrix is used. The lighting position of the light-emitting diodes is changed according to the site to be imaged under the control of the drive control unit 202. The light from the display unit 241 is guided to the eye to be examined through the lens 242. The light emitted from the display unit 241 is 520 nm, and a desired pattern is displayed by the drive control unit 202.

[0020] The coherence gate stage 205 is controlled by the drive control unit 202 to cope with differences in the axial length of the eye to be examined, etc. The coherence gate represents the position where the optical distances of the measurement light and the reference light in OCT are equal. Furthermore, by controlling the position of the coherence gate as an imaging method, imaging on the retinal layer side or deeper than the retinal layer is controlled. Here, the structure and image of the eye acquired by the image processing system will be described with reference to FIG. 2.

[0021] Fig. 2(a) shows a schematic diagram of the eyeball. In Fig. 2(a), C represents the cornea, CL represents the lens, V represents the vitreous body, M represents the macula (the center of the macula represents the fovea), and D represents the optic nerve papilla. The tomographic imaging device 200 according to the present embodiment will mainly be described for the case of imaging the posterior pole of the retina including the vitreous body, the macula, and the optic nerve papilla. Although not described in the present embodiment, the tomographic imaging device 200 can also image the anterior eye part of the cornea and the lens.

[0022] Fig. 2(b) shows an example of a tomographic image when the tomographic imaging device 200 images the retina. In Fig. 2(b), AS represents the unit of image acquisition in the OCT tomographic image called an A scan. A plurality of these A scans gather to form one B scan. And this B scan is called a tomographic image (or tomogram). In Fig. 2(b), V represents the vitreous body, M represents the macula, D represents the optic nerve papilla, and La represents the lamina cribrosa. Also, L1 represents the boundary between the inner limiting membrane (ILM) and the nerve fiber layer (NFL), L2 represents the boundary between the nerve fiber layer and the ganglion cell layer (GCL), L3 represents the inner segment - outer segment junction of photoreceptor cells (ISOS), L4 represents the retinal pigment epithelium layer (RPE), L5 represents the Bruch's membrane (BM), and L6 represents the choroid. In the tomographic image, the horizontal axis (the main scanning direction of OCT) is the x - axis, and the vertical axis (the depth direction) is the z - axis.

[0023] Fig. 2(c) shows an example of a fundus image acquired by the fundus imaging device 400. The fundus imaging device 400 is a device for imaging the fundus image of the eye, and examples of such a device include a fundus camera and an SLO (Scanning Laser Ophthalmoscope), etc. In Fig. 2(c), M represents the macula, D represents the optic nerve papilla, and the thick curve represents the blood vessels of the retina. In the fundus image, the horizontal axis (the main scanning direction of OCT) is the x - axis, and the vertical axis (the sub - scanning direction of OCT) is the y - axis. Note that the device configurations of the tomographic imaging device 200 and the fundus imaging device 400 may be integrated or separate.

[0024] The image processing apparatus 300 includes an image acquisition unit 301, a storage unit 302, an image processing unit 303, an instruction unit 304, and a display control unit 305. The image acquisition unit 301 consists of a tomographic image generation unit 311, acquires the signal data of the tomographic image captured by the tomographic image capturing apparatus 200, and generates a tomographic image by performing signal processing. Also, it acquires the fundus image data captured by the fundus image capturing apparatus 400. Then, it stores the generated tomographic image and fundus image in the storage unit 302. The storage unit 302 stores various kinds of information and various kinds of data (including image data). Furthermore, the storage unit 302 stores a program for realizing various functions by being read and executed by, for example, the processing circuit 250. For example, the storage unit 302 is realized by a semiconductor memory element such as a RAM (Random Access Memory), a flash memory, a hard disk, an optical disk, etc. The storage unit 302 stores statistical values (normal eye database) of the thickness of the retinal layers related to multiple eyes.

[0025] The image processing unit 303 consists of an analysis unit 331 and a generation unit 332. The analysis unit 331 detects the retinal layers from the captured OCT data, and generates two-dimensional thickness data (hereinafter also referred to as a thickness map) of the retinal layers as a first analysis result using the detected layer information. The generation unit 332 has a data generation engine (machine learning model) including a machine learning engine, and generates a thickness map as a second analysis result with the thickness map generated by the analysis unit 331 as the first analysis result as the input.

[0026] The display control unit 305 controls the display unit 600. The display control unit 305 controls the display unit to display information regarding the second analysis result. The information regarding the second analysis result is, for example, the thickness map itself as the second analysis result. Also, the information regarding the second analysis result is, for example, the comparison result of comparing the second analysis result with the first analysis result. Here, the comparison result includes, for example, the difference information between the second analysis result and the first analysis result.

[0027] Note that the analysis unit 331 and the generation unit 332 do not necessarily have to be separate entities and may be implemented as a single generation unit. Alternatively, the generation unit 332 may be provided in an external device different from the image processing device 3100. In that case, the image processing device 3100 outputs the first analysis result generated by the analysis unit 331 to the external device. The external device generates a second analysis result using the input first analysis result and outputs the generated second analysis result to the image processing device 3100. Then, the image processing device 3100 displays the input second analysis result on the display unit 600.

[0028] Note that the image processing device 300 may be configured using, for example, a general-purpose computer. Further, the image processing device 300 may be configured using a dedicated computer for an ophthalmic device. The image processing device 300 includes a processor (not shown) and a storage medium including a memory such as an optical disk or a ROM (Read Only Memory). The processor may be a CPU (Central Processing Unit), an MPU (Micro Processing Unit), or the like. Note that the processor is not limited to a CPU or an MPU and may be a GPU (Graphics Processing Unit), an FPGA (Field-Programmable Gate Array), or the like. Each component other than the storage unit 302 of the image processing device 300 may be configured by a software module executed by a processor such as a CPU, an MPU, or a GPU. Further, each of the components may be configured by a circuit or an independent device that performs a specific function such as an ASIC. Further, the processor or the circuit may include a digital signal processor (DSP), a data flow processor (DFP), or a neural processing unit (NPU).

[0029] The external storage unit 500 holds the information regarding the eye to be examined (measurement values regarding the eye such as the patient's name, race, age, gender, and axial length of the eye), the captured image data, the imaging parameters, the image analysis parameters, and the parameters set by the operator, in association with each other.

[0030] The input unit 700 is, for example, a mouse, a keyboard, a touch operation screen, etc. The operator gives instructions to the image processing apparatus 300, the tomographic imaging apparatus 200, and the fundus imaging apparatus 400 via the input unit 700.

[0031] Next, with reference to FIG. 3, the processing procedure of the image processing apparatus 300 according to the present embodiment will be described. FIG. 3 is a flowchart showing the flow of the operation processing including imaging and analysis of the entire system in the present embodiment.

[0032] <Step S301> In step S301, an unillustrated eye examination information acquisition unit acquires a subject identification number from the outside as information for identifying the eye to be examined. Then, based on the subject identification number, information regarding the eye to be examined held in the external storage unit 500 is acquired and stored in the storage unit 302.

[0033] <Step S302> In step S302, the eye to be examined is scanned and imaged. When the operator selects an unillustrated scan start, the tomographic imaging apparatus 200 controls the drive control unit 202 and operates the galvanometer mirror 201 to scan the tomographic image. The galvanometer mirror 201 is composed of an X scanner for the horizontal direction and a Y scanner for the vertical direction. Therefore, by changing the directions of these scanners respectively, it is possible to scan in the horizontal direction (X) and the vertical direction (Y) in the device coordinate system. And by changing the directions of these scanners simultaneously, it is possible to scan in the direction synthesized by the horizontal direction and the vertical direction, so it is possible to scan in any direction on the fundus plane.

[0034] In photographing, various photographing parameters are adjusted. Specifically, at least the position of the internal fixation light, the scan range, the scan pattern, the coherence gate position, and the focus are set. The drive control unit 202 controls the light-emitting diode of the display unit 241 to control the position of the internal fixation light 204 so as to photograph the center of the macula or the optic disc. The scan pattern is set to a raster scan, radial scan, cross scan, or other scan pattern for photographing a three-dimensional volume. In this embodiment, the scan pattern is described as a 3D scan for photographing a three-dimensional volume. After completing the adjustment of these photographing parameters, the operator selects the start of photographing (not shown) to perform photographing. Although not described in this embodiment, the subject's eye is tracked to scan the subject's eye while reducing the influence of fixation eye movement.

[0035] <Step S303> In step S303, a tomographic image is generated. The tomographic image generating unit 311 generates a tomographic image by performing a general reconstruction process on each of the interference signals.

[0036] First, the tomographic image generating unit 311 performs fixed pattern noise reduction from the interference signal. Fixed pattern noise reduction is performed by extracting fixed pattern noise by averaging multiple detected A-scan signals and subtracting this from the input interference signal. Next, the tomographic image generating unit 311 performs a desired window function processing to optimize the depth resolution and dynamic range, which are in a trade-off relationship when Fourier transform is performed in a finite section. Next, a tomographic signal is generated by performing FFT processing.

[0037] <Step S304> In step S304, the analysis unit 331 performs layer detection. Boundaries of retinal layers are detected in a plurality of tomographic images captured by the tomographic imaging apparatus 200. The boundaries of retinal layers are detected by a machine learning model such as deep learning, or by image processing.

[0038] For example, when data is input into a machine learning model, data according to the design of the machine learning model is output. Output data highly likely to correspond to the input data is output according to the trend trained from the teacher data. Also, for each type of output data trained from the teacher data, the possibility is output as a numerical value, and so on. Specifically, for example, when an OCT-acquired tomographic image is input into a machine learning model trained by first teacher data, the imaging site label captured in the image is output, or depending on the design, the probability for each imaging site label is output. Also, for example, a machine learning model trained by second teacher data outputs a label image when an OCT tomographic image is input. Note that the format and combination of the input data and output data of the pair group constituting the teacher data are implemented in a combination suitable for the embodiment, such as one being an image and the other being a numerical value, one being composed of a plurality of image groups and the other being a character string, or both being images. Note that when learning is performed by dividing an image into rectangular regions, the retinal layer is detected in each rectangular region. That is, a group of rectangular region images with labels assigned to each pixel is obtained. The analysis unit 331 arranges each of the rectangular region images with labels assigned thereto in the same positional relationship as each of the group of rectangular region images and combines them to obtain a label image in which the retina is detected. Based on the output label image, the boundary between each layer of the retina is detected as a boundary line.

[0039] In this embodiment, the boundaries between the ILM and NFL, the NFL and GCL boundaries, the ISOS, the RPE, and the BM are detected. Although not shown in the figure, as other boundary lines, the boundary between the outer plexiform layer (OPL) and the outer nuclear layer (ONL), the boundary between the inner plexiform layer (IPL) and the inner nuclear layer (INL), the boundary between the INL and the OPL, the boundary between the GCL and the IPL, etc. may be detected. Then, for the detected boundary line, a process of smoothly correcting the shape of the boundary line is executed. For example, regarding the coordinate values of the boundary line shape as time-series data by a signal, the shape of the boundary line may be smoothed by a Savitzky-Golay filter, a simple moving average, a weighted moving average, an exponential moving average, or other smoothing processes.

[0040] <Step S305> In Step S305, the detected boundary line and the tomographic image are displayed on the display unit 600. An example of the screen displayed on the display unit 600 is shown in FIG. 4. 410 represents the entire screen, 401 represents the patient tab, 402 represents the imaging tab, 403 represents the report tab, 404 represents the settings tab, and the diagonal lines in the report tab 403 represent the active state of the report screen. In the present embodiment, an example of displaying the report screen 410 will be described. 411 is a tomographic image, and 415 and 416 are examples of superimposing the boundary lines on the tomographic image. 412 is a graph of the retinal thickness obtained from the boundary lines 415 and 416. 405 is an SLO image, and 406 is a thickness map which is the first analysis result obtained by analyzing using the result of the boundary line obtained by the analysis unit 331. In the present embodiment, an example of the thickness obtained from the boundary lines 415 and 416 is shown, in which a color thickness map assigned a color according to the thickness value is superimposed on the SLO image 405. 407 is a thickness map generated by the generation unit 332 as the second analysis result with the thickness map 406 as an input. Similar to the thickness map 406, the generated thickness map 407 is also a color thickness map assigned a color according to the thickness value. 408 is a color bar showing the relationship between the retinal thickness and the color when a color is assigned according to the thickness value when the thickness map is displayed on the display unit. The generation unit 332 is a model including an image processing system that outputs a thickness map, for example, by a rule-based or machine learning (particularly, deep learning technology). Hereinafter, a machine learning algorithm including an image processing system by deep learning technology will be described with reference to FIGS. 5 to 7. In the present embodiment, the machine learning model is, for example, a trained machine learning model 3325 trained using learning data including a group of thickness maps of normal eyes related to medical images.

[0041] The teacher data is composed of a group of thickness maps of one or more normal eyes. The machine learning model shown in Fig. 5 is an example of an autoencoder. An autoencoder consists of an encoder and a decoder. Here, the encoder converts the input data into low-dimensional data, and the decoder restores the data from the low-dimensional data. When training the autoencoder, it is trained using the group of thickness maps corresponding to normal eyes. When performing inference using the model 3325 trained with the group of thickness maps of normal eyes, in Fig. 5(a), the thickness map 501 of the normal eye is tensorized and input into the machine learning model 3325, and the generation unit 332 statically images the tensor output by the machine learning model 3325 to output a thickness map 502 like that of a normal eye. In Fig. 5(b), the thickness map 511 of the diseased eye is tensorized and input into the machine learning model 3325, and the generation unit 332 statically images the tensor output by the machine learning model 3325 to output a thickness map 512 like that of a normal eye. Here, the thickness map 512 like that of a normal eye can also be said to be a thickness map obtained by assuming that the diseased eye was a normal eye. Note that the tensor in the description of this embodiment is a form in which a group of pixel values of an image (thickness map) is expressed as a multi-dimensional array, which is the data input / output format to the machine learning model 3325, and it is assumed that the image (thickness map) and the tensor can be mutually converted.

[0042] As types of thickness maps, for example, there are thickness maps only of the NFL, thickness maps of all layers from the ILM to the RPE, thickness maps of the GCC (Ganglion Cell Complex) including the NFL, GCL, and IPL, etc. In this embodiment, an example of the thickness map of all layers from the ILM to the RPE will be described.

[0043] Next, an example of learning the thickness map in FIG. 6 is shown. The figure is a diagram for explaining the preprocessing for learning the thickness map. 611 shows the thickness map of the entire layer of the left eye. 602 is the thickness map of the entire layer of the right eye, and the thickness map 612 is an example in which the left and right of the thickness map 602 are inverted. The thickness map 613 is an example of the thickness map of the entire layer that is rotated using the position information of the optic nerve head 604 and the macula 605 in the thickness map 603 so that the positions of the optic nerve head 604 and the macula 605 become horizontal. It is assumed that the positions of the optic nerve head 604 and the macula 605 are detected by the image processing unit 303 when the OCT data used for the training data is taken. During learning, it is desirable to learn by aligning the position information of the characteristic parts to some extent as in the thickness maps 611 to 613 shown in FIG. 6. In particular, for the left and right eyes, left-right inversion is performed so as to align with one of the directions. Although not shown, the thickness maps may be scaled so that the distances between the macula and the papilla are the same for different thickness maps. Further, the macula may be shifted so as to be at the center position of the image. In FIG. 6, the thickness map in the case where the position of the fixation light at the time of shooting is the macula center (or the posterior pole center) is shown, but it is not limited to this, and a thickness map with the position of the fixation light at the time of shooting as the optic nerve head center may be used. In that case, in order to align the position information of the characteristic parts to some extent as in the example shown in FIG. 6, the position information of the optic nerve head detected by the image processing unit 303 may be shifted so as to be aligned with the image center.

[0044] Here, in the example of the thickness map 613, a rotated thickness map in which the positions of the optic nerve head 604 and the macula 605 are horizontal is shown as an example having the same size as before rotation, but this is not limiting. For example, even when the thickness map is rotated or shifted as shown in the thickness map 623, an area where the thickness map can fit may be prepared and rotation may be performed within it. For example, when the size of the thickness map is 256×256, the size of the area is 300×300. In the example of the thickness map 623 in the figure, an example is shown in which it rotates so that the positions of the optic nerve head 604 and the macula 605 are horizontal and the macula 605 is shifted to the center of the area. When the area is expanded as shown in the thickness map 623, learning can be performed without any part of the data being missing even when the thickness map is rotated or shifted. When learning is performed by expanding the area of the thickness map in this way, preprocessing is performed so that the thickness maps 611 and 612 are also expanded in size in the same way as the thickness map 623, and then learning is performed.

[0045] Note that for the accuracy evaluation and error (loss) calculation between the thickness map in the teacher data assigned for training and verification and the thickness map output by the machine learning model 3325, a calculation method based on the following method can be adopted. Specifically, for example, a method of quantifying the error and similarity by methods such as MSE (Mean Squared Error) and SSIM (Structural Similarity) can be adopted. Also, in the learning of the machine learning model 3325, for the accuracy evaluation and error (loss) calculation, a semantic region that is an area in the thickness map included in the learning data and that can be distinguished according to the mode depicted in the thickness map or information related to the thickness map may be considered, and the calculation target may be selected. Specifically, the semantic region is the retinal thickness value. As areas without meaning, there are blank areas that appear when the image is rotated like the thickness map 613 and expanded areas like the thickness map 623. Since no thickness value is depicted in this blank area or expanded area (it does not affect the diagnosis), in the learning of the machine learning model 3325, only the semantic region that affects the diagnosis may be used as the target for accuracy evaluation and error (loss) calculation to adjust the performance and characteristics of the machine learning model 3325.

[0046] Next, an example of inferring the thickness map in FIG. 7 is shown. When the generation unit 332 generates a thickness map using the machine learning model 3325 that has been trained using the group of thickness maps as shown in FIG. 6, preprocessing similar to that during learning is applied to the thickness map for inference. For example, in FIG. 7(a), an example is shown in which the machine learning model 3325 was trained to fit the left eye during learning. In FIG. 7(a), when inputting the right-eye thickness map 706 to the generation unit 332, the right-eye thickness map is horizontally flipped to obtain the thickness map 701 and input to the generation unit 332, and the output thickness map 702 is horizontally flipped to obtain the thickness map 707 for display. Note that when the machine learning model 3325 was trained to fit the right eye during learning, the left-eye thickness map is horizontally flipped to obtain the thickness map 701 and input to the generation unit 332.

[0047] Similarly, in FIG. 7(b), an example is shown in which the inclination was adjusted using the position information of the optic nerve papilla and the macula during learning and learning was performed in the expanded region as shown by the thickness map 623. In FIG. 7(b), when inputting the thickness map 716 to the generation unit 332, a thickness map 711 is generated by rotating the thickness map so that the positions of the optic nerve papilla and the macula are horizontal within the same region size as during learning. Then, it is input to the generation unit 332, and the thickness map 712 output from the generation unit 332 is rotated so that the positions of the optic nerve papilla and the macula return to their original positions, and the expanded region is clipped to return to the original size, and the thickness map 717 is displayed.

[0048] Note that although an example has been described in which the thickness map in the present embodiment is learned or inferred using the thickness value itself for learning and inference, and colors are assigned according to the thickness value when displaying on the display unit 600, the present invention is not limited to this. Learning and inference may be performed using a color thickness value map obtained by converting from the thickness value to a color. In that case, when displaying on the display unit 600, the inferred thickness map may be displayed without assigning a color value according to the thickness value.

[0049] <Step S306> In step S306, an instruction acquisition unit (not shown) acquires an instruction from the outside on whether to end the tomography image capture by the image processing system 100. This instruction is input by an operator using the input unit 700. When an instruction to end the process is acquired, the image processing system 100 ends the process. On the other hand, when continuing the capture without ending the process, the process returns to step S302 to continue the capture. Through the above, the process of the image processing system 100 is performed.

[0050] According to the configuration described above, in this embodiment, a thickness map that looks normal can be inferred from the input thickness map. Since it is a thickness map corresponding to the input, it is possible to display a thickness map that is only normal while maintaining the eye structure of the eye to be examined.

[0051] [Modification Example 1 of the First Embodiment] In this embodiment, an autoencoder was described as an example of the machine learning model, but it is not limited to this. For example, a variational autoencoder, a convolutional autoencoder, etc. may also be used. Furthermore, generative adversarial networks, and its derivatives such as AnoGAN, Efficient-GAN, GANomaly, etc. may be used.

[0052] [Embodiment 2] In the first embodiment, an example of displaying a thickness map inferred from the thickness map of the input image was shown. In this embodiment, difference information between the thickness map of the input image and the inferred thickness map is generated. Furthermore, difference information between the thickness map of the input image and the statistical value of the thickness of a normal eye (normal eye database) is generated, and the purpose is to display this difference information to the operator.

[0053] FIG. 8 is a diagram showing an example of a schematic configuration of an image processing system 1000 including an image processing apparatus 3103 according to a second embodiment. In FIG. 8, components having the same configuration as those shown in FIG. 1 are denoted by the same reference numerals, and detailed descriptions thereof are omitted. Here, in the image processing system 1000 of FIG. 8, an image processing unit 3103 different from that of the first embodiment will be described. The image processing unit 3103 includes a first difference calculation unit 333 and a second difference calculation unit 334. The first difference calculation unit 333 calculates difference information (hereinafter also referred to as a difference thickness map) between the thickness map of the retinal layer, which is the first analysis result, and the thickness map generated by the generation unit 332, which is the second analysis result. The second difference calculation unit 334 calculates difference information between the thickness map of the retinal layer, which is the first analysis result, and the statistical value of the thickness of the retinal layer for a plurality of eyes stored in the storage unit 302. Then, at least two or more of the thickness map of the retinal layer, which is the first analysis result, the thickness map of the retinal layer, which is the second analysis result, the difference information generated by the first difference calculation unit 333, and the difference information generated by the second difference calculation unit 334 are displayed on the display unit 600.

[0054] Regarding this, an explanation will be given with reference to FIG. 9. FIG. 9 shows a report screen 410 to be displayed on the display unit 600 in the present embodiment, and is an example of being displayed in a layout different from that of FIG. 4. In FIG. 9, an example is shown in which the thickness map 406 of the retinal layer, the thickness map 407 generated by the generation unit 332, the differential thickness map 927 generated by the first difference calculation unit 333, and the differential thickness map 926 generated by the second difference calculation unit 334 are displayed together with the tomographic image 411 and the tomographic image 912 of the vertical plane. Here, the thickness map and the differential thickness map will be described with reference to FIG. 10. In FIG. 10(a), the thickness map 406 is the first analysis result generated from the retinal layer analyzed by the analysis unit, and shows an example in which the running of the nerve fiber 1001 is typical and the thickness is normal, and a bulge can be seen around the macula 1002 due to a disease. The thickness map 407 is a thickness map generated by the generation unit 332 with the thickness map 406 as an input, and is an example in which the thickness around the macula becomes a normal thickness and is output. The differential thickness map 927 is differential information generated by the first difference calculation unit 333, and is a differential thickness map between the thickness maps 406 and 407. The differential thickness map 926 is differential information generated by the second difference calculation unit 334, and is a differential thickness map between the thickness map 406 and the statistical value of the thickness of the retinal layer related to a plurality of eyes possessed by the storage unit 302. Note that the differences calculated by the first difference calculation unit 333 and the second difference calculation unit 334 represent the differences between different thickness maps, such as differences and ratios in a plurality of thickness maps. As shown in FIG. 10(a), in the case of a typical eye, as differential information, the differences 1003 and 1008 due to the bulge information around the macula are respectively displayed.

[0055] In FIG. 10(b), the thickness map 1406 is the first analysis result generated from the retinal layers analyzed by the analysis unit. The course of the nerve fibers 1011 is not typical, but the thickness is normal, and an example is shown where a bulge can be seen around the macula 1012 due to a disease. The thickness map 1407 is a thickness map generated by the generation unit 332 using the thickness map 1406 as an input, and is an example where the thickness around the macula becomes a normal thickness and is output. The differential thickness map 1927 is the differential information generated by the first differential calculation unit 333, and is the differential thickness map between the thickness maps 1406 and 1407. The differential thickness map 1926 is the differential information generated by the second differential calculation unit 334, and is the differential thickness map between the thickness map 1406 and the statistical values of the thicknesses of the retinal layers related to a plurality of eyes held in the storage unit 302. As shown in the differential thickness map 1926, an example is shown where the differential information 1014 is displayed at locations that deviate from the statistical values even though the thickness is within the normal range. As described here, since the generation unit 332 generates a thickness map based on the input thickness map, it is less affected by the shape difference, thickness difference, and the influence of the eye axis length difference for each individual, and generates a thickness map based on the learned information.

[0056] In the present embodiment, the thickness map and the differential information as shown in FIG. 10 are displayed on the report screen 410. Further, on the report screen 410, grids 916 and 917 are superimposed and displayed on the thickness maps 406 and 407 of the retinal layers, respectively, and the numerical values within each grid are displayed on the grids 9160 and 9170. Although not shown in the figure, the numerical values within the grid may be displayed not only in the grid but also in a table format or a graph format.

[0057] Note that in the report screen 410 of the present embodiment, an example is shown where the thickness map and the differential thickness map described above are arranged and displayed, but it is not limited to this. For example, either the thickness map generated by the analysis unit 331 or the thickness map generated by the generation unit 332 can be switched and displayed, and the differential thickness map can also be switched and displayed accordingly.

[0058] According to the configuration described above, in this embodiment, the operator can make an integrated judgment by referring to any one of the thickness map obtained by the analysis unit analyzing the retinal layer, the thickness map generated by the generation unit, the first difference calculation unit, and the difference information generated by the second difference calculation unit.

[0059] [Embodiment 3] In the first and second embodiments, examples of generating a thickness map and generating difference information of the thickness map were described. In this embodiment, a condition acquisition unit for acquiring conditions at the time of shooting and conditions of the thickness map will be described, and an example of generating a thickness map based on those conditions will be described.

[0060] FIG. 11 is a diagram showing an example of a schematic configuration of an image processing system 2000 including an image processing apparatus 3203 according to the third embodiment. In FIG. 11, the same components as those shown in FIG. 1 and FIG. 8 are denoted by the same reference numerals, and detailed descriptions thereof are omitted. Here, in the image processing system 2000 of FIG. 11, an image processing unit 3203 different from the first and second embodiments will be described. The image processing unit 3203 includes a condition acquisition unit 335. The generation unit 3320 generates a thickness map based on the conditions acquired by the condition acquisition unit 335. Regarding this, a machine learning algorithm including an image processing system based on deep learning technology will be described with reference to FIG. 12. FIG. 12 is a diagram for explaining an image generation model 33250 included in the generation unit 3320 according to the third embodiment.

[0061] Specifically, the machine learning model 33250 in FIG. 12 inputs into the machine learning model 33250 a tensorized thickness map 1201 and a scalar value indicating the condition 335 given to the tensor of the thickness map. Then, the generation unit 3320 statically images the tensor output by the machine learning model 33250 and outputs it as a thickness map 1202.

[0062] For example, examples of conditions acquired by the condition acquisition unit 335 include imaging conditions, conditions of the eye to be examined, analysis conditions, and the like. Examples of imaging conditions include information regarding the imaging date and time, the name of the imaging site, the imaging region, the imaging angle of view, the imaging method, the resolution and gradation of the image, the pixel size of the image, the image filter, the data format of the image, and the like. Information on the eye to be examined includes age (date of birth), gender, race, and the like. The analysis condition is, for example, information regarding the retinal layer included in the thickness map. Information regarding the retinal layer is information on the layer included in the thickness map generated from the tomographic image (type of thickness map). Examples of the types of thickness maps include a thickness map having the thickness of each layer, a thickness map having the thickness of all layers from the ILM to the RPE, a thickness map of the GCC including the NFL, GCL, and IPL, and the like.

[0063] In Fig. 12(a), an example is shown where, together with the thickness map, a condition that it is a thickness map of all layers is input as an analysis condition, and a thickness map corresponding thereto is output as an output. In Fig. 12(b), an example is shown where, together with the thickness map, a condition that it is a thickness map of the GCC is input as an analysis condition, and a thickness map corresponding thereto is output as an output. For example, the scalar value of the analysis condition is set to 001001 for the thickness map of all layers and 001002 for the thickness map of the GCC. As other conditions in the case of Fig. 12, the macula center may be input as the imaging site, 10×10 mm may be input as the imaging region, and age, gender, race, measurement values related to the eye (axial length, refractive value, etc.) may also be input as information on the eye to be examined. Note that the condition to be given to the thickness map does not necessarily have to be one, and a plurality of conditions may be given simultaneously. Alternatively, a plurality of conditions expressed by one scalar value may also be used.

[0064] According to the configuration described above, in the present embodiment, by learning the condition together with the thickness map when learning the thickness map, various types of thickness maps can be learned simultaneously. Then, at the time of inference, by inputting the condition together with the thickness map, a thickness map based on the input condition can be output at the time of output.

[0065] [Embodiment 4] In this embodiment, an example of displaying a time-series thickness map will be described. The time-series thickness map is a thickness map generated using each of a plurality of tomographic images taken at different times. Regarding those having the same functions as the first to third embodiments described above, the description will be omitted here.

[0066] FIG. 13 is a display example of displaying a time-series thickness map in this embodiment. In FIG. 13, the time-series thickness maps are arranged and displayed horizontally. Below each thickness map, the differential thickness maps are arranged and displayed vertically. 1301 is a thickness map (Thickness Map) generated by the analysis unit. 1302 is a differential thickness map (Deviation) between the thickness map 1301 of the retinal layer generated by the second difference calculation unit 334 and the statistical value of the thickness of the retinal layer stored in the storage unit 302. 1303 is a differential thickness map (Estimate Deviation) between the thickness map of the retinal layer which is the first analysis result generated by the first difference calculation unit 333 and the thickness map generated by the generation unit 332 which is the second analysis result.

[0067] As shown in this embodiment, a plurality of thickness maps and a plurality of differential thickness maps can be displayed. Note that the differential thickness map 1303 generated by the first difference calculation unit 333 may display the difference from a reference map which is a thickness map (second analysis result) inferred from the thickness map (first analysis result) obtained at a certain reference time. Alternatively, the difference between the thickness map (each first analysis result) obtained at each time and the thickness map (each second analysis result) inferred therefrom may be displayed. Alternatively, the image processing unit 3203 has an integration unit (not shown), and using a plurality of thickness maps (second analysis results) inferred from the thickness maps (first analysis results) obtained at each time, a statistical thickness map (for example, an average map obtained by averaging each second analysis result) obtained by integrating each second analysis result is generated, and the difference from the statistical thickness map may be displayed. Note that the statistical thickness map obtained by integrating each second analysis result is an example of the third analysis result.

[0068] Furthermore, although not shown in the drawings, a thickness map generated by the generation unit 332 at each time may be displayed. These thickness maps may be displayed side by side or may be switched and displayed. Furthermore, not limited to time-series data, for the thickness maps of the left and right eyes, a thickness map may be generated from each thickness map, and furthermore, a differential thickness map may be generated.

[0069] According to the configuration described above, in the present embodiment, a normal-like thickness map can be inferred from a plurality of input thickness maps such as time series and left and right eyes. And the difference from the statistical value of the thickness of a normal eye and the difference from the inferred thickness map can be displayed simultaneously.

[0070] (Modification Example 1) In the machine learning models of the various embodiments described above, an example of learning only one type of thickness map and an example of corresponding to multiple types by learning multiple types of thickness maps together with conditions (imaging conditions, conditions of the eye to be examined, analysis conditions, etc.) have been described, but it is not limited to this. For example, a machine learning model that learns one type or several types of thickness maps by condition during learning may be learned, and a configuration may be adopted in which the machine learning model used is switched according to the conditions. For example, as an example of switching the machine learning model learned by condition, conditions of the eye to be examined such as race, age, and gender are learned together for each type of thickness map such as each layer, the thickness map of all layers, and the thickness map of GCC. Then, at the time of inference, the machine learning model may be selected according to the analysis conditions (the type of thickness map used for input), and inference may be performed together with the thickness map and the conditions of the eye to be examined. Not limited to this example, the types of thickness maps may be collectively learned for the conditions of the eye to be examined (race, gender, age, etc.) to form a machine learning model, and the machine learning model may be selected according to the conditions of the eye to be examined at the time of inference. Regarding age, instead of dividing by one year at a time, it may be grouped with a certain width such as the 30s to 40s and the 50s to 60s.

[0071] (Modification Example 2) In the machine learning models of the various embodiments described above, an example of learning using learning data including a thickness map group of normal eyes has been described, but it is not limited to this. For example, learning may be performed using learning data including a thickness map group of diseased eyes. In that case, in this machine learning model, when a normal thickness map is input, a thickness map such as that of a diseased eye is output. And when a thickness map of a diseased eye is input, a thickness map of a diseased eye is output. Therefore, the more normal the eye is, the more the difference is calculated when generating difference information from the output thickness map. Furthermore, the machine learning model may be learned for each type of disease. For example, the types of diseases include glaucoma, diabetic retinopathy, age-related macular degeneration, etc. Thereby, by performing inference with a machine learning model corresponding to each disease, it is also possible to predict a disease such that the possibility is high that the result with the least difference in the thickness map between the input and the output is that disease.

[0072] (Other Embodiments) Also, the disclosed technology can also be realized by executing the following processing. That is, the disclosed technology supplies software (program) that realizes one or more functions of the various embodiments described above to a system or device via a network or a storage medium, and a computer (or CPU, MPU, etc.) of the system or device reads and executes the program. A computer may have one or more processors or circuits and may include a network of separate computers or separate processors or circuits for reading and executing computer-executable instructions. At this time, the processor or circuit may include a central processing unit (CPU), a microprocessing unit (MPU), a graphics processing unit (GPU), an application-specific integrated circuit (ASIC), or a field-programmable gateway (FPGA). Also, the processor or circuit may include a digital signal processor (DSP), a data flow processor (DFP), or a neural processing unit (NPU).

[0073] (Configuration 1) Using the tomographic image of the eye to be examined, a first analysis result having two-dimensional thickness data of the retinal layer of the eye to be examined is generated, and using a machine learning model trained using a plurality of thickness maps and the first analysis result, a generation unit that generates a second analysis result different from the first analysis result; A display control unit that controls a display unit to display information regarding the second analysis result; An information processing apparatus comprising:

[0074] (Configuration 2) The information processing apparatus according to Configuration 1, wherein the machine learning model is a machine learning model trained using the plurality of thickness maps corresponding to a plurality of normal eyes.

[0075] (Configuration 3) The information processing apparatus according to any one of Configuration 1 or 2, wherein the display control unit controls the display unit to display, as information regarding the second analysis result, a comparison result between the first analysis result and the second analysis result.

[0076] (Configuration 4) The display control unit The comparison result between the first analysis result and the second analysis result, and The information processing apparatus according to any one of Configuration 1 to 3, which controls the display unit to display at least one of a comparison result between the first analysis result and a thickness map acquired from a normal eye database.

[0077] (Configuration 5) The generation unit generates a second analysis result different from the first analysis result using the machine learning model, the first analysis result, and information regarding the eye to be examined. The information processing apparatus according to any one of Configuration 1 to 4.

[0078] (Configuration 6) The information regarding the eye to be examined is at least one of the race of the subject, the age of the subject, the gender of the subject, and the axial length of the eye to be examined. The information processing apparatus according to Configuration 5.

[0079] (Configuration 7) The generation unit generates a second analysis result different from the first analysis result by using the machine learning model, the first analysis result, and information regarding the retinal layer included in the first analysis result. The information processing apparatus according to any one of Configurations 1 to 6.

[0080] (Configuration 8) The information regarding the retinal layer is information on the layer included in the first analysis result. The information processing apparatus according to Configuration 7.

[0081] (Configuration 9) The display control unit controls the display unit to display the first analysis result and the second analysis result side by side. The information processing apparatus according to any one of Configurations 1 to 8.

[0082] (Configuration 10) The display control unit controls the display unit to display the comparison result between the first analysis result and the second analysis result, the comparison result between the first analysis result and the thickness map obtained from the normal eye database, the first analysis result, and the second analysis result side by side. The information processing apparatus according to any one of Configurations 1 to 9.

[0083] (Configuration 11) The display control unit a plurality of first analysis results generated using each of a plurality of tomographic images obtained by photographing the eye to be examined at different times, each of the comparison results between each of the plurality of first analysis results and the thickness map obtained from the normal eye database, each of the comparison results between each of the plurality of first analysis results and the second analysis result generated using the machine learning model and at least one of the plurality of first analysis results, and controls the display unit to display them side by side. The information processing apparatus according to any one of Configurations 1 to 10.

[0084] (Configuration 12) The display control unit A plurality of first analysis results generated using each of a plurality of tomographic images obtained by photographing a subject eye at different times, Each of the comparison results between each of the plurality of first analysis results and a thickness map obtained from a normal eye database, Each of the comparison results between each of the plurality of first analysis results and a third analysis result obtained by averaging each of the plurality of second analysis results generated using the machine learning model and each of the plurality of first analysis results, The information processing apparatus according to any one of Configurations 1 to 11, which controls a display unit to display them side by side.

[0085] (Configuration 13) The generation unit has a machine learning model that takes as input a first analysis result of one of the right eye and the left eye of the subject eye, The generation unit, When the first analysis result of the one is input, the second analysis result is generated by inputting the first analysis result of the one into the machine learning model, The information processing apparatus according to any one of Configurations 1 to 12, wherein when the first analysis result of the other of the right eye and the left eye of the subject eye is input, the second analysis result is generated by inverting the first analysis result of the other horizontally and inputting it into the machine learning model.

[0086] (Configuration 14) An analysis unit that generates a first analysis result having two-dimensional thickness data of the retinal layer of the subject eye using a tomographic image of the subject eye, A display control unit that controls a display unit to display information regarding a second analysis result different from the first analysis result generated using a machine learning model learned using a plurality of thickness maps and the first analysis result, An information processing apparatus comprising:

[0087] (Configuration 15) An OCT device that photographs a tomographic image of a subject eye, An information processing system comprising the OCT device and an image generation device according to any one of Configurations 1 to 14, which is communicably connected to the OCT device.

[0088] (Method 1) A generation step of generating a first analysis result having two-dimensional thickness data of the retinal layer of the eye to be examined using a tomographic image of the eye to be examined, and generating a second analysis result different from the first analysis result using a machine learning model learned using a plurality of thickness maps and the first analysis result; A display control step of controlling a display unit to display information regarding the second analysis result; An information processing method comprising the above.

[0089] (Program 1) A program for causing a computer to execute the information processing method described in Method 1.

Explanation of Signs

[0090] 100 Image processing system 200 Tomographic imaging device (OCT device) 300 Image processing device 400 Fundus imaging device 500 External storage unit 600 Display unit 700 Input unit

Claims

1. A generating unit that generates a first analysis result having two-dimensional thickness data of a retinal layer of the eye to be examined using a tomographic image of the eye to be examined, and generates a second analysis result different from the first analysis result using a machine learning model learned using a plurality of thickness maps and the first analysis result; A display control unit that controls a display unit to display information regarding the second analysis result; An information processing apparatus comprising:

2. The information processing apparatus according to claim 1, wherein the machine learning model is a machine learning model learned using the plurality of thickness maps corresponding to a plurality of normal eyes.

3. The information processing apparatus according to claim 1, wherein the display control unit controls the display unit to display, as information regarding the second analysis result, a comparison result between the first analysis result and the second analysis result.

4. The display control unit controls the display unit to display at least one of a comparison result between the first analysis result and the second analysis result and a comparison result between the first analysis result and a thickness map obtained from a normal eye database.

5. The information processing apparatus according to claim 1, wherein the generating unit generates a second analysis result different from the first analysis result using the machine learning model, the first analysis result, and information regarding the eye to be examined.

6. The information processing apparatus according to claim 5, wherein the information regarding the eye to be examined is at least one of a race of the subject, an age of the subject, a gender of the subject, and an axial length of the eye to be examined.

7. The information processing apparatus according to claim 1, wherein the generating unit generates a second analysis result different from the first analysis result using the machine learning model, the first analysis result, and information regarding a retinal layer included in the first analysis result.

8. The information processing apparatus according to claim 7, wherein the information regarding the retinal layer is information of a layer included in the first analysis result.

9. The information processing apparatus according to claim 1, wherein the display control unit controls the display unit to display the first analysis result and the second analysis result side by side.

10. The information processing apparatus according to claim 1, wherein the display control unit controls the display unit to display side by side a comparison result between the first analysis result and the second analysis result, a comparison result between the first analysis result and a thickness map obtained from a normal eye database, the first analysis result, and the second analysis result.

11. The display control unit uses each of a plurality of first analysis results generated using each of a plurality of tomographic images obtained by photographing the eye to be examined at different times, each of the comparison results between each of the plurality of first analysis results and a thickness map acquired from a normal eye database, each of the comparison results between each of the plurality of first analysis results and a second analysis result generated using the machine learning model and at least one of the plurality of first analysis results, The information processing apparatus according to claim 1, which controls a display unit to display them side by side.

12. The display control unit uses each of a plurality of first analysis results generated using each of a plurality of tomographic images obtained by photographing the eye to be examined at different times, each of the comparison results between each of the plurality of first analysis results and a thickness map acquired from a normal eye database, each of the comparison results between each of the plurality of first analysis results and a third analysis result obtained by averaging each of the plurality of second analysis results generated using the machine learning model and the plurality of first analysis results, The information processing apparatus according to claim 1, which controls a display unit to display them side by side.

13. The generation unit has a machine learning model that takes as input a first analysis result of one of the right eye and the left eye of the eye to be examined, The generation unit when the first analysis result of the one is input, generates the second analysis result by inputting the first analysis result of the one into the machine learning model, The information processing apparatus according to claim 1, wherein when the first analysis result of the other of the right eye and the left eye of the eye to be examined is input, the second analysis result is generated by horizontally flipping the first analysis result of the other and inputting it into the machine learning model.

14. An analysis unit that generates a first analysis result having two-dimensional thickness data of the retinal layer of the eye to be examined using a tomographic image of the eye to be examined, A display control unit that controls a display unit to display information regarding a second analysis result different from the first analysis result generated using a machine learning model learned using a plurality of thickness maps and the first analysis result, An information processing apparatus comprising:

15. An OCT device that photographs a tomographic image of the eye to be examined, An information processing system comprising the OCT device and the image generation device according to any one of claims 1 to 14, which is communicably connected to the OCT device.

16. Using the tomographic image of the eye to be examined, generate a first analysis result having two-dimensional thickness data of the retinal layer of the eye to be examined, and use a machine learning model learned using a plurality of thickness maps and the first analysis result to generate a second analysis result different from the first analysis result; a generation step; a display control step of controlling a display unit to display information regarding the second analysis result; An information processing method comprising the above.

17. A program for causing a computer to execute the information processing method according to claim 16.

Images