Information processing device and program

Abstract

The information processing device of the embodiment is provided with: a learning section that carries out learning of a machine learning model that inputs one or more non-polarized OCT images, which do not have polarization information and outputs pseudo-polarized OCT images that correspond to polarized OCT images, which have polarization information; and a storage unit for storing the learning results of the learning section.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus and a program.

Background Art

[0002] An optical coherence tomography (OCT) device that performs optical coherence tomography (hereinafter, for convenience of explanation, referred to as an OCT device) is known. In an OCT device, an OCT image (hereinafter, for convenience of explanation, referred to as an OCT image) is acquired.

[0003] As OCT devices, a polarization-sensitive (PS) OCT device (hereinafter, for convenience of explanation, referred to as a polarization OCT device) and a non-polarization-sensitive (non-PS) OCT device (hereinafter, for convenience of explanation, referred to as a non-polarization OCT device) are known.

[0004] A non-polarization OCT device can measure an OCT image that does not have polarization information (hereinafter, for convenience of explanation, referred to as a non-polarization OCT image), but cannot measure an OCT image that has polarization information (hereinafter, for convenience of explanation, referred to as a polarization OCT image). On the other hand, a polarization OCT device can measure a polarization OCT image and can also measure a non-polarization OCT image.

[0005] The non-polarization OCT device and the polarization OCT device are used, for example, in fields targeting medical and biological subjects. Furthermore, the polarization OCT device is used as a non-invasive three-dimensional tomography technology targeting medical and biological subjects. In a polarization OCT image obtained by a polarization OCT device, the polarization characteristics of a biological tissue can be visualized.

[0006] For example, polarization uniformity (DOPU: Degree of Polarization Uniformity) is a parameter that quantifies the local variation in the polarization state (polarization scambling) caused by a sample. DOPU is used for analyzing retinal images and is known to enhance the pathology of the retinal pigment epithelium (RPE) using polarized OCT images.

[0007] Furthermore, various studies have been conducted on OCT (see, for example, Non-Patent Documents 1-13). [Prior art documents] [Non-patent literature]

[0008] [Non-Patent Document 1] Y. Jia, O. Tan, J. Tokayer, B. Potsaid, Y. Wang, JJ Liu, MF Kraus, H. Subhash, JG Fujimoto, j. Hornegger, D. Huang, “Split-spectrum amplitude-decorrelation angiography with optical coherence tomography”, Opt. Express Vol. 20, 4710-4725 (2012). [Non-Patent Document 2] KAVermeer, J.Mo, JJAWeda, HGLemij, JFde Boer, “Depth-resolved model-based reconstruction of attenuation coefficients in optical coherence tomography”, Biomed.Opt.Express Vol.5, 322-337 (2014). [Non-Patent Document 3] MJJu, Y.-J. Hong, S. Makita, Y. Lim, K. Kurokawa, L. Duan, M. Miura, S. Tang, and Y. Yasuno, “Advanced multi-contrast Jones matrix optical coherence tomography for Doppler and polarization sensitive imaging,” Opt.Express Vol.21, 19412-19436(2013).

Outdoor Tools 4

Direct Environment 5

Outdoor Configuration6

Direct Environment 7

Outdoor Track 8

Outdoor Tools9

Outdoor Tools 10

Outdoor Content11

Outdoor Tools 12

Outdoor Content13

[0009] However, conventional non-polarized OCT devices could not acquire DOPU images, and acquiring DOPU images required the hardware of a polarized OCT device, which is more expensive and complex than conventional non-polarized OCT devices. Furthermore, while DOPU images were used as an example above, other types of polarized OCT images also exist. Acquiring these types of polarized OCT images also presents the challenge of requiring expensive and complex polarized OCT equipment hardware. Currently, non-polarized OCT devices are being used in many medical facilities, and the adoption of polarized OCT devices has not been sufficient.

[0010] This invention has been made in consideration of these circumstances, and aims to provide an information processing device and program that can acquire a pseudo-polarized OCT image from a non-polarized OCT image. [Means for solving the problem]

[0011] As an example configuration, the system includes a learning unit that takes one or more unpolarized OCT images, which are OCT images without polarization information, as input, and outputs a pseudo-polarized OCT image corresponding to a polarized OCT image, which is an OCT image with polarization information, as output; and a storage unit that stores the learning results of the learning unit, wherein the unpolarized OCT image and the polarized OCT image that serves as the training image for the learning are images acquired by a single polarized OCT device, and when making a judgment using the learning results, an unpolarized OCT image acquired by a non-polarized OCT device different from the polarized OCT device is taken as input. As input, OCT images and OCTA images The input is a combination of multi-channel images. It is an information processing device.

[0012] As a configuration example, one or more non-polarized OCT images that are OCT images without polarization information are input, and based on the learning result of a machine learning model, by the machine learning model, according to the non-polarized OCT image that is the input image, a determination unit is provided that determines a pseudo-polarized OCT image corresponding to a polarized OCT image that is an OCT image with polarization information. The non-polarized OCT image is an image acquired by a non-polarized OCT device, and the learning result is obtained by learning in which a non-polarized OCT image acquired by a polarized OCT device different from the non-polarized OCT device is input and the polarized OCT image acquired by the polarized OCT device is used as a teacher image. The input is a combination of an OCT image and an OCTA image as a multi-channel image. It is an information processing device. As an example configuration, the system includes a learning unit that takes one or more unpolarized OCT images, which are OCT images without polarization information, as input, and outputs a pseudo-polarized OCT image corresponding to a polarized OCT image, which is an OCT image with polarization information, as output, and a storage unit that stores the learning results of the learning unit, wherein the unpolarized OCT image and the polarized OCT image that serves as the training image for the learning are images acquired by a single polarized OCT device, and further, at the time of determination, one or more unpolarized OCT images, which are OCT images without polarization information, are taken as input, and the machine learning model performs the input based on the learning results of the machine learning model, The information processing device includes a determination unit that determines a pseudo-polarized OCT image corresponding to a polarized OCT image, which is an OCT image containing polarization information, based on a non-polarized OCT image, which is an image. The non-polarized OCT image is an image acquired by a non-polarized OCT device different from the polarized OCT device, and the input is a combination of an OCT image and an OCTA image as a multi-channel image.

[0013] As a configuration example, a program causes a computer to perform learning of a machine learning model that takes one or more non-polarized OCT images that are OCT images without polarization information as input and outputs a pseudo-polarized OCT image corresponding to a polarized OCT image that is an OCT image with polarization information, and stores the learning result in a storage unit. The non-polarized OCT image and the polarized OCT image that is the teacher image of the learning are images acquired by one polarized OCT device. When making a determination using the learning result, a non-polarized OCT image acquired by a non-polarized OCT device different from the polarized OCT device is input. As the input, OCT image and OC The input is a combination of the TA image and a multi-channel image. It is a program.

[0014] As a configuration example, a computer is caused to acquire the learning result of a machine learning model, input one or more non-polarized OCT images that are OCT images without polarization information, and based on the acquired learning result, the machine learning model is caused to determine a pseudo-polarized OCT image corresponding to a polarized OCT image that is an OCT image with polarization information according to the non-polarized OCT image that is the input image. The non-polarized OCT image is an image acquired by a non-polarized OCT device, and the learning result is obtained by learning in which a non-polarized OCT image acquired by a polarized OCT device different from the non-polarized OCT device is input, and the polarized OCT image acquired by the polarized OCT device is used as a teacher image. The input is a combination of an OCT image and an OCTA image as a multi-channel image. It is a program.

Effect of the Invention

[0015] According to the information processing apparatus and the program according to the present invention, a pseudo-polarized OCT image can be acquired from a non-polarized OCT image.

Brief Description of the Drawings

[0016] [Figure 1] It is a figure which shows an example of a schematic functional block of the information processing apparatus which concerns on embodiment. [Figure 2] It is a figure which shows an example of the neural network which concerns on embodiment. [Figure 3] It is a figure which shows an example of the procedure of the process at the time of learning performed in the information processing apparatus which concerns on embodiment. [Figure 4] It is a figure which shows an example of the procedure of the process at the time of determination performed in the information processing apparatus which concerns on embodiment. [Figure 5A] It is a figure which shows an example of the non-polarized OCT image which consists of the intensity signal regarding a normal eye. [Figure 5B] It is a figure which shows an example of the DOPU image regarding a normal eye. [Figure 5C] It is a figure which shows an example of the pDOPU image regarding a normal eye. [Figure 6A] It is a figure which shows an example of the non-polarized OCT image which consists of the intensity signal regarding a pathological eye. [Figure 6B] This figure shows an example of a DOPU image related to a pathological eye. [Figure 6C] This figure shows an example of a pDOPU image related to a pathological eye. [Figure 7A] This figure shows an example of a non-polarized OCT image consisting of intensity signals related to a pathological eye. [Figure 7B] This figure shows an example of a DOPU image related to a pathological eye. [Figure 7C] This figure shows an example of a pDOPU image related to a pathological eye. [Figure 8A] This figure shows an example of a non-polarized OCT image consisting of intensity signals related to a pathological eye. [Figure 8B] This figure shows an example of a DOPU image related to a pathological eye. [Figure 8C] This figure shows an example of a pDOPU image related to a pathological eye. [Figure 9A] This figure shows an example of a non-polarized OCT image consisting of intensity signals related to a pathological eye. [Figure 9B] This figure shows an example of a DOPU image related to a pathological eye. [Figure 9C] This figure shows an example of a pDOPU image related to a pathological eye. [Figure 10] This figure shows a table representing the number of clinical features analyzed for DOPU images and pDOPU images. [Figure 11A] This figure shows an example of an OCT image. [Figure 11B] This figure shows an example of an OCTA image. [Figure 11C] This figure shows an example of a true DOPU image. [Figure 11D] This figure shows an example of a pseudo-DOPU image generated using the Model 1 method. [Figure 11E] This figure shows an example of a pseudo-DOPU image generated using the Model 2 method. [Figure 12A] This figure shows an example of an OCT image. [Figure 12B] This figure shows an example of an OCTA image. [Figure 12C] This figure shows an example of a true DOPU image. [Figure 12D] This figure shows an example of a pseudo-DOPU image generated using the Model 1 method. [Figure 12E] This figure shows an example of a pseudo-DOPU image generated using the Model 2 method. [Figure 13A] This figure shows an example of an OCT image. [Figure 13B] This figure shows an example of an OCTA image. [Figure 13C] This figure shows an example of a true DOPU image. [Figure 13D] This figure shows an example of a pseudo-DOPU image generated using the Model 1 method. [Figure 13E] This figure shows an example of a pseudo-DOPU image generated using the Model 2 method. [Modes for carrying out the invention]

[0017] Embodiments of the present invention will be described below with reference to the drawings.

[0018] [Information Processing Device] Figure 1 shows an example of a schematic functional block of the information processing device 1 according to this embodiment. In this embodiment, the information processing device 1 is a computer. In this embodiment, the information processing device 1 has a function to perform machine learning and a function to make decisions based on the results of the machine learning.

[0019] The information processing device 1 may be configured as a single device, as in this embodiment, or it may be configured as a group of separate devices. If the information processing device 1 is configured as multiple separate devices, two or more of these devices may communicate with each other directly or via a network. This communication may be wired or wireless. Furthermore, if the information processing device 1 is configured as multiple separate devices, for example, a device for performing machine learning and a device for making decisions based on the results of the machine learning may be provided as separate devices.

[0020] The information processing device 1 shown in Figure 1 will be described below. The information processing device 1 comprises an input unit 11, an output unit 12, a storage unit 13, and a control unit 14. The control unit 14 comprises a learning unit 31 and a determination unit 32.

[0021] The input unit 11 is used to input information. For example, the input unit 11 may include an operation unit such as a keyboard and a mouse. In this case, the input unit 11 receives information corresponding to the operation performed by the user on the operation unit. As another example, the input unit 11 may receive information from an external device. This external device may be, for example, a portable storage medium.

[0022] The output unit 12 outputs information. For example, the output unit 12 may have a display with a screen. In this case, the output unit 12 displays and outputs information to the screen. As another example, the output unit 12 may output information to an external device. This external device may be, for example, a portable storage medium.

[0023] The memory unit 13 stores information. The control unit 14 performs various processes and controls. In this embodiment, the control unit 14 is equipped with a processor such as a CPU (Central Processing Unit), and performs various processes and controls by executing a control program (program) using this processor. This control program is stored, for example, in the storage unit 13. The learning unit 31 performs machine learning using a predetermined machine learning model. The results of the machine learning are stored in the storage unit 13. The determination unit 32 reads the results of machine learning stored in the memory unit 13. Then, the determination unit 32 makes a determination based on the machine learning model, using the machine learning model to read the machine learning results.

[0024] [Machine learning models] Any machine learning model may be used. This embodiment describes a case where a neural network is used as the machine learning model, and more specifically, a case where a U-Net neural network is used.

[0025] Figure 2 shows an example of a neural network according to the embodiment. In this embodiment, the information processing device 1 performs machine learning using the neural network shown in Figure 2, and makes decisions based on the results of the machine learning. The neural network shown in Figure 2 is a U-Net convolutional neural network (CNN), which has a U-shape. Furthermore, the neural network used is not limited to U-Net; other neural networks may also be used. Examples of other neural networks include ResNet (Residual Network).

[0026] The overview of the neural network shown in Figure 2 will be explained. In Figure 2, the rectangular frames represent images or feature maps, and the arrows represent various processing steps. Five types of arrows are used in Figure 2, each representing a different processing step. In Figure 2, the numbers below the square frame represent the number of channels.

[0027] In the example in Figure 2, the rectangular frames used for explanation are labeled with symbols (T1-T3, T11-T12, T21-T22, T31-T32, T41-T42, T51-T52, T111, T121, T131, T141, T211-T213), while the symbols are omitted for the other rectangular frames.

[0028] This section explains the transition in depth from the first layer to the sixth layer. In the first layer, the input image T1 is subjected to convolution and other processing to generate feature map T2, and feature map T3 is generated by performing similar convolution and other processing on feature map T2.

[0029] Here, these convolution and other operations may include, for example, a convolution with a kernel size of 3x3, batch normalization, and the operation of the activation function LeakyReLU, but are not limited to these. For example, any size may be used as the kernel size for a convolution. Furthermore, other functions may be used as the activation function.

[0030] The feature map T3 is subjected to pooling (also known as downsampling) to generate the feature map T11 of the second layer.

[0031] Here, the pooling process could be, for example, max-pooling with a kernel size of 2x2, but it is not limited to this. For example, any size may be used as the kernel size for pooling.

[0032] In the second layer, just like in the first layer, the feature map T11 is subjected to two convolution operations, followed by pooling, which generates the feature map T21 for the third layer.

[0033] The same applies to the transition from the third layer to the fourth layer, and also to the transition to layers of any greater depth. In the example in Figure 2, the feature maps T31 for the 4th layer, T41 for the 5th layer, and T51 for the 6th layer are also shown.

[0034] In the example shown in Figure 2, a U-Net with a depth of 1 to 6 layers is displayed, but this depth (number of layers) is not limited to this.

[0035] Next, we will explain the transition in depth from the 6th layer to the 1st layer. In the sixth layer, feature map T51 undergoes two convolution operations to generate feature map T52. Then, the feature map T52 is subjected to deconvolution (also called upsampling) to generate the feature map T111 of the fifth layer. Here, the deconvolution process performed is, for example, a deconvolution with a kernel size of 2x2. Note that the inverse convolution operation is equivalent to the inverse operation of the convolution operation.

[0036] Furthermore, in the fifth layer, feature map T41 undergoes two convolutional operations, and the result is processed using a skip connection, which generates feature map T121. Here, the skip connection process involves a copy operation. In other words, feature map T121 is equivalent to a copy of feature map T41 after two convolution operations have been performed on it. Furthermore, during the skip connection processing, cropping may be performed as needed.

[0037] Then, in the fifth layer, the feature map T121 from the fifth layer and the feature map T111 from the sixth layer are merged, and two convolutional operations are performed on the result to generate the feature map T42. Subsequently, the feature map T42 is deconvolved to generate the feature map for the fourth layer.

[0038] In the fourth layer, similar to the fifth layer, the final feature map T32 for the fourth layer is generated using the aforementioned feature map. Similarly, for the third and second layers, the example in Figure 2 shows the final feature map T22 for the third layer and the final feature map T12 for the second layer.

[0039] Furthermore, in the first layer, the feature map T141 generated by skip connection processing on feature map T3 and the feature map T131 generated by deconvolution processing on feature map T12 of the second layer are merged. Then, the result of this merger is subjected to a first convolution or similar process to generate feature map T211, and subsequently, a second convolution or similar process is performed to generate feature map T212.

[0040] Then, the final convolution process is performed on the feature map T212 to generate the output image T213. Here, the final convolution operation is performed, for example, a convolution with a kernel size of 1x1. In the example shown in Figure 2, all convolution operations except for the final convolution are performed using the same kernel size.

[0041] Thus, the U-Net neural network has a two-tower structure of downsampling and upsampling. The output (feature map) from a layer at a certain depth to the layer above is integrated with the feature map of the layer above through skip connection processing in that layer, thereby enabling the reconstruction of overall positional information while preserving local features.

[0042] Note that the U-Net structure shown in Figure 2 is just one example, and the U-Net structure is not limited to the example in Figure 2; other structures of the U-Net may be used.

[0043] [Input and output images of a neural network] Examples of input and output images for the neural network in this embodiment are shown. First, let's look at an example of an input image for a neural network. The input image is an OCT image that can be acquired using a standard OCT device (in this embodiment, a non-polarized OCT device). As a standard OCT device (in this embodiment, a non-polarized OCT device), for example, an FD (Fourier Domain)-OCT device may be used.

[0044] <Example of input image 1> An example of an input image is a normal OCT image that does not contain polarization information (in this embodiment, an example of an unpolarized OCT image). The OCT of a normal OCT image is sometimes called conventional OCT, standard OCT, or scattering OCT.

[0045] <Example of input image 2> An example of an input image is an OCTA (OCT Angiography) image. OCTA is an image that highlights vascular structures by analyzing the temporal fluctuations of OCT signals. OCTA is described, for example, in Non-Patent Document 1.

[0046] <Example of input image 3> An example of an input image is an image of the attenuation coefficient. The attenuation coefficient is the amount of attenuation of the OCT signal in a small depth region. This attenuation is related to the density of the tissue and the intensity of light absorption. The degree of attenuation is greater with increasing tissue density and stronger absorption. The damping coefficient is described, for example, in Non-Patent Document 2.

[0047] <Regarding the number of input images> Here, one input image may be used, or multiple input images may be used. When multiple input images are used, these multiple input images may, for example, be images taken at different points in the same location on the sample at different times, or images taken at the same time on different locations in the vicinity of the sample, or other images may be used.

[0048] For example, if OCTA is composed of multiple image frames (for example, 4 images) taken in succession over time, it is not necessarily required that all of these multiple image frames be used as input images. A portion of these multiple image frames (one or more images, but less than the total number) may be used.

[0049] Next, we will show an example of the output image of a neural network. The output image is an image that can be acquired by a polarized OCT device, which is special hardware, except that it generates a polarized OCT image from a non-polarized OCT image, as in the information processing device 1 according to this embodiment.

[0050] <Example of output image 1> An example of an output image is an image of the polarization phase difference ((Cumulative) Phase Retardation). Polarization phase difference is described, for example, in Non-Patent Documents 3 and 4.

[0051] <Example of output image 2> An example of an output image is a local phase retardation image. Local polarization phase difference is described, for example, in Non-Patent Documents 3 and 4.

[0052] <Example of output image 3> One example of an output image is a birefringence image. The birefringence value referred to here is a quantity related to the strength of birefringence in the tissue. Note that birefringence itself is a general concept. Birefringence is described, for example, in Non-Patent Documents 4 and 5.

[0053] Here, phase retardation is the phase difference between the two polarization states of the OCT probe light, which is caused by birefringence. Generally, when the term "phase retardation" is used, it often refers to cumulative phase retardation. This is the total polarization phase difference experienced by the probe light from the tissue surface to the measurement depth. Furthermore, local phase retardation is the amount of local phase retardation near the measurement depth. This is proportional to the birefringence at that depth.

[0054] The birefringence described above does not directly represent the birefringence inherent in the tissue, but rather is defined by multiplying the local phase retardation by a theoretically determined coefficient (constant). In the machine learning applications of this embodiment, birefringence and local phase retardation may be considered almost synonymous.

[0055] <Example of output image 4> An example of an output image is an image showing polarization uniformity (DOPU). Polarization uniformity is described, for example, in Non-Patent Documents 6 and 7.

[0056] <Example of output image 5> An example of an output image is a depolarization (DOP) image. Depolarization is described, for example, in Non-Patent Document 8.

[0057] <Example of output image 6> An example of an output image is an image of (polarization) Shannon entropy. (Polarization) Shannon entropy is described, for example, in Non-Patent Document 9.

[0058] The polarization uniformity, depolarization, and (polarization) Shannon entropy mentioned above are all quantities that indicate the local variation in the polarization characteristics (near a certain area of ​​tissue) of the polarization signal measured by a polarization OCT device. Polarization uniformity and depolarization take values ​​between 0 and 1 (although their mathematical definitions differ) and are almost the same. (Polarization) Shannon entropy takes different values, but by definition, it has a one-to-one correspondence (bijective relationship) with polarization uniformity.

[0059] <Example of output image 7> An example of an output image is an image of the polarization axis (Optic Axis). The polarization axis is described, for example, in Non-Patent Document 10.

[0060] The polarization axis referred to here represents the orientation of the axis that is paired with the amount of polarization, such as phase retardation. Here, generally, there are two unique polarization states in the tissue. The phase difference that occurs between such two unique polarization states is the phase retardation. The direction of the unique polarization state is the optic axis. Although there are two unique polarization states, generally these are orthogonal to each other, so it is often done to set the direction of either one of the two as the optic axis.

[0061] <Example 8 of the output image> As an example of the output image, there is an image of optic axis uniformity. Optic axis uniformity is described, for example, in Non-Patent Documents 11 and 12. Optic axis uniformity is the uniformity of the optic axis within a local (i.e., small) region of the tissue.

[0062] [Example of hardware for measuring OCT images] Hardware for measuring OCT images is exemplified.

[0063] <Example 1 of hardware for measuring OCT images> As an example of hardware for measuring OCT images, there is a full-functional polarization OCT device. The full-functional polarization OCT device can capture all of the above-mentioned images (the above-mentioned input image and the above-mentioned output image). The full-functional polarization OCT device is described, for example, in Non-Patent Documents 3, 4, and 5.

[0064] Note that the attenuation coefficient is calculated by applying a predetermined algorithm (for example, the algorithm described in Non-Patent Document 2) to the OCT image obtained by the full-functional polarization OCT device. In addition, although birefringence or local phase retardation is not calculated in Non-Patent Document 3, it can be calculated by applying a predetermined algorithm (for example, the algorithm used in Non-Patent Document 4).

[0065] <Example 2 of Hardware for Measuring OCT Images> As an example of hardware for measuring OCT images, there is a simplified polarization OCT device (PAF-OCT device). The simplified polarization OCT device is described in, for example, Non-Patent Document 13. The simplified polarization OCT device can capture images excluding a part of the above-mentioned images (the above-mentioned input image and the above-mentioned output image). The part of the images that cannot be captured are images of cumulative phase retardation, local phase retardation, birefringence, optic axis, and optic axis uniformity.

[0066] [Processing during Learning] The processing when machine learning is performed by the information processing apparatus 1 will be described. In the present embodiment, machine learning of a convolutional neural network that generates a polarization OCT image from a non-polarization OCT image is performed. In this case, as correct value teacher data (also referred to as a teacher image for convenience of explanation), a polarization OCT image acquired using a non-polarization OCT device is used.

[0067] FIG. 3 is a diagram showing an example of the procedure of processing during learning performed in the information processing apparatus 1 according to the embodiment. In the information processing apparatus 1, the learning unit 31 of the control unit 14 performs learning of the convolutional neural network shown in FIG. 2 using an input image and teacher data (teacher image) that is the correct value of the output image (step S1). Then, in the information processing device 1, the learning unit 31 of the control unit 14 stores the learning results in the storage unit 13 (step S2).

[0068] In this embodiment, deep learning (DCNN) is performed in the convolutional neural network. Furthermore, the learning results may be subjected to validation and testing.

[0069] In this embodiment, an unpolarized OCT image without polarization information is used as the input image. Furthermore, a polarized OCT image with polarization information is used as the training image for the output image. As a result, the information processing device 1 learns a convolutional neural network that takes an unpolarized OCT image as input and outputs a polarized OCT image.

[0070] In this case, during training, it is preferable that the input image (unpolarized OCT image) and the training image (polarized OCT image) are obtained from the results measured by the same polarization OCT device. In this case, the positions of the input image and the training image match at the pixel level (i.e., the registrations are correct). As another example, the input image (unpolarized OCT image) may be measured by an unpolarized OCT device, and the training image (polarized OCT image) may be measured by a polarized OCT device. In this case, a process is performed to match the registrations of the input image (unpolarized OCT image) and the training image (polarized OCT image). As yet another example, the input image (unpolarized OCT image) may be measured by a polarized OCT device, and the training image (polarized OCT image) may be measured by another polarized OCT device. In this case, a process is performed to match the registrations of the input image (unpolarized OCT image) and the training image (polarized OCT image).

[0071] [Processing during determination] The process when a determination is made by the information processing device 1 will be explained below. In this embodiment, the results of machine learning of a convolutional neural network are memorized, and based on such learning results, an output image (a pseudo polarized OCT image inferred by the convolutional neural network) corresponding to an input image (a non-polarized OCT image) is determined.

[0072] FIG. 4 is a diagram showing an example of the procedure of the process at the time of determination performed in the information processing apparatus 1 according to the embodiment. In the information processing apparatus 1, the determination unit 32 of the control unit 14 inputs an input image (non-polarized OCT image) to be determined (step S11). Next, the determination unit 32 of the control unit 14 determines an output image (pseudo polarized OCT image) for the input input image based on the learning results stored in the storage unit 13 (step S12). Then, the determination unit 32 of the control unit 14 outputs the determined output image by display or the like (step S13).

[0073] Here, at the time of determination, as the input image (non-polarized OCT image), for example, a non-polarized OCT image measured by a non-polarized OCT apparatus is used. Thereby, in the information processing apparatus 1, an output image (pseudo polarized OCT image) can be acquired from the input image (non-polarized OCT image) based on the learning results. That is, a pseudo polarized OCT image is generated from the non-polarized OCT image measured by the non-polarized OCT apparatus.

[0074] [Specific Example] Hereinafter, specific examples of machine learning and determination based on the results of machine learning are shown.

[0075] <OCT Apparatus Used for Acquisition of OCT Image> In this specific example, both a non-polarized OCT image composed of intensity signals and a polarized OCT image as a teacher image serving as a correct value were acquired by a simplified version of a polarized OCT apparatus (PAF-OCT apparatus) described in Non-Patent Document 13. That is, in this specific example, machine learning was performed by acquiring an input image (non-polarized OCT image) and a teacher image (polarized OCT image) using the same polarized OCT apparatus.

[0076] The center wavelength in a polarized optical coherence tomography (OCT) device is 1 μm. The sweep rate in a polarized OCT device is 100,000 A-lines / s. The sensitivity of the polarized OCT device is 89.5 dB.

[0077] Thus, in this specific example, an unpolarized OCT image consisting of intensity signals, acquired by a simplified polarized OCT device (PAF-OCT device) described in Non-Patent Document 13, was used as the input image. Furthermore, in this specific example, a DOPU image was calculated from the signals of two polarization channels acquired by a simplified polarization OCT device (PAF-OCT device) described in Non-Patent Document 13. This DOPU image was used as a training image during the learning process. This DOPU image was also used to evaluate the accuracy of the judgment based on the learning results.

[0078] The non-polarized OCT image, consisting of intensity signals, was calculated by summing the complex OCT signals of two polarization channels acquired by a simplified polarized OCT device (PAF-OCT device) described in Non-Patent Document 13, after correcting for the constant phase between each channel, and then taking the square of the absolute value.

[0079] In this specific example, for the sake of explanation, the output image (pseudo-DOPU image) which is the result of a judgment based on the learning results may be referred to as a pDOPU image (pseudo-DOPU image).

[0080] <Learning Status> During training, the pDOPU image, which represents the judgment result, was compared with the DOPU image, which represents the correct answer, and the mean absolute error (MAE) between them was calculated. Then, the convolutional neural network was trained to minimize the error obtained from this calculation. Other methods may be used to calculate the error.

[0081] Retinal data measured during a specified period (February 2019 to September 2020) was used for training. We examined 117 eyes from 96 patients and 4 normal eyes from 4 other patients. 105 diseased eyes were used to train the neural network. The unpolarized OCT image, consisting of intensity signals from the B-scan, and the DOPU image from the B-scan were each extracted as (64x64) pixel image patches. A total of 5000 image patches were generated. Of these 5,000 image patches, 4,000 were used for training, updating the parameters of the neural network (such as weights). Additionally, the remaining 1000 image patches were used for validation. Additionally, image patches of the remaining 12 pathological eyes and 4 normal eyes were used as a test dataset.

[0082] <Evaluation Method> Experienced ophthalmologists selected abnormal regions in diseased eyes using unpolarized OCT images from a test dataset. Subsequently, five B-scan images were randomly selected from the abnormal areas of each test eye.

[0083] DOPU and pDOPU images obtained by B-scan were provided to another skilled evaluator. The evaluator then independently and visually assessed the DOPU and pDOPU images for predetermined symptoms. These predetermined symptoms included RPE defect, RPE thickening, RPE elevation, and hyper-reflective foci (HRF).

[0084] Furthermore, for normal eyes, the same evaluators as above assessed the apparent health of the RPE in five equally spaced B-scan images of DOPU and pDOPU images for each test eye.

[0085] <Evaluation Results> In this specific example, the information processing device 1 takes an unpolarized OCT image consisting of intensity signals as an input image and generates a pDOPU image by inferring a DOPU image based on the results of machine learning using the convolutional neural network shown in Figure 2.

[0086] Specific examples of OCT images are shown below with reference to Figures 5A-5C, 6A-6C, 7A-7C, 8A-8C, and 9A-9C. In the following examples, the DOPU image and pDOPU image are originally color images, but for illustrative purposes, they are shown as black and white grayscale images. Furthermore, in the following example, the image is a B-scan image.

[0087] <<Results of evaluation regarding normal eyes>> Regarding the normal eyes, in two out of four eyes, DOPU images were found that showed RPE abnormalities in the findings, but this is due to noise that often occurs in DOPU images. On the other hand, in the case of normal eyes, no RPE abnormalities were found in the pDOPU images.

[0088] Figures 5A, 5B, and 5C show these results. Figure 5A shows an example of an unpolarized OCT image 111 consisting of intensity signals for a normal eye. Figure 5B shows an example of a DOPU image 112 related to a normal eye. In Figure 5B, the RPE defect is indicated by an arrow labeled "RPE defect". Figure 5C shows an example of a pDOPU image 113 related to a normal eye.

[0089] <<Results of evaluation regarding pathological eyes: Case 1>> Figure 6A shows an example of an unpolarized OCT image 131 consisting of intensity signals related to a pathological eye. Figure 6B shows an example of a DOPU image 132 related to a pathological eye. Figure 6C shows an example of a pDOPU image 133 related to a pathological eye.

[0090] In this example, as shown in Figures 6B and 6C, an RPE defect was observed in both the DOPU image 132 and the pDOPU image 133. In Figures 6B and 6C, the RPE defect is indicated by an arrow labeled "RPE defect".

[0091] <<Results of evaluation regarding pathological eyes: Case 2>> Figure 7A shows an example of an unpolarized OCT image 151 consisting of intensity signals related to a pathological eye. Figure 7B shows an example of DOPU image 152 related to a pathological eye. Figure 7C shows an example of pDOPU image 153 related to a pathological eye.

[0092] In this example, as shown in Figures 7B and 7C, RPE elevation and RPE thickening were observed in both DOPU image 152 and pDOPU image 153. In Figures 7B and 7C, the locations of these abnormalities are indicated by arrows labeled "RPE elevation" and "RPE thickening," respectively. Note that in Figure 7C, the pDOPU image 153 is labeled "RPE defect" with an arrow, but this was an erroneous observation.

[0093] <<Results of evaluation regarding pathological eyes: Case 3>> Figure 8A shows an example of an unpolarized OCT image 171 consisting of intensity signals related to a pathological eye. Figure 8B shows an example of DOPU image 172 related to a pathological eye. Figure 8C shows an example of pDOPU image 173 related to a pathological eye.

[0094] In this example, as shown in Figures 8B and 8C, RPE elevation, RPE thickening, and HRF were found in both DOPU image 172 and pDOPU image 173. In Figures 8B and 8C, the locations of these abnormalities are indicated by arrows labeled "RPE elevation," "RPE thickening," and "HRF," respectively.

[0095] <<Results of evaluation regarding pathological eyes: Case 4>> Figure 9A shows an example of an unpolarized OCT image 191 consisting of intensity signals related to a pathological eye. Figure 9B shows an example of a DOPU image 192 related to a pathological eye. Figure 9C shows an example of pDOPU image 193 related to a pathological eye.

[0096] In this example, as shown in Figure 9B, RPE elevation and HRF were found in DOPU image 192. In Figure 9B, the locations of these abnormalities are indicated by arrows labeled "RPE elevation" and "HRF". On the other hand, as shown in Figure 9C, no RPE elevation or HRF was detected in pDOPU image 193. Note that in Figure 9C, the pDOPU image 193 is labeled "RPE defect" with an arrow, but this was an erroneous observation.

[0097] Figure 10 shows Table 1011, which represents the number of clinical features analyzed for DOPU images and pDOPU images. Table 1011 summarizes the number of clinical features analyzed independently by ophthalmologists for both DOPU images and pDOPU images, respectively, regarding pathological eyes. In this specific example, the clinical features are RPE defect, RPE thickening, RPE elevation, and HRF, respectively.

[0098] Regarding RPE defects, 15 cases were positive in both DOPU and pDOPU images. Regarding RPE defects, there were 11 cases that were positive in DOPU images but negative in pDOPU images. Regarding RPE defects, there were 16 cases where the DOPU image was negative and the pDOPU image was positive. The number of cases that were negative in both DOPU and pDOPU images is unknown.

[0099] Regarding RPE thickening, 21 cases were positive in both DOPU and pDOPU images. Regarding RPE thickening, there were 3 cases that were positive in DOPU images but negative in pDOPU images. Regarding RPE thickening, the number of cases that were negative in DOPU images and positive in pDOPU images was 4. The number of cases that were negative in both DOPU and pDOPU images is unknown.

[0100] Regarding RPE elevation, 25 cases were positive in both DOPU and pDOPU images. Regarding RPE elevation, there were 4 cases that were positive on DOPU images but negative on pDOPU images. Regarding RPE elevation, the number of cases that were negative in DOPU images and positive in pDOPU images was 1. The number of cases that were negative in both DOPU and pDOPU images is unknown.

[0101] Regarding HRF, the number of cases that were positive in both DOPU and pDOPU images was 2. Regarding HRF, the number of cases that were positive on DOPU images but negative on pDOPU images was 9. Regarding HRF, there were 5 cases that were negative on DOPU imaging but positive on pDOPU imaging. The number of cases that were negative in both DOPU and pDOPU images is unknown.

[0102] In the example in Table 1011, the degree of agreement between the DOPU image and the pDOPU image for RPE defects was 35.7%. In the example in Table 1011, the degree of agreement in the detection of abnormalities between DOPU images and pDOPU images for RPE thickening was 75.0%. In the example in Table 1011, the degree of agreement in the detection of abnormalities between DOPU images and pDOPU images for RPE elevation was 83.3%. In the example in Table 1011, the degree of agreement in the detection of abnormalities between DOPU images and pDOPU images for HRF was 12.5%.

[0103] In the example in Table 1011, the pDOPU image yielded results that closely resembled the actual DOPU image, particularly in terms of RPE thickening and RPE elevation. In the example in Table 1011, the agreement between RPE defect and HRF and the actual DOPU image is lower compared to RPE thickening and RPE elevation, but it is expected that the agreement will improve with further training. Furthermore, regarding normal RPE, pDOPU images tended to have less noise from RPE defects compared to DOPU images.

[0104] [Regarding the above embodiments] As described above, the information processing device 1 according to this embodiment can perform learning to generate an image equivalent to a polarized OCT image (a pseudo-polarized OCT image) from an unpolarized OCT image. Furthermore, the information processing device 1 according to this embodiment can generate an image equivalent to a polarized OCT image (a pseudo-polarized OCT image) from an unpolarized OCT image based on the learning results.

[0105] Therefore, the information processing device 1 according to this embodiment can acquire a pseudo-polarized OCT image from an unpolarized OCT image. In this embodiment, for example, even without using an expensive and complex polarized OCT device, an image equivalent to a polarized OCT image (a pseudo-polarized OCT image) can be obtained using a non-polarized OCT image obtained with an inexpensive non-polarized OCT device.

[0106] Thus, in this embodiment, for example, an image equivalent to a polarized OCT image can be generated using an unpolarized OCT image obtained from an already widely available unpolarized OCT device. As a result, medical facilities using unpolarized OCT devices can obtain an image equivalent to a polarized OCT image from an unpolarized OCT image using the information processing device 1 according to this embodiment and perform diagnosis without having to purchase an expensive polarized OCT device. Furthermore, in this case, the medical facility can also utilize conventional image processing algorithms and diagnostic algorithms that have been developed for polarized OCT images.

[0107] The information processing device 1 according to this embodiment may be applied, for example, to ophthalmic imaging (particularly to retinal diseases) and cardiovascular imaging (coronary artery) imaging. Furthermore, the information processing device 1 according to this embodiment may be applied, for example, to quality control of cultured tissues and organoids for regenerative medicine, to animal experiments, and to drug efficacy evaluation when measuring drug efficacy using cultured tissues. Furthermore, the information processing device 1 according to this embodiment may be applied, for example, to improve the efficiency of drug development by using it in animal experiments.

[0108] For example, in the field of ophthalmology, analyzing non-polarized OCT images obtained from already widely used non-polarized OCT devices for the fundus will make it possible to construct images that show abnormalities in the pigment epithelium. This is expected to facilitate the diagnosis of age-related macular degeneration and Harada's disease. For example, in the cardiovascular field, using a non-polarized OCT device for coronary catheters, which are already covered by insurance, makes it possible to obtain images equivalent to DOPU images. This allows for discrimination related to the risk of arteriosclerotic substances. For example, in the field of OCT microscopy, the use of inexpensive and simple non-polarized OCT devices (in this case, non-polarized OCT microscopes) makes it possible to visualize the distribution of melanin within tissues. This can accelerate the adoption of OCT microscopes as development equipment for drugs and cosmetics that act on melanin.

[0109] Furthermore, a program to realize the function of any component in any of the devices described above may be recorded on a computer-readable recording medium, and that program may be loaded into a computer system and executed. Here, "computer system" includes hardware such as an operating system (OS) or peripheral devices. "Computer-readable recording medium" refers to portable media such as flexible disks, magneto-optical disks, ROMs, CD (Compact Disc)-ROMs, and storage devices such as hard disks built into computer systems. Moreover, "computer-readable recording medium" also includes volatile memory (RAM) inside computer systems that act as servers or clients when a program is transmitted via a network such as the Internet or a communication line such as a telephone line, which retains the program for a certain period of time.

[0110] Furthermore, the above program may be transmitted from a computer system that stores the program in a memory device or the like to another computer system via a transmission medium or by transmission waves within the transmission medium. Here, the "transmission medium" used to transmit the program refers to a medium that has the function of transmitting information, such as a network like the Internet or a communication line like a telephone line. Furthermore, the above program may be intended to implement only a portion of the functions described above. Moreover, the above program may be a so-called differential file, capable of implementing the aforementioned functions in combination with programs already recorded in the computer system. A differential file may also be called a differential program.

[0111] The functions of any component in any device described above may be implemented by a processor. For example, each process in the embodiment may be implemented by a processor that operates based on information such as a program, and a computer-readable recording medium that stores information such as a program. Here, the processor may be implemented by having the functions of each part implemented by separate hardware, or by having the functions of each part implemented by integrated hardware. For example, the processor includes hardware, and the hardware may include at least one of a circuit that processes digital signals and a circuit that processes analog signals. For example, the processor may be configured using one or more circuit devices or one or both of one or more circuit elements mounted on a circuit board. An IC (Integrated Circuit) may be used as the circuit device, and a resistor or capacitor may be used as the circuit element.

[0112] Here, the processor may be, for example, a CPU. However, the processor is not limited to a CPU; various types of processors may be used, such as a GPU (Graphics Processing Unit) or a DSP (Digital Signal Processor). The processor may also be, for example, a hardware circuit using an ASIC (Application Specific Integrated Circuit). Furthermore, the processor may be composed of, for example, multiple CPUs, or multiple hardware circuits using ASICs. The processor may also be composed of, for example, a combination of multiple CPUs and multiple hardware circuits using ASICs. The processor may also include, for example, one or more amplifier circuits or filter circuits that process analog signals.

[0113] <Example Configuration> <<Example of a learning structure>> As an example configuration, the information processing device 1 includes a learning unit 31 that takes one or more unpolarized OCT images, which are OCT images without polarization information, as input and outputs a pseudo-polarized OCT image corresponding to a polarized OCT image, which is an OCT image that has polarization information, as output, and a storage unit 13 that stores the learning results of the learning unit 31.

[0114] As an example configuration, in the information processing device 1, the machine learning model is a convolutional neural network model. As an example configuration, in the information processing device 1, the input image for the machine learning model is a normal OCT image, an OCTA image, or an image of the attenuation coefficient. The output image for the machine learning model is a pseudo-image of polarization phase difference, local polarization phase difference, birefringence, polarization uniformity, depolarization, Shannon entropy, polarization axis, or polarization axis uniformity. As one example configuration, in the information processing device 1, the unpolarized OCT image and the polarized OCT image that serves as the training image are images acquired by a single polarized OCT device.

[0115] As an example configuration, the program involves having a computer (in this embodiment, the computer constituting the information processing device 1) take one or more unpolarized OCT images, which do not contain polarization information, as input, and having it train a machine learning model that outputs a pseudo-polarized OCT image corresponding to a polarized OCT image, which does contain polarization information, and then storing the training results in the storage unit 13.

[0116] <<Example configuration for performing the judgment>> As an example configuration, the information processing device 1 includes a storage unit 13 that stores the learning results of a machine learning model that takes one or more unpolarized OCT images, which are OCT images without polarization information, as input and outputs a pseudo-polarized OCT image corresponding to a polarized OCT image, which is an OCT image that has polarization information, and a determination unit 32 that, based on the learning results stored in the storage unit 13, uses the machine learning model to determine a pseudo-polarized OCT image according to the input image, which is an unpolarized OCT image.

[0117] As an example configuration, in the information processing device 1, the machine learning model is a convolutional neural network model. As an example configuration, in the information processing device 1, the input image for the machine learning model is a normal OCT image, an OCTA image, or an image of the attenuation coefficient. The output image for the machine learning model is a pseudo-image of polarization phase difference, local polarization phase difference, birefringence, polarization uniformity, depolarization, Shannon entropy, polarization axis, or polarization axis uniformity. As an example configuration, in the information processing device 1, the non-polarized OCT image is an image acquired by a non-polarized OCT device.

[0118] As an example configuration, the program causes a computer (in this embodiment, the computer constituting the information processing device 1) to read the learning results stored in a storage unit 13 that stores the learning results of a machine learning model, which takes one or more unpolarized OCT images (OCT images that do not have polarization information) as input and outputs pseudo-polarized OCT images corresponding to polarized OCT images (OCT images that do have polarization information) as output. Based on the read learning results, the program causes the machine learning model to determine a pseudo-polarized OCT image according to the input image, which is an unpolarized OCT image.

[0119] [Differentiation] A modified example of the embodiment is shown. In the modified example, for the sake of explanation, Figure 1 will be used to describe the information processing device 1 related to the modified example. For the sake of explanation, we will use the symbols shown in Figure 1.

[0120] <Example 1> In this modified example, the information processing device 1 acquires information on the learning results of machine learning stored in an external device, and the determination unit 32 makes a determination based on the acquired learning result information.

[0121] Here, the external device may be any device, for example, a server device located on a network such as the Internet. In this case, the information processing device 1 communicates with the server device via the network, receives information on the learning results of machine learning from the server device, and uses the received information for determination by the determination unit 32. This communication may be, for example, a wired communication or a wireless communication.

[0122] The external device in question may, for example, be a device that provides a service that provides information on the learning results of machine learning. This service may be a paid service or a free service. The information obtained from the machine learning training may be stored, for example, in a storage device such as a database that can be accessed by the external device. Furthermore, the information regarding the learning results of machine learning may be updated at any time.

[0123] The information processing device 1 may, when necessary, access the external device and receive information on the learning results of machine learning from the external device. The information processing device 1 may, for example, access an external device in response to instructions from a person (user) operating the information processing device 1 and receive information on the learning results of machine learning from the external device, or it may automatically access the external device and receive information on the learning results of machine learning from the external device when predetermined conditions are met.

[0124] The information processing device 1 may also store the information received from the external device in the storage unit 13. In this case, the information provided by the external device may be temporarily stored and referenced in the storage unit 13 of the information processing device 1, but may not be stored long-term. In this configuration, the information processing device 1 accesses the external device whenever necessary to receive information on the learning results of machine learning from the external device.

[0125] Furthermore, a system including the information processing device 1 and the external device (for example, an information processing system) may be implemented.

[0126] As described above, the information processing device 1 may acquire information on the learning results of machine learning from an external device and perform a determination by the determination unit 32 based on the acquired information. In this modified example, the information processing device 1 does not need to include the learning unit 31. In other words, the information processing device 1 may perform a determination by the determination unit 32 using machine learning learning results received from an external source, without having a function to perform machine learning or a function to store information of the learning results of machine learning.

[0127] <<Example configuration for performing the judgment>> As one example configuration, the information processing device 1 takes one or more unpolarized OCT images, which are OCT images that do not have polarization information, as input, and includes a determination unit 32 that, based on the learning results of a machine learning model, uses the machine learning model to determine a pseudo-polarized OCT image that corresponds to a polarized OCT image, which is an OCT image that has polarization information, according to the input unpolarized OCT image.

[0128] As an example configuration, the program involves having a computer (in this modified example, the computer constituting the information processing device 1) acquire the training results of a machine learning model, inputting one or more unpolarized OCT images that do not contain polarization information, and then, based on the acquired training results, having the machine learning model determine a pseudo-polarized OCT image that corresponds to a polarized OCT image that contains polarization information, according to the input unpolarized OCT image.

[0129] <Modification 2> Figure 1 will be used to explain other variations. The information processing device 1 has a function to perform machine learning using the learning unit 31, but it may also transmit the learning results of the machine learning to an external device (for example, an external memory device) and store the information in that external device. In this case, the external device may be installed in any location, for example, it may be installed so as to be directly connectable to the information processing device 1, or it may be installed so as to be able to communicate with the information processing device 1 via a network such as the Internet. The information processing device 1 accesses the external device when necessary, receives information on the learning results of machine learning from the external device, and uses it.

[0130] Furthermore, a system including the information processing device 1 and the external device (for example, an information processing system) may be implemented.

[0131] <Variation 3> Figure 1 will be used to explain other variations. In the information processing device 1, the determination unit 32 may use the functions of an external device to perform a determination (in this modified example, obtaining a determination result as an example of a determination). For example, when the determination unit 32 determines output information corresponding to the input information, it may transmit the input information to the external device, receive the output information obtained by the external device from the external device, and perform a process to use the obtained output information as the determination result.

[0132] In this case, the external device has the function of obtaining information (output information) corresponding to the information (input information) received from the information processing device 1. The external device, for example, uses a predetermined machine learning model to determine, based on the learning results of the machine learning model, information (output information) corresponding to the information (input information) received from the information processing device 1 by the machine learning model. The external device may, for example, be a device that provides a service to the information processing device 1 by sending information (output information from the machine learning model) in response to information received from the information processing device 1 (input information to the machine learning model). This service may be a paid service or a free service.

[0133] Here, the information processing device 1 may transmit information (which may be stored in the storage unit 13) of the learning results of machine learning performed by the learning unit 31 to the external device. In this case, the external device uses the information received from the information processing device 1 to obtain information (output information) corresponding to the information (input information) received from the information processing device 1. In this case, the external device may also store the information received from the information processing device 1 in a storage device such as a database.

[0134] Alternatively, the information processing device 1 does not need to have the functions of the learning unit 31. In this case, the information processing device 1 does not need to have the functions of performing machine learning and storing information of the learning results of machine learning. In this case, the external device stores information about the learning results of machine learning in a storage device such as a database.

[0135] The external device may be, for example, connected to a network such as the Internet, and communicate with the information processing device 1 via the network. This communication may be, for example, wired communication or wireless communication.

[0136] Furthermore, a system including the information processing device 1 and the external device (for example, an information processing system) may be implemented.

[0137] <Modification 4> Figure 1 will be used to explain other variations. The information processing device 1 has a function for performing machine learning and a function for storing information on the learning results of machine learning, but it does not have to have the function of the determination unit 32. In this case, the information processing device 1 may, for example, provide (e.g., transmit) information of the learning results of machine learning stored in the memory unit 13 to another device.

[0138] Furthermore, a system (e.g., an information processing system) may be implemented that includes an information processing device 1 having a machine learning function, and another device (e.g., another information processing device) that makes decisions based on the learning results of the machine learning.

[0139] <Modification 5> Let's explain other variations. In this modified version, both OCT and OCTA images are used as inputs and simultaneously during the training and decision-making phases of the machine learning model.

[0140] Specific examples are shown with reference to Figures 11A-11E, 12A-12E, and 13A-13E. In the following examples, the DOPU image is actually a color image, but for illustrative purposes, it is shown as a black and white grayscale image. Furthermore, in the following example, the image is a B-scan image.

[0141] In this example, we will compare the methods of Model 1 and Model 2. The Model 1 method is the method of the embodiment described above, and is a method that uses OCT images as input during the training and decision-making phases of the machine learning model. The Model 2 method is a modified version of this method, which uses a combination of OCT and OCTA images as a multi-channel image as input during the training and decision-making phases of the machine learning model.

[0142] Examples from Figures 11A to 11E will be explained. Figure 11A shows an example of an OCT image 311 used in Model 1 and Model 2. Figure 11B shows an example of an OCTA image 312 used in Model 2. Figure 11C shows an example of a true DOPU image 313, which is a true DOPU image used in Model 1 and Model 2. Figure 11D shows an example of a pseudo-DOPU image 314 generated by the Model 1 method. Figure 11E shows an example of a pseudo-DOPU image 315 generated by the Model 2 method.

[0143] In Model 1, the machine learning model took the OCT image 311 as input and the true DOPU image 313 as the true value (training image). Then, Model 1 generated a pseudo-DOPU image 314. In the Model 2 method, the machine learning model took OCT image 311 and OCTA image 312 as input, and the true DOPU image 313 was used as the true value (training image). Then, the Model 2 method generated a pseudo-DOPU image 315.

[0144] These results show that the pseudo-DOPU image 315 obtained using Model 2 was closer to the true DOPU image 313 than the pseudo-DOPU image 314 obtained using Model 1. In other words, Model 2 improved the accuracy of pseudo-DOPU image generation compared to Model 1.

[0145] For example, Figure 11C shows a true DOPU image 313, a predetermined portion 321a of an object included in the image, and an enlarged image of the predetermined portion 321a (enlarged image 321b). Similarly, Figure 11D shows a pseudo-DOPU image 314 generated by the Model 1 method, including a predetermined portion 322a of the object and an enlarged image of the predetermined portion 322a (enlarged image 322b). Similarly, Figure 11E shows a pseudo-DOPU image 315 generated by the Model 2 method, including a predetermined portion 323a of the object and an enlarged image of the predetermined portion 323a (enlarged image 323b).

[0146] Here, these predetermined parts (predetermined part 321a, predetermined part 322a, predetermined part 323a) are the same part of the same object. As shown in Figures 11C to 11E, it can be seen that the Model 2 method has higher accuracy in generating pseudo-DOPU images than the Model 1 method.

[0147] Examples from Figures 12A to 12E will be explained. Figure 12A shows an example of an OCT image 341 used in Model 1 and Model 2. Figure 12B shows an example of an OCTA image 342 used in Model 2. Figure 12C shows an example of a true DOPU image 343, which is a true DOPU image used in Model 1 and Model 2. Figure 12D shows an example of a pseudo-DOPU image 344 generated by the Model 1 method. Figure 12E shows an example of a pseudo-DOPU image 345 generated by the Model 2 method.

[0148] In Model 1, the machine learning model took OCT image 341 as input and true DOPU image 343 as the true value (training image). Then, Model 1 generated a pseudo-DOPU image 344. In the Model 2 method, the machine learning model took OCT image 341 and OCTA image 342 as input, and the true DOPU image 343 was used as the true value (training image). Then, the Model 2 method generated a pseudo-DOPU image 345.

[0149] These results show that the pseudo-DOPU image 345 obtained using Model 2 was closer to the true DOPU image 343 than the pseudo-DOPU image 344 obtained using Model 1. In other words, Model 2 improved the accuracy of pseudo-DOPU image generation compared to Model 1.

[0150] For example, Figure 12C shows three predetermined parts 351a, 352a, and 353a of an object included in the true DOPU image 343, as well as magnified images of these three predetermined parts 351a, 352a, and 353a (magnified images of the three parts 351b, 352b, and 353b). Similarly, Figure 12D shows the pseudo-DOPU image 344 generated by the Model 1 method, including three predetermined parts 354a, 355a, and 356a of the object, and magnified images of these three predetermined parts 354a, 355a, and 356a (magnified images of the three parts 354b, 355b, and 356b). Similarly, Figure 12E shows the pseudo-DOPU image 345 generated by the Model 2 method, including three predetermined parts 357a, 358a, and 359a of the object, and magnified images of these three predetermined parts 357a, 358a, and 359a (magnified images of the three parts 357b, 358b, and 359b).

[0151] Here, the first of these three designated parts (designated part 351a, designated part 354a, designated part 357a) is the same part of the same object. Similarly, the second of these three designated parts (designated part 352a, designated part 355a, designated part 358a) is the same part of the same object. Similarly, the third of these three designated parts (designated part 353a, designated part 356a, designated part 359a) is the same part of the same object. As shown in Figures 12C to 12E, it can be seen that the Model 2 method has higher accuracy in generating pseudo-DOPU images than the Model 1 method.

[0152] Examples from Figures 13A to 13E will be explained. Figure 13A shows an example of an OCT image 371 used in Model 1 and Model 2. Figure 13B shows an example of OCTA image 372 used in Model 2. Figure 13C shows an example of a true DOPU image 373, which is a true DOPU image used in Model 1 and Model 2. Figure 13D shows an example of a pseudo-DOPU image 374 generated by the Model 1 method. Figure 13E shows an example of a pseudo-DOPU image 375 generated using the Model 2 method.

[0153] In Model 1, the machine learning model took OCT image 371 as input and true DOPU image 373 as the true value (training image). Then, Model 1 generated a pseudo-DOPU image 374. In Model 2, the machine learning model took OCT image 371 and OCTA image 372 as input, and the true DOPU image 373 was used as the true value (training image). Then, in Model 2, a pseudo-DOPU image 375 was generated.

[0154] These results show that the pseudo-DOPU image 375 obtained using Model 2 was closer to the true DOPU image 373 than the pseudo-DOPU image 374 obtained using Model 1. In other words, Model 2 improved the accuracy of pseudo-DOPU image generation compared to Model 1.

[0155] For example, Figure 13C shows a true DOPU image 373, a predetermined portion 381a of an object included in the image, and an enlarged image of the predetermined portion 381a (enlarged image 381b). Similarly, Figure 13D shows a pseudo-DOPU image 374 generated by the Model 1 method, including a predetermined portion 382a of the object and an enlarged image of the predetermined portion 382a (enlarged image 382b). Similarly, Figure 13E shows a pseudo-DOPU image 375 generated by the Model 2 method, including a predetermined portion 383a of the object and an enlarged image of the predetermined portion 383a (enlarged image 383b).

[0156] Here, these predetermined parts (predetermined part 381a, predetermined part 382a, predetermined part 383a) are the same part of the same object. As shown in Figures 13C to 13E, it can be seen that the Model 2 method has higher accuracy in generating pseudo-DOPU images than the Model 1 method.

[0157] In the example above, Model 2 was shown as a machine learning model that takes multi-channel (2-channel) images as input, using OCT images and OCTA images as multi-channel (2-channel) inputs. However, other types of images can be used as the combination of multiple images. For example, a combination of OCT images and attenuation coefficient images as multi-channel (2-channel) inputs may be used, or a combination of OCTA images and attenuation coefficient images as multi-channel (2-channel) inputs may be used. Furthermore, as an example of the Model 2 method, a machine learning model that takes multi-channel (3-channel) images as input may be used, with a combination of OCT images, OCTA images, and attenuation coefficient images as multi-channel (3-channel) inputs.

[0158] Furthermore, while the above example showed a configuration in which two or more images are used as a multi-channel image, another example is a configuration in which an image resulting from inter-image operations is used as the combination of two or more images. In this case, for example, the image resulting from the inter-image operations is used as the input image to the machine learning model in the above embodiment.

[0159] The image operation may include, for example, operations to "add" two or more images together, operations to "multiply" two or more images together, or operations to "subtract" two or more images from each other. Alternatively, the image operation may involve performing bitwise operations on two or more images.

[0160] As a concrete example, an image may be generated as a result of an inter-image operation by adding, multiplying, or subtracting the pixel data of two or more images that are at the same position on the subject (for example, the same position within the image frame). Furthermore, when adding, multiplying, or subtracting pixel data, the results of these operations may be normalized, for example. As one example, as a form of adding pixel data, averaging of the pixel data may be performed.

[0161] As described above, the input images for the machine learning model may be a combination of two or more of the following: regular OCT images, OCTA images, or attenuation coefficient images. Furthermore, as a combination of two or more images, for example, a method may be used in which these two or more images are used as a multi-channel image, or a method may be used in which an image obtained by performing a predetermined operation on these two or more images is used.

[0162] Furthermore, both of the above two embodiments may be used simultaneously as a combination of two or more images. In other words, a configuration may be used in which the input to the machine learning model is multiple images (multi-channel images), and one or more of these multiple images is an image obtained by performing a predetermined operation on two or more types of images.

[0163] <Variation 6> Let's explain other variations. In this modified version, numerical noise is added to the input image (e.g., an OCT image) during the training of the machine learning model. As a result, in this modified version, learning is generalized by adding numerical noise to the input image, enabling the generation of accurate pseudo-DOPU images even when, for example, OCT images obtained from an OCT device not used in the learning process are used as input.

[0164] In the above embodiment, we show a case where data measured with the same device (Swept-Source (SS)-OCT device) is used for both neural network training and subsequent image generation. In this case, for example, even if data acquired with a different type of device (Spectral-Domain(SD)-OCT device) is input to the trained network, it will not be possible to obtain accurate pseudo-DOPU images.

[0165] Therefore, in this modified version, the generalization of the network was promoted by intentionally adding (numerically) noise to the input image during training. As a result, it was confirmed that reasonable pseudo-DOPU images could be generated whether image data measured with an SS-OCT device was used for training or image data captured with an SD-OCT device was used for judgment.

[0166] Here, the numerical noise added to the image may also be noise added to the image intensity. Furthermore, the numerical noise added to the image may be the same complex noise added to the complex OCT signal. This complex noise may, for example, mimic the distribution of physical noise that takes into account the physical imaging principle of OCT. Furthermore, the numerical noise added to the image may be a combination of two or more of the various types of noise described above.

[0167] Furthermore, as a method for adding noise to the input image, for example, data augmentation, a common technique used in machine learning, may be employed.

[0168] In this modified example, for instance, it becomes possible to generate pseudo-DOPU images with sufficient accuracy even from data acquired using an OCT device from a different individual or a different type of OCT device than the one used for training. The input image to which noise is added can be any type of image used as an input image during training.

[0169] Furthermore, the method of adding noise to the input image does not necessarily have to involve adding noise to the input image itself. As another example, noise may be added to the OCT signal that forms the basis of any type of image used as an input image during training, so that the resulting image (the type of image used as an input image during training) becomes a noisy image. In other words, in addition to methods that directly add noise to the input image itself, methods that add noise to the measurement signal that generates the input image, thereby generating an input image with a low signal-to-noise ratio (SNR), may also be used. This method may be used, for example, to generate an image with noise based on physical principles (for example, an image generated by adding complex noise to the complex OCT signal described above). Although some physical noise can be reproduced even without using this method, using this method allows for the addition of more appropriate noise to the image.

[0170] [Appended Note] The following shows a configuration example. (Configuration Example 1) A learning unit that performs learning of a machine learning model that takes as input one or more non-polarized OCT images that are OCT images without polarization information and outputs a pseudo-polarized OCT image corresponding to a polarized OCT image that is an OCT image with polarization information, A storage unit that stores the learning result of the learning unit, An information processing apparatus comprising:

[0171] (Configuration Example 2) The machine learning model is a model of a convolutional neural network. The information processing apparatus according to (Configuration Example 1).

[0172] (Configuration Example 3) The input image of the machine learning model is a normal OCT image, an OCTA image, an attenuation coefficient image, or a combination of two or more of these images, The output image of the machine learning model is a pseudo-image of polarization phase difference, local polarization phase difference, birefringence, polarization uniformity, depolarization, Shannon entropy, polarization axis, or polarization axis uniformity. The information processing apparatus according to (Configuration Example 1) or (Configuration Example 2).

[0173] (Configuration Example 4) The non-polarized OCT image and the polarized OCT image that is the teacher image for the learning are images acquired by one polarized OCT device. The information processing apparatus according to any one of (Configuration Example 1) to (Configuration Example 3).

[0174] (Configuration Example 5) One or more non-polarized OCT images that are OCT images without polarization information are used as input, Based on the learning result of the machine learning model, a determination unit that determines a pseudo-polarized OCT image corresponding to a polarized OCT image that is an OCT image with polarization information according to the non-polarized OCT image that is the input image by the machine learning model is provided. Information processing device.

[0175] (Configuration example 6) Furthermore, the system includes a storage unit that stores the learning results of the machine learning model, which takes one or more unpolarized OCT images as input and outputs a pseudo-polarized OCT image. The determination unit, based on the learning results stored in the memory unit, uses the machine learning model to determine a pseudo-polarized OCT image according to the input image, which is an unpolarized OCT image. The information processing device described in (Configuration Example 5).

[0176] (Configuration example 7) The aforementioned machine learning model is a convolutional neural network model. The information processing device described in (Configuration Example 5) or (Configuration Example 6).

[0177] (Configuration example 8) The input image for the aforementioned machine learning model is a regular OCT image, an OCTA image, or an image of the attenuation coefficient, or an image of a combination of two or more of these. The output image of the machine learning model is a pseudo-image of polarization phase difference, local polarization phase difference, birefringence, polarization uniformity, depolarization, Shannon entropy, polarization axis, or polarization axis uniformity. An information processing device as described in any one of (Configuration Example 5) to (Configuration Example 7).

[0178] (Configuration example 9) The aforementioned non-polarized OCT image is an image acquired by a non-polarized OCT device. An information processing device as described in any one of (Configuration Example 5) to (Configuration Example 8).

[0179] [Regarding the above] Although embodiments of this invention have been described in detail above with reference to the drawings, the specific configuration is not limited to these embodiments and includes designs and the like that do not depart from the spirit of this invention. [Explanation of Symbols]

[0180] 1. Information Processing Device 11 Input section 12 Output section 13 Storage section 14 Control Unit 31. Learning Department 32 Judgment section Unpolarized OCT image consisting of intensity signals 111, 131, 151, 171, and 191. 112, 132, 152, 172, 192 DOPU images 113, 133, 153, 173, 193 pDOPU images 311, 341, 371 OCT images 312, 342, 372 OCTA images 313, 343, 373 True DOPU images 314, 315, 344, 345, 374, 375 Pseudo-DOPU images 321a, 322a, 323a, 351a, 352a, 353a, 354a, 355a, 356a, 357a, 358a, 359a, 381a, 382a, 383a Specified parts 321b, 322b, 323b, 351b, 352b, 353b, 354b, 355b, 356b, 357b, 358b, 359b, 381b, 382b, 383b Enlarged image 1011 Table T1 input image Feature Maps for T2-T3, T11-T12, T21-T22, T31-T32, T41-T42, T51-T52, T111, T121, T131, T141, and T211-T212 T213 Output Image

Claims

1. A learning unit that trains a machine learning model that takes one or more unpolarized OCT images, which do not have polarization information, as input, and outputs a pseudo-polarized OCT image corresponding to a polarized OCT image, which does have polarization information. A storage unit for storing the learning results of the aforementioned learning unit, Equipped with, The unpolarized OCT image and the polarized OCT image that serves as the training image are images acquired by a single polarized OCT device. When making a determination using the aforementioned learning results, an unpolarized OCT image acquired by a non-polarized OCT device different from the polarized OCT device is used as input. The input is a combination of an OCT image and an OCTA image as a multi-channel image. Information processing device.

2. The input is one or more unpolarized OCT images that do not contain polarization information. Based on the learning results of the machine learning model, the machine learning model determines a pseudo-polarized OCT image that corresponds to a polarized OCT image, which is an OCT image that has polarization information, according to the input image, which is an unpolarized OCT image. The aforementioned non-polarized OCT image is an image acquired by a non-polarized OCT device. The learning results are obtained through learning in which an unpolarized OCT image acquired by a polarized OCT device different from the aforementioned unpolarized OCT device is used as input, and a polarized OCT image acquired by the aforementioned polarized OCT device is used as the training image. The input is a combination of an OCT image and an OCTA image as a multi-channel image. Information processing device.

3. A learning unit that trains a machine learning model that takes one or more unpolarized OCT images, which do not have polarization information, as input, and outputs a pseudo-polarized OCT image corresponding to a polarized OCT image, which does have polarization information. A storage unit for storing the learning results of the aforementioned learning unit, Equipped with, The unpolarized OCT image and the polarized OCT image that serves as the training image are images acquired by a single polarized OCT device. moreover, During the determination process, one or more unpolarized OCT images, which do not contain polarization information, are taken as input. Based on the learning results of the machine learning model, the machine learning model determines a pseudo-polarized OCT image that corresponds to a polarized OCT image, which is an OCT image that has polarization information, according to the input image, which is an unpolarized OCT image. The aforementioned non-polarized OCT image is an image acquired by a non-polarized OCT device different from the polarized OCT device. The input is a combination of an OCT image and an OCTA image as a multi-channel image. Information processing device.

4. On the computer, A machine learning model is trained that takes one or more unpolarized OCT images (which do not contain polarization information) as input and outputs a pseudo-polarized OCT image corresponding to a polarized OCT image (which does contain polarization information). The learning results are stored in the memory unit. It is a program, The unpolarized OCT image and the polarized OCT image that serves as the training image are images acquired by a single polarized OCT device. When making a determination using the aforementioned learning results, an unpolarized OCT image acquired by a non-polarized OCT device different from the polarized OCT device is used as input. The input is a combination of an OCT image and an OCTA image as a multi-channel image. program.

5. On the computer, Obtain the training results of the machine learning model, One or more unpolarized OCT images, which do not contain polarization information, are input. Based on the acquired learning results, the machine learning model determines a pseudo-polarized OCT image that corresponds to a polarized OCT image, which is an OCT image containing polarization information, in accordance with the input image, which is an unpolarized OCT image. It is a program, The aforementioned non-polarized OCT image is an image acquired by a non-polarized OCT device. The learning results are obtained through learning in which an unpolarized OCT image acquired by a polarized OCT device different from the aforementioned unpolarized OCT device is used as input, and a polarized OCT image acquired by the aforementioned polarized OCT device is used as the training image. The input is a combination of an OCT image and an OCTA image as a multi-channel image. program.

Images