DATA PROCESSING METHOD, DATA PROCESSING APPARATUS, THREE-DIMENSIONAL OBSERVATION APPARATUS, LEARNING METHOD, LEARNING APPARATUS, AND RECORDING MEDIUM

Abstract

Provided is a data processing method for generating an index by which the confidence level of estimation is known.　This data processing method comprises: an input step S1 for inputting measurement data to a neural network; an estimation step S2 for generating estimation data from the measurement data; a reconstruction step S3 for generating reconstruction data from the estimation data; and a calculation step S4 for calculating the confidence level of the estimation data on the basis of the measurement data and the reconstruction data. The neural network is a learned model, the measurement data is data obtained by measuring light transmitted through an object, the estimation data is data relating to a three-dimensional optical property of the object estimated from the measurement data, and the three-dimensional optical property is a refractive index distribution or an absorptance distribution. In the estimation, the neural network is used. In the reconstruction, a forward propagation operation is performed on the estimation data, and in the forward propagation operation, wave fronts passing through the object estimated from the measurement data are sequentially obtained in a direction in which the light travels.

Description

[Technical field]

[0001] The present invention relates to a data processing method, a data processing device, a three-dimensional observation device, a learning method, a learning device, and a recording medium. [Background technology]

[0002] Patent Document 1 discloses a microscope using a deep neural network. In this microscope, learning is performed using high-resolution images and low-resolution images. Since a trained deep neural network is used, images with improved image quality can be output at high speed. Image quality includes, for example, spatial resolution, depth of field, signal-to-noise ratio, and contrast. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] US Patent Application Publication No. 2019 / 0333199 Summary of the Invention [Problem to be solved by the invention]

[0004] Deep Neural Networks can sometimes output results with low estimation accuracy. For example, when estimating the 3D optical properties of an object, the estimated 3D optical properties can differ significantly from the 3D optical properties of the actual object, making it difficult to determine whether the estimation results can be trusted.

[0005] The present invention has been made in consideration of such problems, and aims to provide a data processing method, a data processing device, a three-dimensional observation device, and a recording medium for generating an index indicating the reliability of an estimation, and a learning method, a learning device, and a recording medium for generating an index indicating the reliability of an estimation. [Means for solving the problem]

[0006] In order to solve the above problems and achieve the above object, a data processing method according to at least some embodiments of the present invention includes: an input step of inputting measurement data into the neural network; an estimation step of generating estimated data from the measurement data; A restoration step of generating restored data from the estimated data; A calculation step of calculating a reliability of the estimated data based on the measurement data and the restored data, A neural network is a trained model. The measurement data is data obtained by measuring light transmitted through an object, The estimated data is data of three-dimensional optical properties of the object estimated from the measured data, The three-dimensional optical property is a refractive index distribution or an absorptivity distribution, The estimation uses a neural network. In the reconstruction, we perform a forward propagation operation on the estimation data, The forward propagation calculation is characterized by sequentially determining the wavefront passing through the inside of an object, estimated from measurement data, in the direction in which light travels.

[0007] Moreover, a data processing method according to at least some embodiments of the present invention includes: an input step of inputting measurement data into the neural network; an estimation step of generating estimated data from the measurement data; A restoration step of generating restored data from the estimated data; A calculation step of calculating a reliability of the estimated data based on the measurement data and the restored data; A learning step in which the neural network learns using a loss that is inversely proportional to the confidence level; The measurement data is data obtained by measuring light transmitted through an object, The estimated data is data of three-dimensional optical properties of the object estimated from the measured data, The three-dimensional optical property is a refractive index distribution or an absorptivity distribution, The estimation uses a neural network. In the reconstruction, we perform a forward propagation operation on the estimation data, The forward propagation calculation is characterized by sequentially determining the wavefront passing through the inside of an object, estimated from measurement data, in the direction in which light travels.

[0008] Moreover, a data processing device according to at least some embodiments of the present invention comprises: A memory and a processor, The memory stores the measurement data. The processor executes an estimation process to estimate three-dimensional optical properties of the object; The three-dimensional optical property is a refractive index distribution or an absorptivity distribution, The estimation process is as follows: an input step of inputting measurement data into the neural network; an estimation step of generating estimated data from the measurement data; A restoration step of generating restored data from the estimated data; A calculation step of calculating a reliability of the estimated data based on the measurement data and the restored data, A neural network is a trained model. The measurement data is data obtained by measuring light transmitted through an object, The estimated data is data of three-dimensional optical properties of the object estimated from the measured data, The estimation uses a neural network. In the reconstruction, we perform a forward propagation operation on the estimation data, The forward propagation calculation is characterized by sequentially determining the wavefront passing through the inside of an object, estimated from measurement data, in the direction in which light travels.

[0009] Moreover, a data processing device according to at least some embodiments of the present invention comprises: A memory and a processor, The memory stores the measurement data. The processor executes an estimation process to estimate three-dimensional optical properties of the object; The three-dimensional optical property is a refractive index distribution or an absorptivity distribution, The estimation process is as follows: an input step of inputting measurement data into the neural network; an estimation step of generating estimated data from the measurement data; A restoration step of generating restored data from the estimated data; A calculation step of calculating a reliability of the estimated data based on the measurement data and the restored data; A learning step in which the neural network learns using a loss that is inversely proportional to the confidence level; The measurement data is data obtained by measuring light transmitted through an object, The estimated data is data of three-dimensional optical properties of the object estimated from the measured data, The estimation uses a neural network. In the reconstruction, we perform a forward propagation operation on the estimation data, The forward propagation calculation is characterized by sequentially determining the wavefront passing through the inside of an object, estimated from measurement data, in the direction in which light travels.

[0010] In addition, a recording medium according to at least some embodiments of the present invention includes: A computer-readable recording medium having a program recorded thereon for causing a computer having a memory and a processor to execute an estimation process, In the estimation process, the three-dimensional optical properties of the object are estimated, The three-dimensional optical property is a refractive index distribution or an absorptivity distribution, The processor: A process of inputting the measurement data stored in the memory into a neural network; A process of generating estimated data from the measured data; A process of generating restored data from the estimated data; Calculating the reliability of the estimated data based on the measured data and the restored data; A neural network is a trained model. The measurement data is data obtained by measuring light transmitted through an object, The estimated data is data of three-dimensional optical properties of the object estimated from the measured data, The estimation uses a neural network. In the reconstruction, we perform a forward propagation operation on the estimation data, The forward propagation calculation is characterized by sequentially determining the wavefront passing through the inside of an object, estimated from measurement data, in the direction in which light travels.

[0011] In addition, a recording medium according to at least some embodiments of the present invention includes: A computer-readable recording medium having a program recorded thereon for causing a computer having a memory and a processor to execute an estimation process, In the estimation process, the three-dimensional optical properties of the object are estimated, The three-dimensional optical property is a refractive index distribution or an absorptivity distribution, The processor: A process of inputting the measurement data stored in the memory into a neural network; A process of generating estimated data from the measured data; A process of generating restored data from the estimated data; A process of calculating a reliability of the estimated data based on the measurement data and the restored data; The neural network learns by using a loss that is inversely proportional to the confidence level. The measurement data is data obtained by measuring light transmitted through an object, The estimated data is data of three-dimensional optical properties of the object estimated from the measured data, The estimation uses a neural network. In the reconstruction, we perform a forward propagation operation on the estimation data, The forward propagation calculation is characterized by sequentially determining the wavefront passing through the inside of an object, estimated from measurement data, in the direction in which light travels.

[0012] In addition, a method for training a neural network according to at least some embodiments of the present invention includes the steps of: A learning method for a neural network that calculates the reliability of estimated data, comprising: The reliability of the estimated data is calculated based on the measured data and the restored data; The measurement data is data obtained by measuring light transmitted through an object, The estimated data is data of three-dimensional optical properties of the object estimated from the measured data, The three-dimensional optical property is a refractive index distribution or an absorptivity distribution, The restored data is data generated by performing a forward propagation operation on the estimated data, In forward propagation, the wavefront that passes through the inside of the object, estimated from the measurement data, is calculated in the direction in which the light travels. a first learning step of learning using a first learning data set; A second learning step of learning using a second learning data set, The first learning step and the second learning step are repeatedly performed, The first learning data set includes the first data, the first correction data, and training data indicating true among true and false, The second learning data set includes the first data, the second correction data, and training data indicating false among true and false, The first correction data is data obtained by performing a correction process on the first data, The second correction data is data obtained by performing a correction process on the second data, The second data is different from the first data, The first data and the second data are characterized in that they are data obtained by measuring light transmitted through an object, or data generated by performing a forward propagation operation on an object model that models the three-dimensional optical characteristics of the object.

[0013] In addition, a learning device according to at least some embodiments of the present invention A memory and a processor, The memory stores the measurement data. The processor executes a learning process on the neural network that calculates the reliability of the estimated data; The reliability of the estimated data is calculated based on the measured data and the restored data; The measurement data is data obtained by measuring light transmitted through an object, The estimated data is data of three-dimensional optical properties of the object estimated from the measured data, The three-dimensional optical property is a refractive index distribution or an absorptivity distribution, The restored data is data generated by performing a forward propagation operation on the estimated data, In the forward propagation calculation, the wavefront passing through the inside of the object estimated from the measurement data is calculated in the direction in which the light travels, The learning process is a first learning step of learning using a first learning data set; A second learning step of learning using a second learning data set, The first learning step and the second learning step are repeatedly performed, The first learning data set includes the first data, the first correction data, and training data indicating true among true and false, The second learning data set includes the first data, the second correction data, and training data indicating false among true and false, The first correction data is data obtained by performing a correction process on the first data, The second correction data is data obtained by performing a correction process on the second data, The second data is different from the first data, The first data and the second data are characterized in that they are data obtained by measuring light transmitted through an object, or data generated by performing a forward propagation operation on an object model that models the three-dimensional optical characteristics of the object.

[0014] In addition, a recording medium according to at least some embodiments of the present invention includes: A computer-readable recording medium having a program recorded thereon for causing a computer having a memory and a processor to execute a learning process for a neural network, The neural network calculates the reliability of the estimated data based on the measurement data and the restored data; The measurement data is data obtained by measuring light transmitted through an object, The estimated data is data of three-dimensional optical properties of the object estimated from the measured data, The three-dimensional optical property is a refractive index distribution or an absorptivity distribution, The restored data is data generated by performing a forward propagation operation on the estimated data, In the forward propagation calculation, the wavefront passing through the inside of the object estimated from the measurement data is calculated in the direction in which the light travels, a first learning step of learning using a first learning data set; A second learning step of learning using a second learning data set, The first learning step and the second learning step are repeatedly performed, The first learning data set includes the first data, the first correction data, and training data indicating true among true and false, The second learning data set includes the first data, the second correction data, and training data indicating false among true and false, The first correction data is data obtained by performing a correction process on the first data, The second correction data is data obtained by performing a correction process on the second data, The second data is different from the first data, The first data and the second data are characterized in that they are data obtained by measuring light transmitted through an object, or data generated by performing a forward propagation operation on an object model that models the three-dimensional optical characteristics of the object. Effect of the Invention

[0015] According to the present invention, it is possible to provide a data processing method, a data processing device, a three-dimensional observation device, and a recording medium for generating an index indicating the reliability of an estimation, and a learning method, a learning device, and a recording medium for generating an index indicating the reliability of an estimation. [Brief description of the drawings]

[0016] [Figure 1] FIG. 2 is a diagram showing how an optical image of an object is acquired. [Diagram 2] 4 is a flowchart of a data processing method according to the first embodiment. [Diagram 3] FIG. 2 is a diagram illustrating a data processing method according to the first embodiment. [Figure 4] FIG. 13 is a diagram for explaining a forward propagation operation. [Diagram 5] FIG. 1 is a diagram showing a first production method. [Figure 6] FIG. 13 is a diagram showing learning data. [Figure 7] FIG. 13 is a diagram showing a method for creating reconstruction data. [Figure 8] FIG. 13 is a diagram showing a second production method. [Figure 9] FIG. 13 is a diagram showing learning data. [Figure 10] FIG. 13 is a diagram showing a third production method. [Figure 11] FIG. 13 is a diagram showing learning data. [Figure 12] FIG. 1 is a diagram showing the learning process in a DNN. [Figure 13] FIG. 1 is a diagram illustrating U-Net. [Figure 14] 10 is a flowchart of a data processing method according to a second embodiment. [Figure 15] FIG. 11 is a diagram illustrating a data processing method according to the second embodiment. [Figure 16] FIG. 1 is a diagram showing the learning process in a DNN. [Figure 17] FIG. 13 is a diagram illustrating a method for calculating reliability. [Figure 18] FIG. 13 is a diagram showing the process of calculating reliability in the second DNN. [Figure 19] FIG. 13 is a diagram showing learning data. [Figure 20] FIG. 1 is a diagram showing the learning process in the second DNN. [Figure 21] FIG. 1 is a diagram illustrating a first presentation method. [Figure 22] FIG. 11 is a diagram showing a second presentation method. [Figure 23] FIG. 1 is a diagram illustrating a data processing device according to an embodiment of the present invention. [Figure 24]FIG. 1 is a diagram showing a three-dimensional observation apparatus according to a first embodiment. [Diagram 25] FIG. 13 is a diagram showing a three-dimensional observation apparatus according to a second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] Prior to the description of the examples, the effects of an embodiment according to a certain aspect of the present invention will be described. In addition, when specifically explaining the effects of the present embodiment, specific examples will be shown. However, as in the case of the examples described later, the exemplified aspects are merely a part of the aspects included in the present invention, and there are many variations in the aspects. Therefore, the present invention is not limited to the exemplified aspects.

[0018] When an object is small, it is difficult to directly know the three-dimensional optical characteristics of the object. In this case, the three-dimensional optical characteristics can be known, for example, by estimation. In the data processing method of this embodiment, a neural network is used to estimate the three-dimensional optical characteristics of the object. The three-dimensional optical characteristics are a refractive index distribution or an absorptance distribution.

[0019] In the following, a case where a Deep Neural Network (hereinafter, referred to as "DNN") is used as the neural network will be described. A neural network has an input layer, an output layer, and one hidden layer. In contrast, a DNN has an input layer, an output layer, and multiple hidden layers.

[0020] When an object is small, an optical system can be used to obtain an optical image of the object. The optical image of the object reflects the three-dimensional optical characteristics. It is preferable that the optical image of the object is an enlarged image. If the image is an enlarged image, the data required to estimate the three-dimensional optical characteristics can be easily obtained from the captured image.

[0021] FIG. 1 is a diagram showing how an optical image of an object is acquired. FIG. 1(a) is a diagram showing a specific example of an image acquisition method. FIG. 1(b) is a diagram showing images of a plurality of XY cross sections. FIG. 1(c) is a diagram showing an image of an XZ cross section. For ease of explanation, the object is drawn large.

[0022] The Z-axis is an axis parallel to the optical axis AX. The X-axis is an axis perpendicular to the optical axis AX. The Y-axis is an axis perpendicular to the X-axis and the Z-axis.

[0023] 1(a), an object 1 is irradiated with illumination light 2. The object 1 is illuminated with the illumination light 2. The light transmitted through the object 1 is incident on an imaging unit 3. The imaging unit 3 has an optical system 4 and an imaging element 5.

[0024] In the optical system 4, an optical image IM is formed on an image plane IP. The optical image IM is an enlarged image of the object 1. The position of the image plane IP is conjugate with the position Zfo. Therefore, the optical image IM is an optical image of the object 1 at the position Zfo. The position Zfo is the focal position of the optical system 4.

[0025] The imaging surface of the imaging element 5 is located on the imaging plane IP. An image Imea(x, y) can be obtained by capturing an optical image IM with the imaging element 5. The image Imea(x, y) is an image of the optical image IM, and is an image on an XY cross section.

[0026] The thickness of the object 1 is large. In order to acquire an optical image of the entire object 1, an image Imea(x,y) can be acquired while moving the position Zfo between positions Z1 and Z2. As a result, as shown in FIG. 1(b), a plurality of images Imea(x,y) can be acquired.

[0027] To acquire multiple images Imea(x,y), the object 1 and the imaging unit 3 can be moved relatively in the Z-axis direction. To move the object 1, the object 1 can be held on a stage and the stage can be moved in the Z-axis direction. Alternatively, the optical system 4 can be formed of an infinity corrected objective lens and an imaging lens, and only the infinity corrected objective lens can be moved in the Z-axis direction.

[0028] The data required to estimate the three-dimensional optical properties can be obtained from multiple images Imea(x,y). Also, as shown in Figure 1(c), the XZ cross-sectional image Imea(x,z) can be obtained from multiple images Imea(x,y).

[0029] The data processing method of this embodiment will be explained using the data processing method of the first embodiment and the data processing method of the second embodiment.

[0030] The data processing method of the first embodiment includes an input step of inputting measurement data into a neural network, an estimation step of generating estimated data from the measurement data, a restoration step of generating restored data from the estimated data, and a calculation step of calculating the reliability of the estimated data based on the measurement data and the restored data.

[0031] The neural network is a trained model, the measurement data is data obtained by measuring light transmitted through an object, and the estimated data is data on the three-dimensional optical properties of the object estimated from the measurement data, and the three-dimensional optical properties are a refractive index distribution or an absorptivity distribution.

[0032] The estimation uses a neural network, and the restoration involves performing a forward propagation operation on the estimated data. In the forward propagation operation, the wavefront passing through the inside of the object estimated from the measurement data is sequentially obtained in the direction in which the light travels.

[0033] Fig. 2 is a flowchart of the data processing method of the first embodiment. Fig. 3 is a diagram showing the data processing method of the first embodiment. The image shown in Fig. 3 is an image of an XZ cross section. In Fig. 3, for ease of viewing, the object is shown as an image O(x,z). Description will be made with reference to Figs. 2 and 3.

[0034] The data processing method of the first embodiment includes steps S1, S2, S3, and S4.

[0035] Step S1 is an input step. In step S1, measurement data is input to the neural network. The measurement data is data necessary for estimating three-dimensional optical properties, and is obtained by measuring light transmitted through an object. An optical image of an object is formed by the light transmitted through the object. Therefore, the measurement data can be obtained from an image of the optical image of the object.

[0036] As shown in Fig. 3, an optical image of an object is formed by an optical system 4. By capturing the optical image of the object, an image of the optical image of the object is obtained. Measurement data is obtained from the image of the optical image of the object. An image Imea(x,z) is an image representing the measurement data.

[0037] In the data processing method of the first embodiment, a neural network is used. This neural network is a trained model. The trained model is a trained neural network. The DNN is also a trained DNN.

[0038] Neural networks use parameters such as weights and biases. Trained models use trained parameters. Since the parameters are optimally set, accurate estimation can be performed.

[0039] As described above, in step S1, the measurement data is input to the neural network. In Fig. 3, the measurement data is input to the DNN. After step S1 is completed, step S2 is executed.

[0040] Step S2 is an estimation step. In step S2, estimated data is generated from the measurement data. The estimated data is data on the three-dimensional optical characteristics of the object estimated from the measurement data.

[0041] When an object is thick, an optical image of the object is formed by light from a focused position and light from a defocused position. A clear optical image is formed by light from a focused position, but a clear optical image is not formed by light from a defocused position. A clear optical image is not formed because a unclear optical image is superimposed on a clear optical image. As a result, a clear image cannot be obtained.

[0042] The thickness of the object shown in Fig. 3 is as thick as that of the object 1 shown in Fig. 1(a). Therefore, the optical image of the object is not clear. Since the optical image is not clear, the image Imea(x,z) is also not clear.

[0043] As shown in Figure 3, the measured image Imea(x,z) is significantly different from the image O(x,z) of the object being measured. It is difficult for an observer to infer the image O(x,z) by looking at the image Imea(x,z).

[0044] As shown in Figure 3, the DNN calculates the estimated data from the measured data. The image Oest(x,z) is an image that represents the estimated data. By looking at the image Oest(x,z), the observer can infer the image O(x,z) of the observed object.

[0045] Images estimated by DNNs are heavily influenced by the training data. If the training data does not contain an image O(x,z) close to the observed object and its corresponding image Imea(x,z), it is not possible to generate highly accurate estimated data Oest(x,z). The generated estimated data Oest(x,z) is likely to generate an image close to the image O(x,z) contained in the training data. It is not possible to determine whether the generated estimated data image Oest(x,z) represents the image O(x,z) of the observed object.

[0046] Step S3 is a restoration step. In step S3, restored data is generated from the estimated data. In the restoration, a forward propagation operation is performed on the estimated data. In the forward propagation operation, the wavefront passing through the inside of the estimated object is sequentially obtained in the direction in which light travels.

[0047] The measurement data is obtained from an optical image of an object. The estimated data is data on the estimated three-dimensional optical properties of the object. Therefore, if the optical image can be calculated from the estimated data, data similar to the measurement data (hereinafter referred to as "reconstructed data") can be obtained.

[0048] If the estimated data is data of the estimated object, the reconstructed data is data obtained from an optical image of the estimated object (hereinafter referred to as the "estimated optical image"). The measured data is data obtained from an image of the optical image of the object. Since both the measured data and the reconstructed data are data obtained from optical images, it is possible to compare the measured data with the reconstructed data. By comparing the measured data with the reconstructed data, it is possible to determine whether the estimated data correctly represents the three-dimensional optical characteristics of the object.

[0049] To obtain restored data from the estimated data, a forward propagation operation is performed. Various types of operations are performed in the forward propagation operation. The wavefront passing through the inside of an object can be calculated, for example, by the beam propagation method. In the beam propagation method, the estimated object is replaced with multiple thin layers. Then, the change in the wavefront as the light passes through each layer is calculated sequentially.

[0050] Fig. 4 is a diagram for explaining the forward propagation calculation, Fig. 4(a) is a diagram for explaining the beam propagation method, and Fig. 4(b) is a diagram for explaining the processing in the forward propagation calculation.

[0051] The propagation of the wavefront is Fresnel propagation. We will explain this by replacing the wavefront with an electric field. In Figure 4(a), the solid line represents the complex refractive index of the object. The dotted line represents the electric field (the scalar field of light). The arrows represent Fresnel propagation. The beam propagation method is a technique for repeatedly calculating Fresnel propagation between thin layers.

[0052] The beam propagation method will be described. At position P1, the electric field Eout1 on the exit side is calculated using the electric field Ein1 on the entrance side and the complex refractive index N1.

[0053] From the electric field Eout1, the electric field Ein2 on the incident side at position P2 is calculated. Propagation from the electric field Eout1 to the electric field Ein2 is Fresnel propagation. The electric field Eout2 on the exit side can be calculated using the electric field Ein2 and the complex refractive index N2. This propagation that repeatedly repeats Fresnel propagation while reflecting the complex refractive index of the object is beam propagation.

[0054] In this way, the electric field on the exit side can be calculated from the electric field on the entrance side by using the beam propagation method. As a result, the electric field Eout on the exit side at the position PN can be obtained. When the position of the wavefront exiting from the estimated object is the position PN, the electric field Eout represents the wavefront exiting from the estimated object.

[0055] The forward propagation calculation will now be described. In the forward propagation calculation, an estimated optical image is obtained from an estimated object. The estimated optical image corresponds to an optical image of the object. During measurement, an imaging optical system is used as shown in FIG. 4(b). By measuring the electric field intensity distribution at the imaging position 13 of the imaging lens 11 while scanning the objective lens 10 in the optical axis direction, an image Imea(x,z) of the measurement data can be obtained.

[0056] Since the estimated object Oest(x,z) of the observed object O(x,z) is obtained, if the incident wavefront Win is known, the exit wavefront Wout propagating through the object can be obtained from the estimated object Oest(x,z) and the incident wavefront Win by the beam propagation method. From the exit wavefront Wout, the electric fields at each of the focal positions Fo1 and Fo2 when the objective lens 10 is scanned in the optical axis direction can be obtained by Fresnel propagation. If the aberration of the objective lens 10 and the imaging lens 11 is small, the intensity distribution of these electric fields corresponds to the image Imea(x,z) of the measurement data. The image Iest(x,z) of the intensity distribution obtained by such forward propagation calculation is an image representing the restored data.

[0057] After step S3 is completed, step S4 is executed.

[0058] Step S4 is a calculation step in which the reliability of the estimated data is calculated based on the measured data and the restored data.

[0059] As mentioned above, since both the measured data and the reconstructed data are obtained from optical images, it is possible to compare the measured data with the reconstructed data. By comparing the measured data with the reconstructed data, it is possible to determine whether the estimated data correctly represents the 3D optical properties of the object.

[0060] For example, if an object with a shape not included in the training data is measured, the measured optical image will also be an image not included in the training data. However, the estimated image of the object generated by the DNN from this optical image is likely to be similar to the estimated image of the object included in the training data.

[0061] Therefore, the optical image as restored data obtained by forward propagation calculation using the estimated image of the estimated object is also an optical image included in the training data. The measured optical image is not included in the training data, but the optical image of the restored data is included in the training data. Since the measured optical image and the restored optical image are different, in this case it can be determined that the estimated image of the object generated by the DNN does not correctly represent the 3D optical properties of the measured object.

[0062] The smaller the deviation of the restored data from the measured data, the closer the estimated 3D optical properties are to the 3D optical properties of the object. For example, if the degree of deviation of the restored data from the measured data is taken as the reliability, the reliability of the estimated data can be calculated based on the measured data and the restored data.

[0063] In the data processing method of the first embodiment, the three-dimensional optical characteristics of an object are calculated using a trained model. Therefore, the three-dimensional optical characteristics of the object can be calculated in a short time. Furthermore, the reliability of the calculated three-dimensional optical characteristics of the object is calculated. Therefore, it is possible to determine whether the estimated three-dimensional optical characteristics correctly represent the three-dimensional optical characteristics of the object based on the reliability.

[0064] To use a trained model, it is necessary to train the neural network in advance. Training data is necessary to train the model. A method for creating the training data will be described. The training data includes training input data and training output data.

[0065] (First creation method) FIG. 5 is a diagram showing a first creation method. FIG. 5(a) is a diagram showing a processing flow in the first creation method. FIG. 5(b) is a diagram showing an image of the reconstructed data. The images shown in FIG. 5(a) and FIG. 5(b) are images of the XZ cross section. In FIG. 5(a), for ease of viewing, the object is shown as an image O(x,z).

[0066] In the first generation method, a reconstruction operation is performed, in which reconstruction data is generated from learning input data.

[0067] As mentioned above, when an object is very small, it is difficult to directly obtain the values ​​of the three-dimensional optical properties of the object (hereinafter referred to as "optical property values"). Therefore, an optical image of the object is formed by the optical system 4. First comparison data is obtained from the optical image. Image SC1(x,z) represents an image of the first comparison data. The first comparison data can be used as learning input data. Image SC1(x,z) represents an image of the learning input data.

[0068] In the reconstruction calculation, an object model is used. Since the optical characteristic values ​​are unknown, the values ​​in the data of the object model (hereinafter referred to as "model data") are estimated values.

[0069] At the start of the reconstruction operation, the estimated values ​​are set to initial values. Any value may be used for the estimated values. The image SM(x,z) represents the image of the model data when the estimated values ​​are set to the initial values.

[0070] In the reconstruction operation, a forward propagation operation is performed using the model data. An optical image of the object model is obtained by the forward propagation operation. Second comparison data is obtained from the optical image of the object model. An image SC2(x,z) represents an image of the second comparison data.

[0071] If the first comparison data and the second comparison data are the same, the optical image of the object and the optical image of the object model are the same. In this case, the estimated value is the same as the optical property value. If the first comparison data and the second comparison data are different, the optical image of the object and the optical image of the object model are different. In this case, the estimated value is different from the optical property value.

[0072] If the first comparison data and the second comparison data are different, the wavefront forming the optical image of the object model (hereinafter referred to as the "estimated wavefront") is different from the wavefront forming the optical image of the object. Therefore, for example, the estimated wavefront is corrected using the difference between the first comparison data and the second comparison data.

[0073] A backpropagation operation is performed using the corrected estimated wavefront. A new estimated value is obtained by the backpropagation operation. The values ​​of the model data are replaced with the new estimated value. In other words, the estimated value is updated.

[0074] The forward propagation calculation, correction of the estimated wavefront, back propagation calculation, and update of the estimated value are repeated until the difference between the first comparison data and the second comparison data becomes smaller than the threshold value.

[0075] The model data when the difference between the first comparison data and the second comparison data is smaller than the threshold value is set as reconstructed data. The image SO(x,z) shown in FIG. 5(b) represents the image of the reconstructed data. The reconstructed data can be used as learning output data.

[0076] Fig. 6 is a diagram showing learning data. The image shown in Fig. 6 is an image of an XZ cross section.

[0077] By performing the reconstruction operation, reconstruction data corresponding to the object, i.e., learning output data, can be obtained. A large amount of learning data is required to perform the learning. Therefore, a large amount of learning output data is also required. In order to obtain a large amount of learning output data, it is sufficient to perform the reconstruction operation for a large number of objects.

[0078] Let the number of objects be N. In Fig. 6, images of the first comparison data and images of the reconstructed data are shown for object 1, object 2, and object N. Object 1, object 2, and object N are illuminated with the same illumination angle, e.g., 0°.

[0079] In Fig. 6, the image of the object is not shown. O1(x,z), O2(x,z) and ON(x,z) are shown to show the correspondence between the object and the first comparison data image and the correspondence between the object and the reconstructed data. O1(x,z) represents object 1. O2(x,z) represents object 2. ON(x,z) represents object N.

[0080] Image SC11(x,z) represents an image of the first comparison data for object 1. Image SC12(x,z) represents an image of the first comparison data for object 2. Image SC1N(x,z) represents an image of the first comparison data for object N.

[0081] Image SO1(x,z) represents an image of the reconstruction data of object 1. Image SO2(x,z) represents an image of the reconstruction data of object 2. Image SON(x,z) represents an image of the reconstruction data of object N.

[0082] In the first creation method, the first comparison data is used as learning input data, and the reconstructed data is used as learning output data. When the number of objects is N, the number of learning input data and the number of learning output data are also N.

[0083] In the reconstruction calculation, the more first comparison data there is for one object, the more accurate the reconstruction data will be. To increase the first comparison data for one object, for example, the object may be illuminated at multiple illumination angles and the first comparison data may be obtained from the optical images at each illumination angle.

[0084] Fig. 7 is a diagram showing a method for creating reconstruction data. The image shown in Fig. 7 is an image of an XZ cross section.

[0085] First comparison data 1 is data obtained from an optical image of an object when the illumination angle θILL is 0°. Image SCθ1(x,z) represents the image of first comparison data 1. First comparison data 2 is data obtained from an optical image of an object when the illumination angle θILL is -40°. Image SCθ2(x,z) represents the image of first comparison data 2.

[0086] The first comparison data 3 is data obtained from an optical image of the object when the illumination angle θILL is -30°. The image SCθ3(x,z) represents the image of the first comparison data 3. The first comparison data N is data obtained from an optical image of the object when the illumination angle θILL is 40°. The image SCθN(x,z) represents the image of the first comparison data N.

[0087] The first comparison data 1, the first comparison data 2, the first comparison data 3, and the first comparison data N have different illumination angles. Therefore, the first comparison data 1, the first comparison data 2, the first comparison data 3, and the first comparison data N have different information regarding optical characteristic values. By using the first comparison data 1, the first comparison data 2, the first comparison data 3, and the first comparison data N in the reconstruction calculation, the accuracy of the reconstruction data can be improved.

[0088] (Second creation method) Fig. 8 is a diagram showing the second creation method. The image shown in Fig. 8 is an image of the XZ cross section.

[0089] As described in FIG. 6, a large amount of learning data is necessary for learning. In the first creation method, forward propagation calculations and back propagation calculations are repeated to obtain reconstructed data (learning output data). The forward propagation calculations and back propagation calculations take a lot of time. If it takes a long time to obtain one piece of reconstructed data, an enormous amount of time is required to obtain a large number of reconstructed data.

[0090] In the second creation method, one reconstructed data is transformed. In this way, a large number of reconstructed data, i.e., a large number of learning output data, can be obtained in a short time. Data transformation can be performed, for example, by enlarging, reducing, rotating, or adding noise.

[0091] The learning requires learning input data that is paired with the learning output data. The learning input data is the first comparison data. The first comparison data is obtained from an optical image of the object. However, in the second creation method, the deformed reconstructed data corresponds to the object.

[0092] Therefore, the deformed reconstructed data is regarded as an object, and a forward propagation operation is performed using the deformed reconstructed data, whereby data corresponding to an optical image of the object is obtained from the deformed reconstructed data.

[0093] In Fig. 5(a), the second comparison data is obtained from the model data by performing a forward propagation operation. Data corresponding to an optical image of an object corresponds to the second comparison data. In the second creation method, the second comparison data is used as learning input data.

[0094] The reconstructed data that has been enlarged is referred to as enlarged reconstructed data, the reconstructed data that has been reduced is referred to as reduced reconstructed data, and the reconstructed data that has been rotated is referred to as rotated reconstructed data.

[0095] Image SO1(x,z) represents an image of the enlarged reconstruction data, image SO2(x,z) represents an image of the reduced reconstruction data, and image SON(x,z) represents an image of the rotated reconstruction data.

[0096] Image SC21(x,z) represents an image of the second comparison data obtained from the enlarged reconstruction data. Image SC22(x,z) represents an image of the second comparison data obtained from the reduced reconstruction data. Image SC2N(x,z) represents an image of the second comparison data from the rotated reconstruction data.

[0097] Fig. 9 is a diagram showing learning data. The image shown in Fig. 9 is an image of an XZ cross section.

[0098] 9 shows images of the second comparison data and images of the transformed reconstructed data for object 1 and object N. The transformed reconstructed data is data obtained by transforming the reconstructed data. Because there are many images, not all of the images will be described.

[0099] In Fig. 9, the image of the object is not shown. O1(x,z) and ON(x,z) are shown to show the correspondence between the object and the second comparison data image and the correspondence between the object and the deformation reconstruction data. O1(x,z) represents object 1. ON(x,z) represents object N.

[0100] Image SC211(x,z) represents an image of the second comparison data obtained from the deformation reconstruction data of object 1. Image SC21N(x,z) represents an image of the second comparison data obtained from the deformation reconstruction data of object N.

[0101] Image SO11(x,z) represents an image of the deformation reconstruction data of object 1. Image SON1(x,z) represents an image of the deformation reconstruction data of object N.

[0102] In the second creation method, the second comparison data is used as learning input data, and the transformed and reconstructed data is used as learning output data. When the number of objects is N, the number of learning input data and the number of learning output data are equal to or greater than N.

[0103] (Third creation method) Fig. 10 is a diagram showing the third creation method. The image shown in Fig. 10 is an image of the XZ cross section.

[0104] In the first creation method, first comparison data (learning input data) and reconstructed data (learning output data) are generated from an object. In the second creation method, reconstructed data is generated from the object, and transformed data (learning input data) and second comparison data (learning input data) are obtained from the reconstructed data. In this way, an object is used in the first creation method and the second creation method.

[0105] In contrast, the third generation method does not use an object, but uses data generated by a computer (hereinafter referred to as "virtual object data") instead of an object.

[0106] The virtual object data corresponds to the reconstructed data. Therefore, as in the second creation method, a forward propagation operation is performed using the virtual object data. In this way, data corresponding to an optical image of the object is obtained from the virtual object data.

[0107] In Fig. 5(a), the second comparison data is obtained from the model data by performing a forward propagation operation. Data corresponding to an optical image of an object corresponds to the second comparison data. In the third generation method, the second comparison data is used as learning input data.

[0108] Image O'1(x,z) represents an image of data for virtual object 1. Image O'2(x,z) represents an image of data for virtual object 2. Image O'N(x,z) represents an image of data for virtual object N.

[0109] Image SC21(x,z) represents an image of the second comparison data obtained from the data of virtual object 1. Image SC22(x,z) represents an image of the second comparison data obtained from the data of virtual object 2. Image SC2N(x,z) represents an image of the second comparison data of the data of virtual object N.

[0110] Fig. 11 is a diagram showing learning data. The image shown in Fig. 11 is an image of an XZ cross section.

[0111] Image SC21(x,z) represents an image of the second comparison data obtained from the data of virtual object 1. Image SC22(x,z) represents an image of the second comparison data obtained from the data of virtual object 2. Image SC2N(x,z) represents an image of the second comparison data obtained from the data of virtual object N.

[0112] Image O'1(x,z) represents an image of data for virtual object 1. Image O'2(x,z) represents an image of data for virtual object 2. Image O'N(x,z) represents an image of data for virtual object N.

[0113] In the third creation method, the second comparison data is used as learning input data, and the data of the virtual objects is used as learning output data. When the number of virtual objects is N, the number of learning input data and the number of learning output data are N. However, the virtual objects can be created easily in a short time, and there is no limit to the number that can be created.

[0114] (Learning in DNN) Fig. 12 is a diagram showing the state of learning in a DNN. The image shown in Fig. 12 is an image of an XZ cross section.

[0115] A training data set is used for training a DNN. The training data set includes training input data and training output data.

[0116] As the learning input data, the first comparison data in the first creation method, the second comparison data in the second creation method, or the second comparison data in the third creation method is used.

[0117] As the learning output data, the reconstructed data in the first creation method, the transformed and reconstructed data in the second creation method, or the virtual object data in the third creation method is used.

[0118] In learning, learning estimation data is output from learning input data. An ideal state is one in which the learning estimation data matches the learning output data. However, in an insufficient learning state, the learning estimation data does not match the learning output data. The difference between the ideal state and an insufficient learning state can be expressed by a loss function. An insufficient learning state also includes a state in which no learning has been performed.

[0119] Various parameters are used in estimation in DNN. In learning, a loss function is used to search for optimal parameters. In searching for optimal parameters, the parameters that minimize the value of the loss function are searched for.

[0120] In learning, the learning estimation data is repeatedly compared with the learning output data. Each time the learning estimation data is compared with the learning output data, the output of the loss function is fed back to the DNN. By repeating the comparison of the learning estimation data with the learning output data and the feedback of the output of the loss function to the DNN, the optimal parameters can be found.

[0121] In the process of outputting the learning estimation data from the learning input data, image region detection is performed. For example, U-Net can be used for the image region detection.

[0122] FIG. 13 is a diagram showing U-Net. FIG. 13(a) is a diagram showing the configuration of U-Net. FIG. 13(b) is a schematic diagram of the data shape. In FIG. 13(b), "64" represents the number of channels or the number of feature maps. "256^3" represents the number of pixels.

[0123] The process in the direction of arrow A is called encoding or downsampling, and the process in the direction of arrow B is called decoding or upsampling.

[0124] In encoding, features of the input data are extracted by alternating convolution and pooling. As the process proceeds in the direction of arrow A, the area with features is subdivided. Therefore, the number of pixels decreases, but the number of channels or feature maps increases.

[0125] Decoding involves deconvolution to create output data of the same size as the input data. The output data is divided into regions with the same characteristics. As a result, the image in the output data is coarser than that in the input data.

[0126] During encoding, position information of characteristic areas is obtained. Therefore, during decoding, the position information obtained during encoding is used to create output data. In Figure 13(a), the transfer of position information from encoding to decoding is represented by "Copy."

[0127] The data processing method of the second embodiment includes an input step of inputting measurement data into a neural network, an estimation step of generating estimated data from the measurement data, a restoration step of generating reconstructed data from the estimated data, a calculation step of calculating the reliability of the estimated data based on the measurement data and the reconstructed data, and a learning step in which the neural network learns using an amount inversely proportional to the reliability as a loss.

[0128] The measurement data is data obtained by measuring light transmitted through an object, and the estimated data is data on the three-dimensional optical properties of the object estimated from the measurement data, where the three-dimensional optical properties are a refractive index distribution or an absorptivity distribution.

[0129] The estimation uses a neural network, and the restoration involves performing a forward propagation operation on the estimated data. In the forward propagation operation, the wavefront passing through the inside of the object estimated from the measurement data is sequentially obtained in the direction in which the light travels.

[0130] FIG. 14 is a flowchart of the data processing method of the second embodiment. Steps that are the same as those in FIG. 2 are given the same numbers, and explanations are omitted. FIG. 15 is a diagram showing the data processing method of the second embodiment. The image shown in FIG. 15 is an image of an XZ cross section. The same configurations as those in FIG. 3 are given the same numbers and terms, and explanations are omitted. The explanation will be given with reference to FIG. 14 and FIG. 15.

[0131] The data processing method of the second embodiment includes steps S1, S2, S3, S4, and S5.

[0132] Step S5 is an output step. In step S5, the neural network learns an amount inversely proportional to the reliability as a loss.

[0133] In the data processing method of the second embodiment, the neural network is not a trained model, so optimal parameters are searched for by performing training.

[0134] In step S5, a quantity inversely proportional to the reliability is obtained from a comparison between the measured data and the restored data. As shown in FIG. 15, the quantity inversely proportional to the reliability is input to the DNN. The quantity inversely proportional to the reliability indicates the degree of deviation of the restored data from the measured data. By inputting the quantity inversely proportional to the reliability to the DNN, learning can be performed in the DNN. As a result, the reliability can be calculated with higher accuracy.

[0135] In addition, by comparing the measured data with the restored data, it is possible to determine whether the estimated data correctly represents the 3D optical properties of the object. As learning in the DNN progresses, the estimated data can obtain more accurate 3D optical properties of the object.

[0136] In the data processing method of the second embodiment, a learning mode and an estimation mode are switched. In the learning mode, an amount inversely proportional to the reliability is input to the DNN. In the estimation mode, the reliability is output.

[0137] Fig. 16 is a diagram showing the state of learning in a DNN. The image shown in Fig. 16 is an image of an XZ cross section.

[0138] Fig. 16 shows how learning is done using multiple measured data. The process of comparison minimization gives a quantity that is inversely proportional to the confidence.

[0139] The more data used for learning, the more accurate the reliability can be calculated, resulting in more accurate 3D optical properties of objects in the estimated data.

[0140] In the data processing method of this embodiment, it is preferable to calculate the difference between the measurement data and the restored data, and calculate the reliability based on the difference.

[0141] The reliability can be calculated by finding the sum of absolute values ​​of the differences between corresponding data in the measured data and the restored data, and using the sum of absolute values ​​as an index. For example, the reliability can be calculated by multiplying the sum of absolute values ​​by a coefficient that is inversely proportional to the sum of absolute values. The reliability can be calculated by comparing with a preset threshold value. The threshold value can be found in advance by experiment. The sum of absolute values ​​can be normalized and compared with the threshold value.

[0142] The reliability can be easily calculated by using the difference between the measured data and the reconstructed data.

[0143] In the data processing method of this embodiment, it is preferable to calculate the correlation between the measurement data and the restored data, and calculate the reliability based on the correlation.

[0144] The correlation peak is expressed by the following formula. er itself may be used as the reliability, or er may be multiplied by a coefficient to obtain the reliability. The reliability may also be calculated by comparing er with a preset threshold. For example, if er is greater than a preset threshold of 0.5, the reliability may be calculated as 1.0 (a value indicating high reliability).

number

[0145] The correlation function corr() between the measured data and the restored data is expressed by the following equation.

number

[0146] By using the correlation between the measurement data and the restored data, the reliability can be calculated with high accuracy.

[0147] As explained above, the correlation peak can be used to calculate the reliability. However, the spread of the correlation, specifically, the spread around the peak, can also be used to calculate the reliability. The smaller the spread, the higher the reliability. For example, the area around the corr() peak is fitted with the following formula to find px, py, pz, and s. When s is small, the reliability is high.

number

[0148] In the data processing method of this embodiment, it is preferable to calculate the sum of squares of the difference between the measurement data and the restored data, and calculate the reliability based on the magnitude of the sum of squares of the difference.

[0149] For example, the reliability can be calculated by multiplying the sum of squares of the difference between the measured data and the restored data by a coefficient. The reliability can be calculated by comparing with a preset threshold. The threshold can be obtained in advance by experiment. The sum of absolute values ​​can be normalized and compared with the threshold.

[0150] The sum of squares of the difference between the measured data and the restored data is expressed by the following formula.

number

[0151] By using the sum of squares of the differences between the measured data and the restored data, the reliability can be calculated with high accuracy.

[0152] The data processing method of this embodiment has a first neural network and a second neural network, the first neural network is a neural network, the second neural network is a trained model, and it is preferable to calculate the reliability using the second neural network.

[0153] Fig. 17 is a diagram showing a method for calculating the reliability. The same components as those in Fig. 3 are given the same numbers and terms, and the explanation will be omitted.

[0154] The data processing method of the present embodiment includes a first neural network and a second neural network. The first neural network is the neural network in the data processing method of the first embodiment or the neural network in the data processing method of the second embodiment.

[0155] The second neural network is a trained model. The second neural network is used to calculate the reliability.

[0156] The case where the first neural network and the second neural network are used will be described. The first DNN is the first neural network. The second DNN is the second neural network. The first DNN is the same as the DNN shown in Figure 3, so the description will be omitted.

[0157] The second DNN is a trained model. Measurement data and restored data are input to the second DNN. The second DNN calculates the reliability from the measurement data and restored data.

[0158] 18 is a diagram showing a process of calculating the reliability in the second DNN. The second DNN has a CNN, an FC, and a softmax.

[0159] CNN is a Convolutional Neural Network. A Convolution Neural Network performs convolution and pooling. Convolution detects the shading patterns in an image and extracts the features of an object. Pooling performs processing to consider an object to be the same even if its position changes. Features are extracted from an image by convolution and pooling.

[0160] FC stands for Full Connection. In Full Connection, feature variables are output from image data from which feature parts have been extracted. In softmax, a value between 0 and 1 is calculated based on the feature variables. The calculated value represents a probability, so the calculated value is used as the reliability.

[0161] The first DNN is the DNN shown in FIG. 3. Therefore, the first DNN calculates estimated data. In contrast, the second DNN calculates reliability. Therefore, learning in the second DNN is different from learning in the first DNN. Learning in the second DNN will be described.

[0162] In the data processing method of this embodiment, it is preferable that the second neural network performs training using the first training data set group and the second training data set group.

[0163] The first learning dataset group has a plurality of first learning datasets, and the first learning datasets include first data, first corrected data, and teacher data indicating true among true and false.

[0164] The second learning dataset group has a plurality of second learning datasets, and the second learning datasets include the first data, the second corrected data, and teacher data indicating false among true and false.

[0165] The first corrected data is data obtained by performing a correction process on the first data, and the second corrected data is data obtained by performing a correction process on the second data, and the second data is different from the first data.

[0166] The first data and the second data are data obtained by measuring light transmitted through an object, or data generated by performing a forward propagation operation on an object model that models the three-dimensional optical characteristics of the object.

[0167] Fig. 19 is a diagram showing learning data, Fig. 19(a) is a diagram showing generation of learning data, and Fig. 19(b) is a diagram showing the relationship between learning input data and learning output data.

[0168] The learning data includes a plurality of data groups. Data group A, data group B, and data group Z are illustrated in FIG. 19(a).

[0169] As shown in Fig. 19(a), the data group includes basic data and modified data. The basic data in one data group is different from the basic data in another data group. When an image is generated using the data, the image in the basic data of data group A is different from the image in the basic data of data group B and the image in the basic data of data group Z.

[0170] The modified data is data obtained by modifying the basic data, for example, by enlarging, reducing, rotating, or adding noise.

[0171] The number of transformed data may be one or more. Fig. 19(a) shows basic data, first transformed data, second transformed data, and Nth transformed data. In the first transformed data, noise is added to the basic data. In the second transformed data, the basic data is rotated. In the Nth transformed data, the basic data is enlarged or reduced.

[0172] For data group A, basic data A, first transformed data a1, second transformed data a2, and Nth transformed data an are illustrated. For data group B, basic data B, first transformed data b1, second transformed data b2, and Nth transformed data bn are illustrated. For data group Z, basic data Z, first transformed data z1, second transformed data z2, and Nth transformed data zn are illustrated.

[0173] As shown in Fig. 18, the reliability is calculated using the measured data and the restored data. In learning in the second neural network, data corresponding to the measured data (hereinafter referred to as "first corresponding data") and data corresponding to the restored data (hereinafter referred to as "second corresponding data") are selected from the learning data.

[0174] The first correspondence data is made of basic data. The second correspondence data is made of modified data. The first correspondence data corresponds to the above-mentioned first data. The second correspondence data corresponds to the above-mentioned first correction data or second correction data.

[0175] The first corresponding data and the second corresponding data can be selected from one data group. For example, from data group B, basic data B is selected as the first corresponding data, and second transformed data b2 is selected as the second corresponding data. The second transformed data b2 is data obtained by transforming basic data B. Therefore, the degree of similarity between the second transformed data b2 and basic data B is high.

[0176] In addition, the first corresponding data and the second corresponding data can be selected from two data groups. For example, basic data A is selected as the first corresponding data from data group A, and first transformed data b1 is selected as the second corresponding data from data group B. The first transformed data b1 is not data obtained by transforming the basic data A. Therefore, the degree of similarity between the first transformed data b1 and the basic data A is low.

[0177] In this way, when the data selected as the first corresponding data and the data selected as the second corresponding data are selected from the same data group, the degree of similarity between the two data is high. In contrast, when the data selected as the first corresponding data and the data selected as the second corresponding data are selected from different data groups, the degree of similarity between the two data is low.

[0178] The degree of similarity can be regarded as reliability. As shown in FIG. 19(b), in the second DNN, the combination of the first corresponding data and the second corresponding data is changed in various ways to create learning input data. The learning input data is input to the second DNN to obtain learning output data. The numerical value in the learning output data is the value of the softmax function, and can be used as a value representing reliability.

[0179] As shown by the dashed dotted lines, if the learning input data and the learning output data are considered as one set, two types of learning datasets are used in learning. The first learning dataset is dataset 1 shown in Figure 19(b). The second learning dataset is dataset 2 shown in Figure 19(b).

[0180] In the learning output data of Data Set 1, the value of True is 1.0 and the value of False is 0.0. If data that indicates the truth or falsity of the output result is considered as training data, then the learning output data of Data Set 1 is training data that indicates the truth of the truth or falsity.

[0181] In the learning output data of Data Set 2, the value of True is 0.0 and the value of False is 1.0. Therefore, the learning output data of Data Set 2 is training data that indicates False among True and False.

[0182] The first learning data set includes the first corresponding data, the second corresponding data, and training data indicating true among true and false. In the first learning data set, the data group to which the second corresponding data belongs is the same as the data group to which the first corresponding data belongs.

[0183] The second learning data set includes the first corresponding data, the second corresponding data, and training data indicating false among true and false. In the second learning data set, the data group to which the second corresponding data belongs is different from the data group to which the first corresponding data belongs.

[0184] As the first correspondence data, for example, the measurement data shown in FIG. 3, the first comparison data shown in FIG. 5, or the second comparison data shown in FIG. 5 can be used.

[0185] The first correspondence data may be data generated by generating an object model that models the three-dimensional optical characteristics of the object and performing a forward propagation operation on the generated object model. This data is the same type of data as the measurement data and corresponds to the measurement data.

[0186] (Learning in the second DNN) Fig. 20 is a diagram showing the state of learning in the second DNN. The image shown in Fig. 20 is an image of an XZ cross section.

[0187] In FIG. 20, the reliability can be used to judge whether two pieces of data are similar or not. Measured data is used as the base data. As described above, the measured data is data obtained by measuring light transmitted through an object. Since the measured data is used as the base data, the deformed data is also data obtained by deforming the measured data.

[0188] The learning in the second DNN is basically the same as the learning in the DNN shown in Fig. 12. Therefore, detailed explanation will be omitted. The difference between the learning in the second DNN and the learning in the DNN shown in Fig. 12 is as follows.

[0189] (I) Number of training input data In the second DNN, the number of pieces of training input data is 2. In the DNN shown in Figure 12, the number of pieces of training input data is 1. (II) The type of learning output data is the same as the type of learning input data In the second DNN, the type of learning output data is different from the type of learning input data. In the DNN shown in FIG. 12, the type of learning output data is the same as the type of learning input data. For example, in the second DNN, numerical values ​​are output from two images. In the DNN shown in FIG. 12, one image is output from one image.

[0190] It is preferable that the data processing method of the present embodiment further comprises a presentation step of presenting the reliability.

[0191] As described above, in the data processing method according to the embodiment, the reliability can be calculated, and therefore, various presentations can be made using the reliability.

[0192] In addition, various sounds can be generated using the reliability. For example, a sound can be generated only when the reliability is low. The reliability can be presented to the user by display or sound.

[0193] (First presentation method) Fig. 21 is a diagram showing a first presentation method. Fig. 21(a) is a flowchart of the first presentation method. Fig. 21(b) and Fig. 21(c) are diagrams showing presentation examples. The same steps as in Fig. 1 are given the same numbers, and descriptions thereof are omitted.

[0194] The first presentation method includes steps S5, S6, and S7. In step S5, the reliability value is compared with a threshold value. If the reliability value is greater than the threshold value, step S6 is executed. If the reliability value is equal to or greater than the threshold value, step S6 is executed. If the reliability value is less than the threshold value, step S7 is executed.

[0195] In step S6, "TRUE" is displayed. As shown in Fig. 21(b), the word "TRUE" can be displayed together with the image. In step S7, "FALSE" is displayed. As shown in Fig. 21(c), the word "FALSE" can be displayed together with the image.

[0196] The threshold value to be compared with the confidence value does not have to be one. For example, two threshold values ​​can be used to display the words "High Confidence", "Medium Confidence", and "Low Confidence".

[0197] Display may be performed only in either step S6 or step S7. Also, different colors may be displayed in steps S6 and S7. For example, blue may be displayed in step S6 and red may be displayed in step S7. LEDs of each color may be used.

[0198] (Second presentation method) Fig. 22 is a diagram showing the second presentation method. Fig. 22(a) is a diagram showing the presentation when the reliability is high. Fig. 22(B) is a diagram showing the presentation when the reliability is medium. Fig. 22(c) is a diagram showing the presentation when the reliability is low.

[0199] The second DNN outputs a numerical value. Therefore, a numerical value of the reliability can be obtained based on the numerical value output from the second DNN. As a result, the reliability can be expressed numerically as shown in FIG. 22(a), FIG. 22(b), and FIG. 22(c). The periphery of the image may be colored according to the reliability value. For example, if the reliability is medium, the periphery of the image is surrounded by yellow. If the reliability is low, the periphery of the image is surrounded by red.

[0200] The following text may also be displayed: (i) Medium confidence "The reconstruction results may be incorrect." (ii) When reliability is low "The reconstruction result is incorrect. Please measure again."

[0201] The data processing device of this embodiment will be described using the data processing device of the first embodiment and the data processing device of the second embodiment.

[0202] The data processing device of the first embodiment comprises a memory and a processor, the memory stores measurement data, and the processor executes an estimation process to estimate three-dimensional optical properties of an object, the three-dimensional optical properties being a refractive index distribution or an absorptivity distribution.

[0203] The estimation process includes an input step of inputting measurement data into a neural network, an estimation step of generating estimated data from the measurement data, a restoration step of generating restored data from the estimated data, and a calculation step of calculating the reliability of the estimated data based on the measurement data and the restored data.

[0204] The neural network is a trained model, the measurement data is data obtained by measuring light transmitted through an object, and the estimated data is data on the three-dimensional optical properties of the object estimated from the measurement data.

[0205] The estimation uses a neural network, and the restoration involves performing a forward propagation operation on the estimated data. In the forward propagation operation, the wavefront passing through the inside of the object estimated from the measurement data is sequentially obtained in the direction in which the light travels.

[0206] The data processing device of the second embodiment comprises a memory and a processor, the memory stores measurement data, and the processor executes an estimation process to estimate three-dimensional optical properties of an object, the three-dimensional optical properties being a refractive index distribution or an absorptivity distribution.

[0207] The estimation process includes an input step of inputting measurement data into a neural network, an estimation step of generating estimated data from the measurement data, a restoration step of generating restored data from the estimated data, a calculation step of calculating the reliability of the estimated data based on the measurement data and the restored data, and a learning step in which the neural network learns using an amount inversely proportional to the reliability as a loss.

[0208] The measured data is data obtained by measuring light transmitted through an object, and the estimated data is data on the three-dimensional optical properties of the object estimated from the measured data.

[0209] The estimation uses a neural network, and the restoration involves performing a forward propagation operation on the estimated data. In the forward propagation operation, the wavefront passing through the inside of the object estimated from the measurement data is sequentially obtained in the direction in which the light travels.

[0210] 23 is a diagram showing a data processing device of this embodiment. The data processing device 20 includes a memory 21 and a processor 22. The memory 21 stores measurement data. The processor 22 executes a process of calculating the reliability of estimated data. The three-dimensional optical property is a refractive index distribution or an absorptance distribution.

[0211] The data processing method of the first embodiment or the data processing method of the second embodiment is used in the data processing device 20. Therefore, a detailed description will be omitted.

[0212] The three-dimensional observation apparatus of this embodiment will be described using a three-dimensional observation apparatus of a first embodiment and a three-dimensional observation apparatus of a second embodiment.

[0213] The three-dimensional observation device of the first embodiment has the data processing device of the first embodiment or the data processing device of the second embodiment, a light source that emits light to illuminate an object, and a sensor that receives the light that has passed through the object and generates a signal.

[0214] Fig. 24 is a diagram showing the three-dimensional observation apparatus of the first embodiment. The same components as those in Fig. 23 are given the same numbers, and the description thereof will be omitted.

[0215] The three-dimensional observation device 30 has a light source 31 and a sensor 32. The light source 31 emits light that illuminates an object 33. The object 33 is illuminated by the light emitted from the light source 31. The object 33 is held in a petri dish 34. The sensor 32 receives the light that has passed through the object 33 and generates a signal (hereinafter referred to as a "detection signal").

[0216] The detection signal is input to the data processing device 20. In the data processing device 20, data is generated from the detection signal. Data may be generated from the detection signal and the generated data may be input to the data processing device 20. The generated data is data obtained by measuring light transmitted through the object 33. Therefore, in the data processing device 20, the reliability is calculated using the measurement data.

[0217] The three-dimensional observation apparatus of the second embodiment has a light source, a sensor, an illumination system that irradiates an object with illumination light, and an optical system that guides the illumination light to the object and guides the light that has transmitted through the object to the sensor.

[0218] Fig. 25 is a diagram showing a three-dimensional observation apparatus according to the second embodiment. The same components as those in Fig. 24 are given the same reference numbers, and the description thereof will be omitted.

[0219] The three-dimensional observation device 40 includes a light source 41, a collimator lens 42, a half mirror 43, a mirror 44, an illumination system 50, a detection system 60, a half mirror 45, and a sensor 46.

[0220] The light source 41 emits light that illuminates the object 33. The sensor 46 receives the light that has passed through the object 33 and generates a signal. The object 33 is held in a petri dish .

[0221] Light emitted from the light source 41 enters the collimator lens 42. Parallel light is emitted from the collimator lens 42. The parallel light is incident on the half mirror 43. Reflected light and transmitted light are emitted from the half mirror 43. The object 33 is located in the optical path along which the transmitted light travels. Nothing is placed in the optical path along which the reflected light travels.

[0222] The transmitted light is reflected by the mirror 44 and enters the illumination system 50. The illumination system 50 has a lens 51 and a lens 52. The parallel light emitted from the illumination system 50 is irradiated onto the object 33. The parallel light emitted from the object 33 enters the detection system 60.

[0223] The detection system 60 has an objective lens 61, a mirror 62, and an imaging lens 63. Parallel light incident on the objective lens 61 is condensed by the objective lens 61 and then enters the imaging lens 63. Diverging light is incident on the imaging lens 63.

[0224] The focal position of the imaging lens 63 coincides with the light collecting position of the objective lens 61. Therefore, parallel light is emitted from the imaging lens 63. The parallel light is incident on the half mirror 45. The parallel light reflected by the half mirror 43 is incident on the half mirror 45.

[0225] The parallel light that has passed through the object 33 passes through the half mirror 43, and the parallel light that has not passed through the object 33 is reflected by the half mirror 43. Therefore, the parallel light that has passed through the object 33 and the parallel light that has not passed through the object 33 are incident on the sensor 46. As a result, interference fringes are formed. The interference fringes are imaged by the sensor 46. A detection signal is output from the sensor 46.

[0226] The detection signal is input to the data processing device 20. In the data processing device 20, data is generated from the detection signal. Data may be generated from the detection signal and the generated data may be input to the data processing device 20. The generated data is data obtained by measuring light transmitted through the object 33. Therefore, in the data processing device 20, the reliability is calculated using the measurement data.

[0227] In the three-dimensional observation device 40, the light illuminating the object 33 is deflected by the mirror 44. Therefore, the object 33 can be illuminated at different illumination angles. In this case, the accuracy of the reconstruction data can be improved, as shown in FIG.

[0228] It is preferable that the three-dimensional observation apparatus of this embodiment further comprises a presentation unit that presents the reliability.

[0229] As shown in Fig. 24, the three-dimensional observation device 30 includes a display unit 35. Also, as shown in Fig. 25, the three-dimensional observation device 40 includes a display unit 35. The display unit 35 is a presentation unit. One example of the display unit 35 is a monitor. By providing the display unit 35, it is possible to display the reliability. The reliability can be displayed together with the restored data.

[0230] A speaker may be provided instead of the display unit 35. The reliability may be output as sound information from the speaker. The display unit 35 and the speaker may be collectively configured as a presentation unit.

[0231] The recording medium of this embodiment will be described using the recording medium of the first embodiment and the recording medium of the second embodiment.

[0232] The recording medium according to the first embodiment is a computer-readable recording medium having a program recorded thereon. The recording medium has a program recorded thereon for causing a computer having a memory and a processor to execute the estimation process.

[0233] In the estimation process, the three-dimensional optical properties of the object are estimated. The three-dimensional optical properties are the refractive index distribution or the absorptance distribution.

[0234] The processor is caused to execute a process of inputting measurement data stored in the memory into a neural network, a process of generating estimated data from the measurement data, a process of generating restored data from the estimated data, and a process of calculating the reliability of the estimated data based on the measurement data and the restored data.

[0235] The neural network is a trained model, the measurement data is data obtained by measuring light transmitted through an object, and the estimated data is data on the three-dimensional optical properties of the object estimated from the measurement data.

[0236] The estimation uses a neural network, and the restoration involves performing a forward propagation operation on the estimated data. In the forward propagation operation, the wavefront passing through the inside of the object estimated from the measurement data is sequentially obtained in the direction in which the light travels.

[0237] The recording medium according to the second embodiment is a computer-readable recording medium having a program recorded thereon. The recording medium has a program recorded thereon for causing a computer having a memory and a processor to execute the estimation process.

[0238] In the estimation process, the three-dimensional optical properties of the object are estimated. The three-dimensional optical properties are the refractive index distribution or the absorptance distribution.

[0239] The processor executes the following processes: inputting measurement data stored in the memory into the neural network; generating estimated data from the measurement data; generating reconstructed data from the estimated data; calculating the reliability of the estimated data based on the measurement data and the reconstructed data; and causing the neural network to learn using an amount inversely proportional to the reliability as a loss.

[0240] The measured data is data obtained by measuring light transmitted through an object, and the estimated data is data on the three-dimensional optical properties of the object estimated from the measured data.

[0241] The estimation uses a neural network, and the restoration involves performing a forward propagation operation on the estimated data. In the forward propagation operation, the wavefront passing through the inside of the object estimated from the measurement data is sequentially obtained in the direction in which the light travels.

[0242] The learning method of the present embodiment is a learning method for a neural network that calculates the reliability of estimated data, and the reliability of the estimated data is calculated based on measurement data and restored data.

[0243] The measurement data is data obtained by measuring light transmitted through an object, and the estimated data is data on the three-dimensional optical properties of the object estimated from the measurement data, where the three-dimensional optical properties are a refractive index distribution or an absorptivity distribution.

[0244] The restored data is data generated by performing a forward propagation operation on the estimated data. In the forward propagation operation, a wavefront passing through the inside of an object estimated from the measured data is sequentially obtained in the direction in which light travels.

[0245] The learning method of this embodiment includes a first learning step of learning using a first learning data set and a second learning step of learning using a second learning data set, and the first learning step and the second learning step are repeatedly performed.

[0246] The first learning data set includes the first data, the first corrected data, and teacher data indicating true of true or false, and the second learning data set includes the first data, the second corrected data, and teacher data indicating false of true or false.

[0247] The first corrected data is data obtained by performing a correction process on the first data, and the second corrected data is data obtained by performing a correction process on the second data, and the second data is different from the first data.

[0248] The first data and the second data are data obtained by measuring light transmitted through an object, or data generated by performing a forward propagation operation on an object model that models the three-dimensional optical characteristics of the object.

[0249] In the learning method of the present embodiment, the correction process applied to the first data and the correction process applied to the second data preferably include at least one process selected from among a transformation process, a rotation process, and a noise addition process.

[0250] The learning device of this embodiment includes a memory and a processor. The memory stores measurement data, and the processor executes a learning process on a neural network that calculates the reliability of estimated data.

[0251] The reliability of the estimated data is calculated based on the measurement data and the restoration data, where the measurement data is data obtained by measuring light transmitted through the object, and the estimated data is data of the three-dimensional optical properties of the object estimated from the measurement data, and the three-dimensional optical properties are a refractive index distribution or an absorptivity distribution.

[0252] The restored data is data generated by performing a forward propagation operation on the estimated data. In the forward propagation operation, a wavefront passing through the inside of an object estimated from the measured data is sequentially obtained in the direction in which light travels.

[0253] The learning process includes a first learning step in which learning is performed using a first learning data set, and a second learning step in which learning is performed using a second learning data set, and the first learning step and the second learning step are repeatedly performed.

[0254] The first learning data set includes the first data, the first corrected data, and teacher data indicating true of true or false, and the second learning data set includes the first data, the second corrected data, and teacher data indicating false of true or false.

[0255] The first corrected data is data obtained by performing a correction process on the first data, and the second corrected data is data obtained by performing a correction process on the second data, and the second data is different from the first data.

[0256] The first data and the second data are data obtained by measuring light transmitted through an object, or data generated by performing a forward propagation operation on an object model that models the three-dimensional optical characteristics of the object.

[0257] The recording medium of this embodiment will be described using the recording medium of the third embodiment.

[0258] The recording medium of the third embodiment is a computer-readable recording medium having a program recorded thereon. The recording medium has a program recorded thereon for causing a computer having a memory and a processor to execute a learning process for a neural network.

[0259] The neural network calculates the reliability of the estimated data based on the measurement data and the restored data, where the measurement data is data obtained by measuring light transmitted through the object, and the estimated data is data on the three-dimensional optical properties of the object estimated from the measurement data, and the three-dimensional optical properties are a refractive index distribution or an absorptivity distribution.

[0260] The restored data is data generated by performing a forward propagation operation on the estimated data. In the forward propagation operation, a wavefront passing through the inside of an object estimated from the measured data is sequentially obtained in the direction in which light travels.

[0261] The method includes a first learning step in which learning is performed using a first learning data set, and a second learning step in which learning is performed using a second learning data set, and the first learning step and the second learning step are repeatedly performed.

[0262] The first learning data set includes the first data, the first corrected data, and teacher data indicating true of true or false, and the second learning data set includes the first data, the second corrected data, and teacher data indicating false of true or false.

[0263] The first corrected data is data obtained by performing a correction process on the first data, and the second corrected data is data obtained by performing a correction process on the second data, and the second data is different from the first data.

[0264] The first data and the second data are data obtained by measuring light transmitted through an object, or data generated by performing a forward propagation operation on an object model that models the three-dimensional optical characteristics of the object. [Industrial Applicability]

[0265] The present invention is suitable for a data processing method, a data processing device, a three-dimensional observation device, and a recording medium for generating an index indicating the reliability of an estimation, and a learning method, a learning device, and a recording medium for generating an index indicating the reliability of an estimation. [Explanation of symbols]

[0266] 1 object 2. Illumination 3 Imaging unit 4 Optical system 5. Image sensor 10 Objective Lens 11 Imaging lens 12 Object 12' Estimated object 13 Imaging plane 20 Data Processing Device 21 Memory 22 processors 30 Three-dimensional observation device 31 Light source 32 Sensors 33 Object 34 Petri dish 35 Display section 40 Three-dimensional observation device 41 Light source 42 Collimating Lens 43 Half Mirror 44 Mirror 45 Half Mirror 46 Sensors 50 Lighting System 51, 52 Lenses 60 Detection System 61 Objective Lens 62 Mirror 63 Imaging Lens IM optical image IP image plane Zfo focus position P1, P2, PN position Ein1, Eout1, Ein2, Eout2, Eout Electric field N1, N2 complex refractive index Win incident wavefront Wout Output wavefront Fo1Fo2 Focus position θILL Illumination angle

Claims

1. an input step of inputting measurement data into the neural network; an estimation step of generating estimated data from the measurement data; a restoration step of generating restored data from the estimated data; A calculation step of calculating a reliability of the estimated data based on the measurement data and the restored data, The neural network is a trained model, the measurement data is data obtained by measuring light transmitted through an object, the estimated data is data of three-dimensional optical properties of the object estimated from the measurement data; the three-dimensional optical characteristic is a refractive index distribution or an absorptance distribution, The estimation uses the neural network, The reconstruction performs a forward propagation operation on the estimated data; A data processing method characterized in that, in the forward propagation calculation, a wavefront passing through the inside of the object estimated from the measurement data is sequentially obtained in a direction in which light travels.

2. an input step of inputting measurement data into the neural network; an estimation step of generating estimated data from the measurement data; a restoration step of generating restored data from the estimated data; a calculation step of calculating a reliability of the estimated data based on the measurement data and the restored data; A learning step in which the neural network learns using an amount inversely proportional to the reliability as a loss; the measurement data is data obtained by measuring light transmitted through an object, the estimated data is data of three-dimensional optical properties of the object estimated from the measurement data; the three-dimensional optical characteristic is a refractive index distribution or an absorptance distribution, The estimation uses the neural network, The reconstruction performs a forward propagation operation on the estimated data; A data processing method characterized in that, in the forward propagation calculation, a wavefront passing through the inside of the object estimated from the measurement data is sequentially obtained in a direction in which light travels.

3. Calculating the difference between the measured data and the restored data; 3. The data processing method according to claim 1, wherein the reliability is calculated based on the difference.

4. Calculating a correlation between the measurement data and the restored data; 3. The data processing method according to claim 1, wherein the reliability is calculated based on the correlation.

5. Calculating the sum of squares of the difference between the measurement data and the restored data; 3. The data processing method according to claim 1, wherein the reliability is calculated based on the magnitude of the sum of squares of the differences.

6. A first neural network and a second neural network, the first neural network is the neural network, the second neural network is a trained model; 3. The data processing method according to claim 1, wherein the reliability is calculated using the second neural network.

7. The second neural network is trained using a first training data set group and a second training data set group; The first training data set group includes a plurality of first training data sets, The first learning data set includes first data, first correction data, and teacher data indicating true among true and false, The second training data set group includes a plurality of second training data sets; The second learning data set includes the first data, the second corrected data, and teacher data indicating false among true and false, the first correction data is data obtained by performing a correction process on the first data, the second correction data is data obtained by performing a correction process on the second data, The second data is different from the first data, 7. The data processing method according to claim 6, wherein the first data and the second data are data obtained by measuring light transmitted through the object, or data generated by performing a forward propagation operation on an object model that models three-dimensional optical characteristics of the object.

8. The data processing method according to claim 1 , further comprising a step of presenting the reliability.

9. A memory and a processor, The memory stores measurement data; The processor performs an estimation process to estimate three-dimensional optical properties of an object; the three-dimensional optical characteristic is a refractive index distribution or an absorptance distribution, The estimation process includes: an input step of inputting the measurement data into a neural network; an estimation step of generating estimated data from the measurement data; a restoration step of generating restored data from the estimated data; A calculation step of calculating a reliability of the estimated data based on the measurement data and the restored data, The neural network is a trained model, the measurement data is data obtained by measuring light transmitted through the object, the estimated data is data of three-dimensional optical properties of the object estimated from the measurement data; The estimation uses the neural network, The reconstruction performs a forward propagation operation on the estimated data; A data processing device characterized in that, in the forward propagation calculation, a wavefront passing through the inside of the object estimated from the measurement data is sequentially obtained in a direction in which light travels.

10. A memory and a processor, The memory stores measurement data; The processor performs an estimation process to estimate three-dimensional optical properties of an object; the three-dimensional optical characteristic is a refractive index distribution or an absorptance distribution, The estimation process includes: an input step of inputting the measurement data into a neural network; an estimation step of generating estimated data from the measurement data; a restoration step of generating restored data from the estimated data; a calculation step of calculating a reliability of the estimated data based on the measurement data and the restored data; A learning step in which the neural network learns using an amount inversely proportional to the reliability as a loss; the measurement data is data obtained by measuring light transmitted through the object, the estimated data is data of three-dimensional optical properties of the object estimated from the measurement data; The estimation uses the neural network, The reconstruction performs a forward propagation operation on the estimated data; A data processing device characterized in that, in the forward propagation calculation, a wavefront passing through the inside of the object estimated from the measurement data is sequentially obtained in a direction in which light travels.

11. A data processing device according to claim 9 or 10; A light source that emits light to illuminate the object; and a sensor that receives light that has passed through the object and generates a signal.

12. an illumination system for irradiating the object with illumination light; 12. The three-dimensional observation apparatus according to claim 11, further comprising an optical system for directing the illumination light to the object and directing the light transmitted through the object to the sensor.

13. 13. The three-dimensional observation apparatus according to claim 11, further comprising a presentation unit that presents the reliability.

14. A computer-readable recording medium having a program recorded thereon for causing a computer having a memory and a processor to execute an estimation process, The estimation process includes estimating three-dimensional optical properties of the object; the three-dimensional optical characteristic is a refractive index distribution or an absorptance distribution, The processor, inputting the measurement data stored in the memory into a neural network; generating estimated data from the measured data; generating restored data from the estimated data; and calculating a reliability of the estimated data based on the measurement data and the restored data; The neural network is a trained model, the measurement data is data obtained by measuring light transmitted through an object, the estimated data is data of three-dimensional optical properties of the object estimated from the measurement data; The estimation uses the neural network, The reconstruction performs a forward propagation operation on the estimated data; A computer-readable recording medium having a program recorded thereon, characterized in that in the forward propagation calculation, a wavefront passing through the inside of the object estimated from the measurement data is sequentially obtained in the direction in which light travels.

15. A computer-readable recording medium having a program recorded thereon for causing a computer having a memory and a processor to execute an estimation process, The estimation process includes estimating three-dimensional optical properties of the object; the three-dimensional optical characteristic is a refractive index distribution or an absorptance distribution, The processor, inputting the measurement data stored in the memory into a neural network; generating estimated data from the measured data; generating restored data from the estimated data; A process of calculating a reliability of the estimated data based on the measurement data and the restored data; A process of learning the neural network by using an amount inversely proportional to the reliability as a loss; the measurement data is data obtained by measuring light transmitted through an object, the estimated data is data of three-dimensional optical properties of the object estimated from the measurement data; The estimation uses the neural network, The reconstruction performs a forward propagation operation on the estimated data; A computer-readable recording medium having a program recorded thereon, characterized in that in the forward propagation calculation, a wavefront passing through the inside of the object estimated from the measurement data is sequentially obtained in the direction in which light travels.

16. A learning method for a neural network that calculates the reliability of estimated data, comprising: The reliability of the estimated data is calculated based on the measurement data and the restored data; the measurement data is data obtained by measuring light transmitted through an object, the estimated data is data of three-dimensional optical properties of the object estimated from the measurement data; the three-dimensional optical characteristic is a refractive index distribution or an absorptance distribution, the restored data is data generated by performing a forward propagation operation on the estimated data, In the forward propagation calculation, a wavefront passing through the inside of the object, which is estimated from the measurement data, is sequentially obtained in a direction in which light travels; a first learning step of learning using a first learning data set; A second learning step of learning using a second learning data set, The first learning step and the second learning step are repeatedly performed; The first learning data set includes first data, first correction data, and teacher data indicating true among true and false, The second learning data set includes the first data, the second corrected data, and the teacher data indicating false among true and false, the first correction data is data obtained by performing a correction process on the first data, the second correction data is data obtained by performing a correction process on the second data, The second data is different from the first data, A neural network training method, characterized in that the first data and the second data are data obtained by measuring light transmitted through the object, or data generated by performing a forward propagation operation on an object model that models three-dimensional optical characteristics of the object.

17. 17. The neural network training method according to claim 16, wherein the correction processing applied to the first data and the correction processing applied to the second data include at least one of a transformation processing, a rotation processing, and a noise addition processing.

18. A memory and a processor, The memory stores measurement data; The processor executes a learning process for a neural network that calculates the reliability of the estimated data; The reliability of the estimated data is calculated based on the measurement data and the restored data; the measurement data is data obtained by measuring light transmitted through an object, the estimated data is data of three-dimensional optical properties of the object estimated from the measurement data; the three-dimensional optical characteristic is a refractive index distribution or an absorptance distribution, the restored data is data generated by performing a forward propagation operation on the estimated data, In the forward propagation calculation, a wavefront passing through the inside of the object, which is estimated from the measurement data, is sequentially obtained in a direction in which light travels; The learning process includes: a first learning step of learning using a first learning data set; A second learning step of learning using a second learning data set, The first learning step and the second learning step are repeatedly performed; The first learning data set includes first data, first correction data, and teacher data indicating true among true and false, The second learning data set includes the first data, the second corrected data, and the teacher data indicating false among true and false, the first correction data is data obtained by performing a correction process on the first data, the second correction data is data obtained by performing a correction process on the second data, The second data is different from the first data, A learning device characterized in that the first data and the second data are data obtained by measuring light transmitted through the object, or data generated by performing a forward propagation operation on an object model that models the three-dimensional optical characteristics of the object.

19. A computer-readable recording medium having a program recorded thereon for causing a computer having a memory and a processor to execute a learning process for a neural network, The neural network calculates the reliability of the estimated data based on the measurement data and the restored data; the measurement data is data obtained by measuring light transmitted through an object, the estimated data is data of three-dimensional optical properties of the object estimated from the measurement data; the three-dimensional optical characteristic is a refractive index distribution or an absorptance distribution, the restored data is data generated by performing a forward propagation operation on the estimated data, In the forward propagation calculation, a wavefront passing through the inside of the object, which is estimated from the measurement data, is sequentially obtained in a direction in which light travels; a first learning step of learning using a first learning data set; A second learning step of learning using a second learning data set, The first learning step and the second learning step are repeatedly performed; The first learning data set includes first data, first correction data, and teacher data indicating true among true and false, The second learning data set includes the first data, the second corrected data, and the teacher data indicating false among true and false, the first correction data is data obtained by performing a correction process on the first data, the second correction data is data obtained by performing a correction process on the second data, The second data is different from the first data, A computer-readable recording medium having a program recorded thereon, characterized in that the first data and the second data are data obtained by measuring light transmitted through the object, or data generated by performing a forward propagation operation on an object model that models three-dimensional optical characteristics of the object.

Images