Information processing system, endoscope system, image processing method and program

Abstract

To provide an information processing system and the like that expand a depth of field using an image processing technique based on deep learning while preventing network scales from increasing.SOLUTION: Defocus simulation processing (step S200) is performed for a region on an optical axis of a first imaging system 101 and a region other than on the optical axis in a training image 32, based on a transfer function or a point spread function on the optical axis. A processing unit 1130 uses a trained model 1120 to generate an output image 134 in which a blur of a processing target image which is an image captured by the first imaging system 101 is corrected, and estimates an object distance of the processing target image. The processing unit 1130 acquires a filter characteristic associated with the estimated object distance from a correction table 1400, and performs blur adjustment processing for the output image 134 using the acquired filter characteristic.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an information processing system, an endoscope system, an image processing method, and a program. [Background technology]

[0002] In endoscopic observation and the like, it is desirable to be able to observe a subject at a closer distance and in a magnified manner. However, optically, the depth of field narrows as resolution increases due to miniaturization of pixels, so there is a demand for a technology to extend the depth of field using image processing technology. Patent Document 1 discloses a technology to correct optical degradation of an imaging system using deep learning. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2018 / 037521 Summary of the Invention [Problem to be solved by the invention]

[0004] In Patent Document 1, a reference image captured in advance to which optical degradation information has been added is used as a training image. However, since there are an infinite number of pieces of optical degradation information to be learned depending on the object distance and the image height, a huge number of training images are required, and the scale of the network required for processing becomes large, which raises concerns about a decrease in processing capacity, an increase in implementation costs, etc. [Means for solving the problem]

[0005] One aspect of the present disclosure is an information processing system including: a memory unit that stores a trained model that has been machine-learned using a dataset including training images, ground truth images, and object distance labels; and a correction table that associates an object distance, which is the distance between an imaging system and an object, with a filter characteristic for blur correction; and a processing unit, wherein the training image is generated by performing a defocus simulation process on a predetermined object image that is in focus and captured by an arbitrary imaging system, the defocus simulation process simulating the effect of blur due to defocus of the first imaging system based on a transfer function or point spread function of the first imaging system at the predetermined object distance, the defocus simulation process being performed on an area on an optical axis of the first imaging system in the training image and an area other than on the optical axis, based on the transfer function or point spread function on the optical axis, and the ground truth image is generated by performing a defocus simulation process on a predetermined object image that is in focus and captured by the arbitrary imaging system, the defocus simulation process simulating the effect of blur due to defocus ... the transfer function or point spread function on the optical axis the trained model is machine-learned so that the training image becomes the correct image, and is machine-learned by assigning the object distance of the transfer function or point spread function of the first imaging system used in the defocus simulation process as the object distance label; the processing unit uses the trained model to generate an output image in which blur is corrected for a processing target image, which is an image captured by the first imaging system, estimates the object distance of the processing target image, acquires the filter characteristics associated with the estimated object distance from the correction table, and performs blur adjustment processing on the output image using the acquired filter characteristics.

[0006] Another aspect of the present disclosure relates to an endoscope system including the information processing system described above and an endoscope that captures the processing target image.

[0007] Another aspect of the present disclosure is an image processing method that uses a trained model that has been machine-learned using a dataset that includes training images, ground truth images, and object distance labels, and a correction table that associates object distance, which is the distance between an imaging system and an object, with filter characteristics for blur correction, wherein the training image is generated by performing a defocus simulation process on a predetermined object image that is in focus and captured by an arbitrary imaging system, the defocus simulation process simulating the effect of blur due to defocusing of the first imaging system based on a transfer function or point spread function of the first imaging system at the predetermined object distance, the defocus simulation process being performed on an area on an optical axis of the first imaging system in the training image and an area other than on the optical axis, based on the transfer function or point spread function on the optical axis, and the ground truth image is generated by performing a defocus simulation process on a predetermined object image that is in focus and captured by the first imaging system based on the transfer function or point spread function on the optical axis, the trained model is machine-learned so that the training image becomes the correct image, and is machine-learned by assigning the object distance of the transfer function or point spread function of the first imaging system used in the defocus simulation process as the object distance label, and the trained model includes the steps of generating an output image in which blur of a processing target image, which is an image captured by the first imaging system, is corrected using the trained model; estimating the object distance of the processing target image; and acquiring the filter characteristics associated with the estimated object distance from the correction table, and performing blur adjustment processing on the output image using the acquired filter characteristics.

[0008] Another aspect of the present disclosure relates to a program that causes a computer to execute the image processing method described above. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a block diagram illustrating an example of the configuration of an information processing system. [Figure 2] FIG. 2 is a block diagram illustrating a more detailed configuration example of the information processing system. [Figure 3] 10 is a flowchart illustrating an example of processing by the information processing system. [Figure 4] FIG. 1 is a block diagram illustrating an example of the configuration of a learning device. [Figure 5] A diagram illustrating a training model. [Figure 6] A diagram explaining a neural network. [Figure 7] 10 is a flowchart illustrating a trained model creation process. [Figure 8] FIG. 2 is a diagram illustrating an example of machine learning according to the present embodiment. [Figure 9] FIG. 4 is a diagram illustrating the relationship between the depth of field and the target depth of field. [Figure 10] 5A to 5C are diagrams illustrating an example of image data generation processing. [Figure 11] 10A and 10B are diagrams for explaining another example of image data generation processing. [Figure 12] FIG. 1 is a diagram illustrating a transfer function or a point spread function. [Figure 13] 5A to 5C are diagrams for explaining defocus simulation processing according to the present embodiment. [Figure 14] FIG. 1 is a block diagram illustrating an example of an endoscope system. [Figure 15] FIG. 10 is a block diagram illustrating another example of an endoscope system. [Figure 16] FIG. 10 is a block diagram illustrating another example of the configuration of an information processing system. [Figure 17] FIG. 10 is a block diagram illustrating in more detail another example configuration of the information processing system. [Figure 18] 10 is a flowchart illustrating another example of processing by the information processing system. [Figure 19] FIG. 10 is a diagram illustrating another example of machine learning according to the present embodiment. [Figure 20] 10 is a flowchart illustrating another example of the trained model creation process. [Figure 21] FIG. 10 is a diagram illustrating a second correction learning process. [Figure 22] FIG. 10 is another diagram illustrating the second correction learning process. [Figure 23] FIG. 1 is a diagram illustrating an example of the configuration of a trained model. [Figure 24] 10 is a flowchart illustrating a processing example of blur adjustment processing. [Figure 25] 5A and 5B are diagrams illustrating a correction table and a characteristic filter. [Figure 26] 10A and 10B are diagrams illustrating the relationship between the object distance and the MTF in the defocus simulation process. [Figure 27] FIG. 10 is another diagram illustrating the relationship between the object distance and the MTF in the defocus simulation process. [Figure 28] 10A and 10B are diagrams for explaining a specific calculation method of defocus simulation processing. [Figure 29] FIG. 10 is another diagram illustrating a specific calculation method for the defocus simulation process. [Figure 30] 5A and 5B are diagrams for explaining a specific calculation method of best focus simulation processing. [Figure 31] FIG. 10 is another diagram illustrating a specific calculation method for the best focus simulation process. [Figure 32] FIG. 3 is a diagram illustrating an example of a lens configuration of a first imaging system. [Figure 33] FIG. 4 is another diagram illustrating an example of the lens configuration of the first imaging system. [Figure 34] FIG. 10 is a diagram illustrating the amount of distortion. [Figure 35] FIG. 2 is a diagram illustrating a lens configuration including a phase modulation element. [Figure 36] 10A and 10B are diagrams illustrating an example of a change in MTF due to the inclusion of a phase modulation element. [Figure 37] 10A and 10B are diagrams for explaining another example of image data generation processing. [Figure 38] 10A and 10B are diagrams illustrating another example of the defocus simulation process. [Figure 39] 10A and 10B are diagrams for explaining another example of image data generation processing. [Figure 40] 10A and 10B are diagrams for explaining another example of the defocus simulation process. [Figure 41] 10A and 10B are diagrams for explaining another example of the best focus simulation process. [Figure 42] FIG. 10 is a diagram illustrating another example of the configuration of an information processing system. [Figure 43] 10A and 10B are diagrams for explaining another example of image data generation processing. [Figure 44] 10A and 10B are diagrams for explaining another example of the defocus simulation process. [Figure 45] 1A and 1B are diagrams illustrating the relationship between mosaic processing and demosaic processing. [Figure 46] 10A and 10B are diagrams for explaining another example of the best focus simulation process. [Figure 47] FIG. 10 is a diagram illustrating another example of the configuration of an information processing system. [Figure 48] 10 is a flowchart illustrating another example of processing by the information processing system. [Figure 49] 10 is a flowchart illustrating a first trained model creation process. [Figure 50] 10 is a flowchart illustrating a second trained model creation process. [Figure 51] 10A and 10B are diagrams for explaining another example of image data generation processing. [Figure 52] 10A and 10B are diagrams for explaining another example of the defocus simulation process. [Figure 53] 10A and 10B are diagrams for explaining another example of the best focus simulation process. [Figure 54] FIG. 10 is a diagram illustrating evaluation of frequency characteristics. [Figure 55] 5A and 5B are diagrams illustrating the relationship between the first frequency characteristic and the second frequency characteristic, etc. FIG. [Figure 56] 10A and 10B are diagrams illustrating a modified example of correction processing. [Figure 57] 10A and 10B are diagrams for explaining blur adjustment to an output image in more detail. [Figure 58] FIG. 10 is another diagram illustrating the adjustment of blur to the output image in more detail. [Figure 59] FIG. 10 is a diagram illustrating another example configuration of a trained model. DETAILED DESCRIPTION OF THE INVENTION

[0010] Preferred embodiments of the present disclosure will be described in detail below. Note that the embodiments described below do not unduly limit the scope of the claims, and not all of the configurations described in the embodiments are necessarily essential components.

[0011] FIG. 1 is a block diagram illustrating an example configuration of an information processing system 100 according to this embodiment. The information processing system 100 includes a storage unit 110 and a processing unit 130. The storage unit 110 stores a trained model 120 that has been machine-learned. The trained model 120 is a program module that outputs a corrected image in which blur caused by defocusing of a processing target image is corrected, and is generated or updated through machine learning, as described below. The processing target image is, for example, image data captured by a first imaging system 101 as shown in FIG. 1, but is not limited to this and details will be described later. Note that in this embodiment, image data that can be processed as digital data may simply be referred to as an image. The training image group 32G is a collection of training images 32 consisting of a first training image 32-1, a second training image 32-2, ..., an Nth training image 32-N, and details of the training image 32, along with a target image 36, will be described later. That is, the processing unit 130 of this embodiment corrects blur caused by defocus of the first imaging system 101 in the processing target image, which is an image captured by the first imaging system 101, using the trained model 120. The storage unit 110 and the processing unit 130 are also referred to as a storage device and a processing device, respectively.

[0012] In the present embodiment, the training images 32 are, for example, images of biological tissue or images of a subject simulating biological tissue when the information processing system 100 is incorporated into an endoscope system 300 (described later with reference to FIG. 14 ), but are not limited thereto. For example, in the learning stage (described later), images of charts conforming to a predetermined standard, such as an SFR (Spatial Frequency Response) chart, may be used as the training images 32. Furthermore, images of these charts that have been modified in a predetermined manner may also be used as the training images 32. The predetermined modification is, for example, a modification that makes the contrast of the chart the same as the contrast of the biological tissue.

[0013] In this embodiment, machine learning refers to, for example, supervised learning. Training data in supervised learning is a dataset in which input data is associated with correct labels. Specifically, the trained model 120 in this embodiment is generated by supervised learning based on a dataset in which input data consisting of training images 32 simulating the effects of various blurs is associated with correct labels consisting of focused correct images 36.

[0014] The processing unit 130 of this embodiment is configured with the following hardware. The hardware can include at least one of a circuit for processing digital signals and a circuit for processing analog signals. For example, the hardware can be configured with one or more circuit devices or one or more circuit elements mounted on a circuit board. The one or more circuit devices are, for example, ICs, etc. The one or more circuit elements are, for example, resistors, capacitors, etc.

[0015] The processing unit 130 may also be implemented by the following processor. The processing unit 130 of this embodiment includes a memory that stores information and a processor that operates based on the information stored in the memory. The memory is, for example, the storage unit 110. The information is, for example, a program and various data. The processor includes hardware. Various processors can be used as the processor, such as a central processing unit (CPU), a graphics processing unit (GPU), or a digital signal processor (DSP). The memory may be a semiconductor memory such as a static random access memory (SRAM) or a dynamic random access memory (DRAM), a register, a magnetic storage device such as a hard disk drive, or an optical storage device such as an optical disk drive. For example, the memory stores computer-readable instructions, and the functions of each unit of the processing unit 130 are realized as processing by the processor executing the instructions. The instructions here may be instructions from an instruction set that constitutes a program, or instructions that instruct the hardware circuitry of the processor to operate.

[0016] The trained model 120 of this embodiment may also be used in the information processing system 100 shown in the configuration example of Fig. 2. That is, the trained model 120 of this embodiment is used in the information processing system 100 including a storage unit 110 that stores the trained model 120, an input unit 140, a processing unit 130, and an output unit 150, and is machine-trained using a dataset including a training image group 32G and a master image 36.

[0017] The input unit 140 is an interface that receives a processing target image from the outside. Specifically, for example, as shown in FIGS. 1 and 2, it is an image data interface that receives image data from the first imaging system 101 as the processing target image. For example, the input unit 140 functions as the input unit 140 by using the received processing target image as input data to the trained model 120 and having the processing unit 130 perform the processing described below. That is, in the trained model 120 of this embodiment, the input unit 140 inputs the processing target image, which is an image captured by the first imaging system 101, to the trained model 120.

[0018] The output unit 150 is an interface that transmits the above-mentioned corrected image to the outside. For example, the output unit 150 functions as the output unit 150 by using output data from the trained model 120 as the corrected image transmitted by the output unit 150. The destination of the corrected image is, for example, a predetermined display device connected to the information processing system 100, and the output unit 150 functions as the output unit 150 by, for example, being an interface connectable to the predetermined display device, whereby the corrected image is displayed on the display device. Note that the output destination of the corrected image may be a storage device of an external device, etc.

[0019] FIG. 3 is a flowchart illustrating a method performed by the information processing system 100 of this embodiment. The processing unit 130 reads a processing target image (step S10) and a trained model (step S20), and then performs a correction process (step S30). Specifically, for example, the processing unit 130 inputs the processing target image received via the input unit 140 into the trained model 120 read from the storage unit 110. If the trained model 120 determines that the processing target image, which is input data, is common to the training image 32, it estimates that the data to be output is the correct image 36, and therefore outputs the correct image 36 when the processing target image is input. Comparing the processing target image and the correct image 36, it is found that the correct image 36 is an image in which blurring caused by defocusing of the first imaging system 101 in the processing target image has been corrected. In other words, the processing unit 130 uses the trained model 120 to perform a correction process (step S30) to correct blurring caused by defocusing of the first imaging system 101 in the processing target image.

[0020] Thereafter, the processing unit 130 outputs the corrected image (step S40). Specifically, the output unit 150 functions as described above, and the corrected image is output to a desired output destination. In other words, the output unit 150 outputs the corrected image obtained by the correction process.

[0021] Next, machine learning of the trained model 120 will be described. Machine learning is performed, for example, by a learning device 10. FIG. 4 is a block diagram showing an example configuration of the learning device 10. The learning device 10 includes, for example, a communication unit 12, a learning device processing unit 16, and a learning device storage unit 18.

[0022] The communication unit 12 is a communication interface capable of communicating with the information processing system 100 via a predetermined communication method. The predetermined communication method is, for example, a communication method conforming to a wireless communication standard such as Wi-Fi (registered trademark), but is not limited to this and may also be a communication method conforming to a wired communication standard such as USB. This allows the learning device 10 to transmit a trained model 120 that has been machine-learned using a method described below to the information processing system 100, and the information processing system 100 to update the trained model 120. Note that while FIG. 4 shows an example in which the learning device 10 and the information processing system 100 are separate, this does not preclude a configuration example in which the information processing system 100 includes a learning server equivalent to the learning device 10.

[0023] The learning device processing unit 16 controls the input and output of data between each functional unit, such as the communication unit 12 and the learning device storage unit 18. The learning device processing unit 16 can be realized by a processor similar to the processing unit 130 in FIG. 1. The learning device processing unit 16 executes various arithmetic processes based on a predetermined program read from the learning device storage unit 18 and operation input signals from an operation unit (not shown in FIG. 4), and controls data output operations to the information processing system 100, etc. The predetermined program here includes a machine learning program. In other words, the learning device processing unit 16 performs machine learning functions by reading and executing the machine learning program and necessary data, etc. from the learning device storage unit 18.

[0024] The learning device storage unit 18 stores a machine learning program (not shown), as well as a training model 20, a predetermined subject image 30, and optical system information 40. The learning device storage unit 18 can be realized by a semiconductor memory similar to the storage unit 110 described above. The learning device storage unit 18 may further include other information. The other information may be, for example, image sensor information 50 described below.

[0025] The predetermined subject image 30 is an image of a subject related to the processing target image, and a learning image 32 and a correct answer image 36, which will be described later, are created based on the predetermined subject image 30. That is, the learning device storage unit 18 pre-stores as many predetermined subject images 30 as the number of types of subjects that can become the processing target image. To give a more specific example, when the information processing system 100 is used in an endoscope system 300, which will be described later, an image of an inner cavity or the like captured by an endoscope 310, which will be described later, becomes the predetermined subject image 30. In the following description, for convenience, an imaging system that does not particularly matter for capturing the predetermined subject image 30 will be referred to as an arbitrary imaging system 104. A case in which the predetermined subject image 30 is captured using a limited imaging system will be described later.

[0026] The training model 20 is a model that is the target of machine learning by the learning device processing unit 16. The model here is information that derives the correspondence between estimation target data and estimation result data. More specifically, it is information that derives an output image 34, which is estimation result data, from a training image 32, which is estimation target data. In the training model 20 of this embodiment, a neural network NN is included in at least a part of the model. Details of the neural network NN will be described later with reference to FIG. 6. Note that, as described above, when the information processing system 100 and the learning device 10 are integrated, machine learning may be performed on the trained model 120.

[0027] Note that, in this embodiment, the term "output image" refers to an image output from the training model 20 in the learning stage or an image output from the training model 1020 described later, but may also include an image output from the trained model 1120 in the inference stage in the example described later. Hereinafter, for convenience of explanation, an image output from the training model 20 or the training model 1020 will be referred to as an output image 34, and an image output from the trained model 1120 described later will be referred to as an output image 134.

[0028] For example, when a first training image 32-1 is input to the training model 20, the training model 20 outputs a first output image 34-1. Similarly, when an Nth training image 32-N is input to the training model 20, the training model 20 outputs an Nth output image 34-N. That is, as shown in FIG. 5, in the learning device 10 of this embodiment, N images consisting of the first training image 32-1 to the Nth training image 32-N are input to the training model 20 as a training image group 32G.

[0029] FIG. 6 is a schematic diagram illustrating a neural network (NN). The neural network (NN) has an input layer to which data is input, a hidden layer that performs calculations based on the output from the input layer, and an output layer that outputs data based on the output from the hidden layer. While FIG. 6 illustrates a network with two hidden layers, the hidden layer may be one or three or more. The number of nodes included in each layer is not limited to the example shown in FIG. 6, and various modifications are possible. As shown in FIG. 6, nodes in a given layer are connected to nodes in adjacent layers. Each connection is assigned a weighting coefficient. Each node multiplies the output of the previous node by the weighting coefficient and calculates the sum of the multiplication results. Each node then adds a bias to the sum and applies an activation function to the sum to calculate the output of that node. This process is performed sequentially from the input layer to the output layer to calculate the output of the neural network (NN). Various activation functions, such as the sigmoid function and the ReLU function, are known, and these can be widely applied in this embodiment.

[0030] Various types of neural network NN models are known, and these can be widely applied in this embodiment. For example, the neural network NN may be a CNN (Convolutional Neural Network), an RNN (Recurrent Neural Network), or other models.

[0031] FIG. 7 is a flowchart illustrating an example of the trained model creation process (step S100). The trained model creation process (step S100) is a process for creating or updating a trained model 120 through machine learning. The learning device processing unit 16 reads a predetermined object image (step S110) and then performs image data generation process (step S120). For example, the learning device processing unit 16 reads a predetermined object image 30 from the learning device storage unit 18 and performs predetermined processing to generate a training image 32 and a correct answer image 36 using the predetermined object image 30. The predetermined processing includes a defocus simulation process (step S200), a best focus simulation process (step S300), etc., which will be described in detail later.

[0032] Thereafter, the learning device processing unit 16 performs a correction learning process (step S130). For example, the learning device processing unit 16 reads out the training model 20 from the learning device storage unit 18, inputs the training image 32 generated in the image data generation process (step S120) into the training model 20, and performs machine learning processing based on the output image 34 output from the training model 20 and the correct image 36.

[0033] The machine learning process based on the output image 34 and the correct image 36 is, for example, as shown in FIG. 8, a process of changing the network parameters of the neural network NN so that the first output image 34-1 to the Nth output image 34-N become the correct image 36. Specifically, the process of changing the network parameters of the neural network NN is, for example, a process of updating appropriate weighting coefficients in the neural network NN. The weighting coefficients here include biases. In updating the weighting coefficients, for example, an error backpropagation method can be used, in which weighting coefficients are updated from the output layer to the input layer. That is, the learning device 10 inputs input data from the training data into a model and determines the output by performing a forward calculation in accordance with the model configuration using the weighting coefficients at that time. An error function is calculated based on the output and the correct label, and the weighting coefficients are updated to reduce the error function.

[0034] More specifically, for example, the learning device processing unit 16 inputs the first training image 32-1 as input data to the neural network NN included in the training model 20, and performs forward calculations using the weighting coefficients at that time to output the first output image 34-1 as output data. The learning device processing unit 16 calculates an error function based on the first output image 34-1 and the correct answer image 36, which is the correct label. Then, it performs processing to update the weighting coefficients so as to reduce the error function. The learning device processing unit 16 also repeatedly performs similar processing on the second output image 34-2 to the Nth output image 34-N. In this way, the training model 20 is machine-learned so that one correct answer image 36 can be output for multiple types of training images 32. As a result, the machine-learned training model 20 is output as the trained model 120 to the information processing system 100, and the trained model 120 stored in the storage unit 110 is updated. 4 shows the learning device 10 and the information processing system 100 as being communicatively connected via the communication unit 12, but the learning device 10 and the information processing system 100 do not have to be communicatively connected. In this case, for example, a user can update the trained model 120 by causing the learning device 10 to temporarily store the training model 20 in an information storage medium as the trained model 120, carrying the information storage medium and moving to the location of the information processing system 100, and then causing the information processing system 100 to update the trained model 120 based on the information storage medium.

[0035] FIG. 9 is a diagram illustrating the relationship between the depth of focus and the depth of field for the first imaging system 101 of this embodiment, with the optical axis as the horizontal axis. Note that FIG. 9 is a diagram for convenience and does not depict the specific lens configuration of the first imaging system 101. For example, in FIG. 9, the range indicated by DP1 is the depth of field corresponding to the depth of focus in the optical design of the first imaging system 101. Therefore, for example, if the distance between the subject and the first imaging system 101 is a first object distance indicated by D1, the subject is located outside the depth of field, and when captured by the first imaging system 101, an image to be processed that includes the effects of blur due to defocus is obtained. Also, for example, if the distance between the subject and the first imaging system 101 is a second object distance indicated by D2, the subject is located within the depth of field, and the image to be processed is in focus. Note that, for example, if the distance between the subject and the first imaging system 101 is an object distance indicated by D3, that is, the position indicated by P1 on the optical axis in the depth of field is a position that satisfies the best focus condition. 9, the first object distance indicated by D1 and the second object distance indicated by D2 are shown on the near point side from the position indicated by P1, but are not limited to the near point side and may be on the far point side. In the following description and illustrations, the method of this embodiment will be explained using an object distance on the near point side as an example, but this does not prevent the method of this embodiment from being applicable even when an object distance on the far point side is used.

[0036] For example, in a system equipped with the first imaging system 101, miniaturizing pixels to increase resolution narrows the depth of field, and therefore extending the depth of field is desirable. Also, for example, when the first imaging system 101 is used in an endoscope 310 of an endoscope system 300 (described later), it is difficult to adjust the endoscope 310 to the best focus position for a desired subject, and therefore extending the depth of field is desirable.

[0037] Therefore, in this embodiment, a trained model 120, which has undergone the machine learning described above with reference to FIG. 8 and the like as a data set in which an image obtained by simulating the effect of blur on a predetermined subject image 30 captured in advance is used as a training image 32 and an image in focus is used as a ground truth image 36, is incorporated into the information processing system 100. In this way, a captured image to which the effect of blur due to defocus has been added is used as a processing target image, and by performing the processing of FIG. 3, a corrected image in focus is output from the information processing system 100. This makes it possible to substantially expand the range of the depth of field of the first imaging system 101.

[0038] More specifically, the depth of field can be substantially expanded from the range shown in DP1 in FIG. 9 to the range shown in DP2. Substantially expanding refers to expanding the apparent depth of field to a range where the depth of field is not optically expanded, but the image processing performed by the information processing system 100 allows a subject actually located outside the depth of field to be captured as if it were located within the depth of field. That is, when a subject is located at a position away from the first imaging system 101 by the object distance shown in D1, a blurred processing target image is output from the first imaging system 101. However, since this position is located within the effective depth of field shown in DP2, the processing target image is corrected to a focused corrected image and output from the information processing system 100. In the following description, the effective depth of field shown in DP2 in FIG. 9 expanded using the trained model 120 of this embodiment is referred to as the target depth of field. Note that the focused corrected image here does not necessarily mean that the entire image is strictly focused. For example, even if a portion of the output corrected image is blurred, the user may determine that the function of the information processing system 100 is sufficient as long as a procedure using the endoscope 310 can be performed. In other words, the target depth of field distance in this embodiment is a distance wider than the optically determined depth of field distance, but is a distance that can vary depending on the user's tolerance level, etc. Therefore, DP2 shown in FIG. 9 is shown merely for convenience and does not indicate a fixed length. The same applies to the following explanation.

[0039] The trained model 120 of this embodiment is machine-learned so that a blurred image obtained by capturing an image of a subject located in the range shown in DP10 in Fig. 9 can be corrected to an in-focus image, using the difference between the target depth of field shown in DP2 and the depth of field shown in DP1. In other words, the distance shown in DP10 is the distance required for machine learning.

[0040] The method of image data generation processing (step S120) for generating training images 32 and ground truth images 36 required for the machine learning will be described with reference to Fig. 10. Note that the method of image data generation processing is not limited to that shown in Fig. 10, and various modifications are possible as will be described later. Therefore, the image data generation processing shown in Fig. 10 can also be referred to as step S120-1.

[0041] In any example, the predetermined subject image 30 of this embodiment is assumed to be captured at an object distance at which the imaging system that captured the image is in focus.

[0042] The learning device processing unit 16 generates a learning image 32 by performing a defocus simulation process (step S200) on a predetermined object image 30 captured by an arbitrary imaging system 104. In the following description, for example, the defocus simulation process for generating the first learning image 32-1 may also be referred to as step S200-1, and similarly, the defocus simulation process for generating the Nth learning image 32-N may also be referred to as step S200-N. The same applies to steps S202, S204, S206, S208, S210, S220, and S230 described below. For example, when generating the first learning image 32-1 by the defocus simulation process (step S200-1), the learning device processing unit 16 selects information on the first object distance from the loaded optical system information 40. Similarly, when generating the second learning image 32-2 by step S200-2, the learning device processing unit 16 selects information on the second object distance from the loaded optical system information 40. That is, in this embodiment, the optical system information 40 corresponding to the Nth learning image 32-N is the Nth object distance, and when generating the Nth learning image 32-N, the learning device processing unit 16 can select and expand information on the corresponding Nth object distance from the optical system information 40. In the following explanation, the defocus simulation process will be exemplified as a process for generating the first learning image 32-1, but the same process is also performed when generating the second learning image 32-2 to the Nth learning image 32-N.

[0043] The learning device processing unit 16 also generates a ground truth image 36 by performing a best focus simulation process (step S300) on the predetermined subject image 30. For example, the learning device processing unit 16 selects information on the object distance at which the first imaging system 101 is in focus from the read optical system information 40. The information on the object distance at which the first imaging system 101 is in focus is, for example, as shown in D3, the design distance from the first imaging system 101 to the point shown in P1 in FIG. 9, which is the object distance that corresponds to the so-called best focus condition.

[0044] The image data generation process of this embodiment may be as shown in Fig. 11. The image data generation process shown in Fig. 11 may also be called step S120-2. Furthermore, a description of the same processes as those in Fig. 10 will be omitted as appropriate.

[0045] 11 differs from step S120-1 in Fig. 10 in that the best focus simulation process (step S300) is not performed, and the predetermined subject image 30 itself is used as the target image 36. This is because the predetermined subject image 30 can be used as the target image 36 if it is an image captured at an object distance at which any imaging system 104 is in focus.

[0046] The defocus simulation process (step S200) will be described using FIGS. 12 and 13. Optical system information 40, which is read when performing the defocus simulation process (step S200), includes information on a transfer function or a point spread function. The transfer function or the point spread function varies depending on the amount of defocus in the optical axis direction and the image height in a plane perpendicular to the optical axis. For example, at a first object distance, a region perpendicular to the optical axis and having the same size as a predetermined object image 30 is divided into regions FC11-1, FC12-1, FC13-1, FC21-1, FC22-1, FC23-1, FC31-1, FC32-1, and FC33-1. In this case, the transfer function or the point spread function at the first object distance may exhibit different values ​​for each divided region. Similarly, for example, at the Nth object distance, a region perpendicular to the optical axis and the same size as the predetermined object image 30 is divided into regions FC11-N, FC12-N, FC13-N, FC21-N, FC22-N, FC23-N, FC31-N, FC32-N, and FC33-N. In this case, the transfer function or point spread function at the Nth object distance may exhibit different values ​​for each divided region. Furthermore, the transfer function or point spread function of region FC11-1 may exhibit different values ​​from the transfer function or point spread function of region FC11-N. The same applies to regions FC12-1 and FC12-N, ..., and regions FC33-1 and FC33-N. Thus, if the training image group 32G is a collection of N training images 32, as shown in FIG. 12, the amount of information on the transfer function or point spread function required for machine learning becomes enormous.

[0047] In this regard, in this embodiment, a transfer function or point spread function on the optical axis is used when performing machine learning. In this embodiment, region FC22-1 is a region through which the optical axis of the first imaging system 101 passes. That is, the transfer function or point spread function in region FC22-1 is the transfer function or point spread function on the optical axis of the first imaging system 101 at the first object distance. Similarly, the transfer function or point spread function in region FC22-N at the Nth object distance is the transfer function or point spread function on the optical axis of the first imaging system 101. Note that although the transfer function or point spread function is divided into nine in FIG. 12, this is merely an example, and the same applies to FIG. 13. For example, regions FC22-1 to FC22-N in FIG. 12 are sets each including a predetermined number of images in the vertical and horizontal directions, but each may be one pixel. In other words, the transfer function or point spread function on the optical axis in this embodiment is a transfer function or point spread function in at least one of an area of ​​one pixel passing through the optical axis or an area of ​​a predetermined number of pixels including that pixel.

[0048] As shown in FIG. 13, in the defocus simulation process (step S200), a process (step S210) is performed on the predetermined object image 30 to simulate the effect of blur based on the on-axis transfer function or on-axis point spread function of the first imaging system 101. Details of step S210 will be described later. That is, step S210 is also performed on regions other than those on the optical axis of the predetermined object image 30 based on the on-axis transfer function or on-axis point spread function of the first imaging system 101. For example, suppose the predetermined object image 30 is divided into nine regions AR11, AR12, AR13, AR21, AR22, AR23, AR31, AR32, and AR33, as in FIG. 12. For example, when generating a first training image 32-1, the training device processing unit 16 performs the calculation of step S210-1 on the region AR11 using the on-axis transfer function or on-axis point spread function shown as FC22-1 in FIG. 12. In the following explanation and in the illustration of FIG. 13, this calculation will be abbreviated as AR11*FC22-1. The same applies to calculations such as step S210 that use other regions. Furthermore, the "*" here indicates convolution when, for example, a PSF is used as the point spread function, as will be described in detail later. Furthermore, when, for example, an OTF is used as the transfer function, the "*" indicates that the frequency characteristic obtained by Fourier transforming the region AR11 is multiplied by the OTF of the region FC22-1.

[0049] Furthermore, the learning device processing unit 16 also performs step S210-1 for regions AR12 to AR33 using the transfer function or point spread function on the optical axis shown in FC22-1. That is, although some parts are omitted in FIG. 13, the learning device processing unit 16 performs AR12*FC22-1, AR13*FC22-1, AR21*FC22-1, AR22*FC22-1, AR23*FC22-1, AR31*FC22-1, AR32*FC22-1, and AR33*FC22-1. In this way, the learning device processing unit 16 divides the same region as the predetermined object image 30 into a desired number of regions, and performs step S210 using the transfer function or point spread function of one of the divided regions.

[0050] Similarly, suppose the generated first learning image 32-1 is divided into nine regions: region BR11-1, region BR12-1, region BR13-1, region BR21-1, region BR22-1, region BR23-1, region BR31-1, region BR32-1, and region BR33-1. Region BR11-1 corresponds to the result of performing step S210-1 on the aforementioned region AR11. That is, as shown in FIG. 13, BR11-1 = AR11 * FC22-1. Similarly, BR12-1 = AR12*FC22-1, BR13-1 = AR13*FC22-1, BR21-1 = AR21*FC22-1, BR22-1 = AR22*FC22-1, BR23-1 = AR23*FC22-1, BR31-1 = AR31*FC22-1, BR32-1 = AR32*FC22-1, BR33-1 = AR33*FC22-1.

[0051] This method is also used when generating the Nth training image 32-N. That is, although not shown, the training device processing unit 16 performs the following: BR11-N = AR11 * FC22-N, BR12-N = AR12 * FC22-N, ..., BR22-N = AR22 * FC22-N, ..., BR32-N = AR32 * FC22-N, BR33-N = AR33 * FC22-N. From the above, a defocus simulation process (step S200) is performed on the region (BR22) on the optical axis of the first imaging system 101 in each training image 32 and the regions other than the optical axis (BR11, ..., BR21, BR23, ..., FC33) based on the transfer function or point spread function (FC22) on the optical axis.

[0052] The transfer function in this embodiment can also be called an optical transfer function or OTF. OTF stands for Optical Transfer Function. The point spread function in this embodiment can also be called a point spread function or PSF. PSF stands for Point Spread Function. OTF is the result of a Fourier transform of the PSF. In other words, PSF is the result of an inverse Fourier transform of the OTF. OTF is a complex function, and the absolute value of the OTF is called a modulation transfer function, amplitude transfer function, or MTF. MTF stands for Modulation Transfer Function.

[0053] As described above, the information processing system 100 of this embodiment includes a storage unit 110 that stores a trained model 120 that has been machine-learned using a dataset including a training image group 32G and a gold standard image 36, and a processing unit 130 that uses the trained model 120 to correct blur due to defocus of the first imaging system 101 in a processing target image, which is an image captured by the first imaging system 101. The training image group 32G includes multiple training images 32 generated by performing a defocus simulation process (step S200) on a predetermined object image 30, in which the predetermined object captured by the arbitrary imaging system 104 is focused on the predetermined object, based on the transfer function or point spread function of the first imaging system 101 at multiple object distances. The defocus simulation process is performed on an area on the optical axis of the first imaging system 101 and an area off the optical axis in each of the multiple training images 32, based on the transfer function or point spread function on the optical axis. The correct image 36 is an image generated by performing a best focus simulation process (step S300) that simulates the state in which the first imaging system 101 is in focus for the predetermined object image 30 based on the transfer function or point spread function at the object distance at which the first imaging system 101 is in focus, or is the predetermined object image 30 itself. The trained model 120 is machine-learned so that each training image 32 becomes the correct image 36.

[0054] As described above, the information processing system 100 of this embodiment includes the storage unit 110 that stores the trained model 120 and the processing unit 130. Therefore, even if the processing target image captured by the first imaging system 101 includes the influence of blur due to defocus, a corrected image in which the influence of blur is corrected can be output. This allows the depth of field of the first imaging system 101 to be substantially expanded. Furthermore, since the training image group 32G and the ground truth image 36 are created in advance based on the predetermined subject image 30 captured by an arbitrary imaging system 104, when the subject related to the processing target image is a subject being captured for the first time by the first imaging system 101, the trained model 120 that has been machine-learned in advance can be used. Furthermore, the defocus simulation process (step S200) is performed on the on-axis region and off-axis region of the first imaging system 101 in each training image 32 based on the on-axis transfer function or point spread function. This reduces the amount of information required for the defocus simulation process (step S200). This makes it possible to create a trained model 120 with an appropriate size for the neural network NN required for machine learning. This makes it possible to easily implement the trained model 120 in the information processing system 100.

[0055] The method of this embodiment can also be realized as a trained model 120. That is, the trained model 120 of this embodiment is used in an information processing system 100 including a storage unit 110 that stores the trained model 120, an input unit 140, a processing unit 130, and an output unit 150, and is machine-learned using a dataset including a training image group 32G and a ground truth image 36. The training image group 32G includes a plurality of training images 32 generated by performing a defocus simulation process on a predetermined object image 30, in which the predetermined object image is captured by an arbitrary imaging system 104 and the arbitrary imaging system 104 is focused on the predetermined object, based on the transfer function or point spread function of the first imaging system 101 at a plurality of object distances. The defocus simulation process is performed on an area on the optical axis of the first imaging system 101 and an area off the optical axis in each of the plurality of training images 32, based on the transfer function or point spread function on the optical axis. The correct image 36 is an image generated by performing a best focus simulation process that simulates the state in which the first imaging system 101 is in focus on the predetermined object image 30 based on a transfer function or a point spread function at an object distance where the first imaging system 101 is in focus, or is the predetermined object image 30 itself. The trained model 120 is machine-trained so that each training image 32 becomes the correct image 36. The input unit 140 inputs a processing target image, which is an image captured by the first imaging system 101, to the trained model 120. The processing unit 130 uses the trained model 120 to perform a correction process that corrects blurring of the processing target image due to defocusing of the first imaging system 101. The output unit 150 outputs the corrected image obtained by the correction process. In this manner, the same effect as described above can be obtained.

[0056] The technique of this embodiment can also be realized as an information processing method. That is, the information processing method of this embodiment corrects blur due to defocus of the first imaging system 101 in a processing target image, which is an image captured by the first imaging system 101, using a trained model 120 that has been machine-learned using a dataset including a training image group 32G and a gold standard image 36. The training image group 32G includes a plurality of training images 32 that are generated by performing a defocus simulation process on a predetermined object image 30, in which an arbitrary imaging system 104 focuses on a predetermined object captured by the arbitrary imaging system 104, to simulate the effect of blur due to defocus of the first imaging system 101 based on the transfer function or point spread function of the first imaging system 101 at a plurality of object distances. The defocus simulation process is performed on an area on the optical axis of the first imaging system 101 and an area off the optical axis in each of the plurality of training images 32, based on the transfer function or point spread function on the optical axis. The correct image 36 is an image generated by performing a best focus simulation process that simulates the state in which the first imaging system 101 is in focus on the predetermined object image 30 based on the transfer function or point spread function at the object distance at which the first imaging system 101 is in focus, or is the predetermined object image 30 itself. The trained model 120 is machine-learned so that each training image 32 becomes the correct image 36. In this way, the same effect as above can be obtained.

[0057] The method of this embodiment can also be realized as an information storage medium storing the trained model 120. In this way, the training model 20 obtained by machine learning in the learning device 10 can be stored in the information storage medium. As a result, by connecting the information storage medium to the information processing system 100, the training model 20 can be updated to the latest trained model 120. This makes it possible to obtain the same effect as above even under certain circumstances. Examples of certain circumstances include a situation where the location of the learning device 10 is far from the location of the information processing system 100, or a situation where data communication between the learning device 10 and the information processing system 100 is not possible.

[0058] The method of the present embodiment may also be realized as an endoscope system 300. For example, the endoscope system 300 of the present embodiment includes a processor unit 200 including the above-described information processing system 100, and an endoscope 310 connected to the processor unit 200 and configured to capture an image to be processed. In this way, it is possible to construct an endoscope system 300 including the information processing system 100 having the above-described effects.

[0059] More specifically, the endoscope system 300 can have a configuration example as shown in Fig. 14. The endoscope system 300 includes an endoscope 310, an operation unit 320, a display unit 330, and a processor unit 200. The processor unit 200 includes a storage unit 210, a control unit 220, and an information processing system 100. The information processing system 100 in Fig. 14 further includes a storage interface 160 in addition to the configuration described above in Fig. 2. Note that descriptions of the same configuration as in Fig. 2 will be omitted as appropriate.

[0060] The endoscope 310 includes an imaging device at its tip (not shown). The imaging device includes a first imaging system 101. The tip of the endoscope 310 is inserted into a body cavity, the imaging device captures images of the abdominal cavity, and the captured image data is transmitted from the endoscope 310 to the processor unit 200. The operation unit 320 is a device that allows a user to operate the endoscope system 300, and is, for example, a button, a dial, a foot switch, or a touch panel. The display unit 330 is a device that displays images captured by the endoscope 310, and is, for example, a liquid crystal display, but may also be hardware integrated with the operation unit 320, for example, as a touch panel.

[0061] The processor unit 200 performs various processes such as control and image processing in the endoscope system 300. For example, the control unit 220 performs mode switching, zooming, display switching, etc. of the endoscope system 300 based on information input from the operation unit 320, thereby realizing the functions of the processor unit 200. The storage unit 210 records images captured by the endoscope 310. The storage unit 210 is, for example, a semiconductor memory, a hard disk drive, or an optical drive.

[0062] In the configuration example shown in Figure 14, the function of receiving imaging data from the endoscope 310 is realized by making the connector to which the cable of the endoscope 310 is connected or the interface circuit for receiving imaging data, etc., the input unit 140, but the processor unit 200 may also be provided with an interface circuit for receiving imaging data.

[0063] The storage interface 160 is an interface for accessing the storage unit 210. The storage interface 160 records image data received by the input unit 140 in the storage unit 210. When playing back the recorded image data, the storage interface 160 reads the image data from the storage unit 210 and transmits the image data to the processing unit 130. The processing unit 130 performs the processing described above in FIG. 3 on the image data from the input unit 140 or the storage interface 160 as the image to be processed. As a result, the processing unit 130 outputs the corrected image via the output unit 150, and the corrected image in focus is displayed on the display unit 330.

[0064] The endoscope system 300 of this embodiment may have a configuration example shown in FIG. 15, for example. The configuration example of FIG. 15 differs from the configuration example of FIG. 14 in that the information processing system 100 and the processor unit 200 are provided separately. The information processing system 100 and the processor unit 200 may be connected via device-to-device communication such as USB, or via network communication such as LAN or WAN. The information processing system 100 is configured by one or more information processing devices. When the information processing system 100 is configured by multiple information processing devices, the information processing system 100 may be a cloud system in which multiple PCs or multiple servers connected via a network perform parallel processing. The storage unit 170 in FIG. 15 corresponds to the storage unit 210 in FIG. 14.

[0065] The processor unit 200 includes a control unit 220, an imaging data receiving unit 230, an input unit 240, an output unit 250, a processing unit 260, and a display interface 270. The imaging data receiving unit 230 is configured with an interface circuit and the like similar to the input unit 140 in FIG. 14 , and receives imaging data from the endoscope 310. The processing unit 260 transmits the image data received by the imaging data receiving unit 230 to the information processing system 100 via the output unit 250. The information processing system 100 performs the processing of FIG. 3 on the received image data as a processing target image to generate a corrected image. The input unit 240 receives the corrected image transmitted from the information processing system 100 via the output unit 150 and outputs the corrected image to the processing unit 260. The processing unit 260 outputs the corrected image to the display unit 330 via the display interface 270. As a result, the corrected image is displayed on the display unit 330. Display interface 270 in Fig. 15 is configured with the same hardware as output unit 150 in Fig. 14, and realizes the same functions as output unit 150 in Fig. 14. Note that in Fig. 15, input unit 140 and output unit 150 of information processing system 100 may be configured with separate interfaces, or the functions of input unit 140 and output unit 150 may be realized with a single input / output interface. The same applies to input unit 240 and output unit 250 of processor unit 200.

[0066] The method of this embodiment is not limited to the above, and various modifications are possible. For example, the endoscope system 300 of this embodiment may be configured to include the endoscope 310 described above with reference to FIG. 14 etc., and the information processing system 1000 shown in FIG. 16. The information processing system 1000 shown in FIG. 16 includes a storage unit 1110 and a processing unit 1130. The storage unit 1110 includes a trained model 1120 and a correction table 1400. The storage unit 1110 in FIG. 16 can be implemented using a memory similar to the storage unit 110 in FIG. 1, and the processing unit 1130 can be implemented using a processor similar to the processing unit 130 in FIG. 1. Details of the trained model 1120 and the correction table 1400 will be described later.

[0067] FIG. 17 is a block diagram showing a more detailed example configuration of the information processing system 1000. In FIG. 17, the information processing system 1000 further includes an input unit 1140 and an output unit 1150 in addition to the storage unit 1100 and processing unit 1130 described above. The input unit 1140 in FIG. 17 is an image data interface that receives image data from the first imaging system 101 as an image to be processed, similar to the input unit 140 in FIG. 2. The output unit 1150 in FIG. 17 is an interface that transmits a corrected image to the outside, similar to the output unit 1150 in FIG. 2, and fulfills the function of the output unit 1150 by using output data from the trained model 1120 as the corrected image transmitted by the output unit 1150.

[0068] Furthermore, the processing example according to the method of this embodiment may be performed by modifying the flowchart shown in FIG. 3 to the flowchart shown in FIG. 18. In FIG. 18, the processing unit 1130 reads an image to be processed (step S1010) and reads a trained model (step S1020). Step S1010 in FIG. 18 corresponds to step S10 in FIG. 3, and step S1020 in FIG. 18 corresponds to step S20 in FIG. 3. That is, specifically, for example, the processing unit 1130 inputs the image to be processed received via the input unit 1140 into the trained model 1120 read from the storage unit 1110.

[0069] Thereafter, the processing unit 1130 performs a correction process (step S1030) and a blur adjustment process (step S1032), and outputs a blur-adjusted image 1034 as a corrected image (step S1040). In this way, the corrected image in Fig. 17 differs from the corrected image in Fig. 2 in that it is a blur-adjusted image 1034 generated by performing the correction process (step S1030) on the processing target image and then performing the blur adjustment process (step S1032) on the output image 134.

[0070] The following describes machine learning for generating the trained model 1120 in Fig. 16. The trained model 1120 in Fig. 16 is trained by machine learning using a dataset that includes not only the training images 32 and the ground truth images 36 described above, but also object distance labels.

[0071] Note that the object distance label in this embodiment more specifically refers to an estimated object distance label 1070 and a correct object distance label 1076, which will be described later.

[0072] For example, by using the learning device 10 described above in Fig. 4 or the like to perform machine learning on the training model 1020 described below, the training model 1020 can be updated to become the trained model 1120. Similar to the training model 20 in Fig. 4, at least a part of the training model 1020 includes the neural network NN described above.

[0073] As shown in FIG. 19, the training image 32 described above in FIG. 5 and the like is input to the training model 1020 as input data. There are as many types of training images 32 as there are object distances. That is, as described above in FIG. 10, the first training image 32-1 is generated by performing a defocus simulation process (step S200-1) on a transfer function or point spread function corresponding to a first object distance for a predetermined object image 30. Similarly, the Nth training image 32-N is generated by performing a defocus simulation process (step S200-N) on a transfer function or point spread function corresponding to an Nth object distance for a predetermined object image 30. In this way, in the learning stage, machine learning can be generally considered to be performed by inputting the Nth training image 32-N as input data to the training model 1020.

[0074] 13, the transfer function on the optical axis or the point spread function is used in the defocus simulation process (step S200), and the same applies to the defocus simulation process (step S200) used to generate the learning image 32 input to the training model 1020 in Fig. 19. This makes it possible to reduce the scale of the network required to generate the trained model 1120 in Fig. 16, etc.

[0075] Then, an output image 134 is output from the training model 1020 through calculations by the neural network NN. The output image 134 is estimation result data derived by the training model 1020 from the learning image 32, which is the estimation target data, and is similar to the output image 34 described above in FIG. 5.

[0076] Furthermore, when training image 32 is input to training model 1020, an object distance corresponding to the transfer function or point spread function used for input training image 32 is estimated through calculations by neural network NN. As a result, estimated object distance label 1070 relating to the estimated object distance is output together with output image 134 described above. In other words, estimated object distance label 1070 is a class classification result. From the above, in the learning stage, as a result of Nth training image 32-N being input to training model 1020 as input data, Nth output image 134-N and Nth estimated object distance label 1070-N are output from training model 1020 as output data.

[0077] FIG. 20 is a flowchart illustrating an example of the trained model creation process (step S1100) for the trained model 1120. After reading a predetermined object image (step S1110), the learning device processing unit 16 performs a training data generation process (step S1120). For example, as described above with reference to FIG. 7, the learning device processing unit 16 reads the predetermined object image 30 from the learning device storage unit 18 and generates a training image 32 and a correct image 36 using the predetermined object image 30. In addition, the learning device processing unit 16 generates a correct object distance label 1076 based on the object distance corresponding to the transfer function or point spread function related to the defocus simulation process (step S200) used to generate the training image 32.

[0078] As mentioned above, in Figures 10 and 11, the correct image 36 is an image obtained by performing the best focus simulation process (step S300) on the specified subject image 30, or the specified subject image 30 itself, but the same is true for the correct image 36 generated by step S1120 in Figure 20.

[0079] Thereafter, the learning device processing unit 16 performs a first corrected learning process (step S1131) and a second corrected learning process (step S1132), and ends the flow. For example, the learning device processing unit 16 reads the training model 1020 from the learning device storage unit 18 and performs a process of inputting input data based on the training image 32 generated in the learning data generation process (step S1120) into the training model 1020. As a result, the output image 134 and the estimated object distance label 1070 are output as output data from the training model 1020, as described above in FIG. 19. Then, the learning device processing unit 16 performs machine learning based on the output image 134 and the supervised image 36 as the first corrected learning process (step S1131). The first corrected learning process (step S1131) is similar to the corrected learning process (step S130) described above in FIGS. 7 and 8, and therefore detailed description and illustration thereof will be omitted.

[0080] Furthermore, as a second corrected learning process (step S1132), the learning device processing unit 16 performs machine learning processing based on the estimated object distance label 1070 generated in step S1120 and the correct object distance label 1076. For example, as shown in FIG. 21, the learning device processing unit 16 changes the network parameters of the neural network NN so that the first estimated object distance label 1070-1 to the N-th estimated object distance label 1070-N become the correct object distance label 1076. The significance of changing the network parameters of the neural network NN here is as described above with reference to FIG. 7.

[0081] More specifically, for example, assume that a learning image 32 obtained by performing a defocus simulation process (step S200) on a predetermined subject image 30 using a transfer function or point spread function of the Mth object distance is input to the training model 1020. In this case, step S1120 generates a correct object distance label 1076 corresponding to the Mth object distance.

[0082] Then, the estimated object distance label 1070 output from the training model 1020 is classified into a class according to the estimated object distance. For example, if the object distance associated with the estimated object distance label 1070 output from the training model 1020 is the first object distance, it is classified into class 1. Similarly, if the object distance associated with the estimated object distance label 1070 output from the training model 1020 is the Nth object distance, it is classified into class N. By repeating the same process, a classification table shown, for example, in H20 in FIG. 22 is constructed. Note that in the table shown in H20, the probability PR(1) is the probability that the estimated object distance associated with the estimated object distance label 1070 is the first object distance. Similarly, the probability PR(N) is the probability that the estimated object distance associated with the estimated object distance label 1070 is the Nth object distance.

[0083] Then, in the second correction learning process (step S1132), a process of calculating an error function based on the above-mentioned table H20 and the table shown in H30, and a process of updating the weighting coefficients of the neural network NN so as to reduce the error function are performed. The table shown in H30 is a table in which the probability that the estimated object distance associated with estimated object distance label 1070 is the Mth object distance associated with correct object distance label 1076 is set to 1, and the probability that the estimated object distance associated with estimated object distance label 1070 is an object distance other than the Mth object distance is set to 0.

[0084] Through such optimization of the parameters of the neural network NN, the training model 1020 is updated as the trained model 1120. Then, inference using the updated trained model 1120 is performed as shown in FIG. 23. That is, when the image to be processed is input to the input layer of the neural network NN included in the trained model 1120, calculations are performed by the intermediate layer. Then, based on the calculation results by the intermediate layer, an output image 134 is output from the first output layer, and an estimated object distance label 1070 is output from the second output layer. In other words, the trained model 1120 in FIG. 23 is composed of a single neural network NN whose output layer is branched into a first output layer and a second output layer, and the output image 134 and the estimated object distance label 1070 are output from the single neural network NN.

[0085] The blur adjustment process (step S1032) in Fig. 18 will be described in more detail using the flowchart in Fig. 24. The processing unit 1130 performs a process (step S1033) to acquire a characteristic filter from the correction table 1400 based on the estimated object distance, and performs a process (step S1034) to adjust the blur of the output image 134 based on the acquired characteristic filter.

[0086] Correction table 1400 is a table that associates object distances with filter characteristics for blur correction. More specifically, as shown in H10 in FIG. 25, for example, correction table 1400 is a table of a set of correction coefficients corresponding to combinations of object distances associated with estimated object distance labels 1070 output from trained model 1120 and frequencies at image heights perpendicular to the optical axis. Note that the "object distance" shown in the rows of correction table 1400 is a simplified notation for the object distance associated with estimated object distance label 1070 output from trained model 1120.

[0087] For example, in step S1033, the processing unit 1130 searches the correction table 1400 for correction coefficient data associated with the same object distance as the object distance of the estimated object distance label 1070 generated in the correction process (step S1030). Then, in step S1034, the processing unit 1130 adjusts the blur of the output image 134 using the searched correction coefficient. Furthermore, for example, when the processing unit 1130 searches for correction coefficient data in the row indicated by H11, the processing unit 1130 may convert the searched correction coefficient data into a frequency response function indicated by H12 based on the searched correction coefficient data. In this case, the frequency response function indicated by H12 corresponds to the characteristic filter of step S1033. Then, in step S1034, the processing unit 1130 performs a predetermined filter process on the output image 134 using the frequency response function indicated by H12. Note that the characteristic filter related to step S1033 is not limited to a frequency response function, and details will be described later with reference to FIGS. 57 and 58.

[0088] The frequency on the horizontal axis of the graph shown in H12 of FIG. 25 is a simplified representation of the normalized frequency. In other words, the horizontal axis of the graph shown in H12 of FIG. 25 is the same as the horizontal axis of the graphs in FIGS. 26 and 27 described below.

[0089] In this way, the blur-adjusted image 1034, which is the image obtained by performing the blur adjustment process (step S1032) on the output image 134 by the processing unit 1130, is output as a corrected image from the output unit 1150 in step S1040 of FIG. 18. Note that the output destination of the blur-adjusted image 1034 is a predetermined display device, but this is not limiting, and the output destination of the corrected image may also be a storage device of an external device, etc. Note that the predetermined display device is specifically, for example, the display unit 330 shown in FIG. 14, etc. The same applies to the information processing system 100 described above in FIG. 2.

[0090] As described above, the information processing system 1000 of this embodiment includes a storage unit 1110 that stores a trained model 1120 that has been machine-learned using a dataset including training images 32, ground truth images 36, and object distance labels, a correction table 1400 that associates object distance, which is the distance between the imaging system and the subject, with filter characteristics for blur correction, and a processing unit 1130. The training image 32 is generated by performing a defocus simulation process on a focused predetermined subject image 30 captured by an arbitrary imaging system 104, simulating the effect of blur due to defocus of the first imaging system 101 based on the transfer function or point spread function of the first imaging system 101 at a predetermined object distance. The defocus simulation process is performed on an area on the optical axis of the first imaging system 101 and an area other than the optical axis in the training image based on the transfer function or point spread function on the optical axis. The target image 36 is an image generated by performing a best focus simulation process that simulates a state in which the first imaging system 101 is in focus on the predetermined object image 30 based on a transfer function or a point spread function at an object distance at which the first imaging system 101 is in focus, or is the predetermined object image 30 itself. The trained model 1120 is machine-trained to train the training image 32 to become the target image 36, and is machine-trained by assigning the object distance of the transfer function or point spread function of the first imaging system 101 used in the defocus simulation process as an object distance label. The processing unit 1130 uses the trained model 1120 to generate an output image 134 by correcting the blur of the processing target image, which is an image captured by the first imaging system 101, and estimates the object distance of the processing target image. The processing unit 1130 also acquires filter characteristics associated with the estimated object distance from the correction table 1400 and performs blur adjustment processing on the output image 134 using the acquired filter characteristics.

[0091] As described above, the information processing system 1000 of this embodiment includes the memory unit 110 and the processing unit 130 that store the trained model 1120 and the correction table 1400, and therefore can output a corrected image in which the effects of blurring are more appropriately corrected from the image to be processed. This allows the depth of field of the first imaging system 101 to be substantially expanded. Furthermore, since the defocus simulation process (step S200) is performed on the on-axis region of the first imaging system 101 and the off-axis region in each training image 32 based on the on-axis transfer function or point spread function, the amount of information required for the defocus simulation process (step S200) can be reduced. This allows the trained model 1120 to be created with an appropriate size of the neural network NN required for machine learning. This allows the trained model 1120 to be easily implemented in the information processing system 1000.

[0092] The above-described technique may also be realized as the endoscope system 300 described above with reference to Fig. 14 or 15. That is, the endoscope system 300 of this embodiment includes the above-described information processing system 1000 and an endoscope 310 that captures the image to be processed. In this way, the same effects as those described above can be obtained.

[0093] The technique of this embodiment may also be implemented as an image processing method. That is, the technique of this embodiment relates to an image processing method that uses a trained model 1120 that has been machine-learned using a dataset including training images 32, ground truth images 36, and object distance labels, and a correction table 1400 that associates object distance, which is the distance between the imaging system and the subject, with filter characteristics for blur correction. The image processing method includes a step of generating an output image 134 by correcting blur in a processing target image, which is an image captured by the first imaging system 101, using the trained model 1120, and a step of estimating the object distance of the processing target image (step S1030). The image processing method also includes a step of acquiring filter characteristics associated with the estimated object distance from the correction table 1400, and performing blur adjustment processing on the output image 134 using the acquired filter characteristics (step S1032). The training image 32 is generated by performing a defocus simulation process on a focused predetermined object image 30 captured by an arbitrary imaging system 104, simulating the effect of blur due to defocusing of the first imaging system 101 based on the transfer function or point spread function of the first imaging system 101 at a predetermined object distance. The defocus simulation process is performed on an area on the optical axis of the first imaging system 101 and an area off the optical axis of the training image 32 based on the transfer function or point spread function on the optical axis. The target image 36 is an image generated by performing a best focus simulation process on the predetermined object image 30, simulating a focused state of the first imaging system 101 based on the transfer function or point spread function at an object distance at which the first imaging system 101 is focused, or is the predetermined object image 30 itself. The trained model 1120 is machine-trained so that the training image 32 becomes the target image 36, and is machine-trained by assigning the object distance of the transfer function or point spread function of the first imaging system 101 used in the defocus simulation process as an object distance label. By doing so, the same effect as above can be obtained.

[0094] The above-described method may also be realized as a program. That is, the program of this embodiment causes a computer to execute the above-described image processing method. The method of this embodiment may also be realized as a non-transitory information storage medium including the above-described program. In this way, the same effects as those described above can be obtained.

[0095] Furthermore, in the information processing system 1000 of this embodiment, the trained model 1120 is configured by one neural network NN, and the neural network NN may include an input layer to which the image to be processed is input, an intermediate layer that performs calculations on the output of the input layer, a first output layer that generates the output image 134 from the output of the intermediate layer, and a second output layer that estimates the object distance from the output of the intermediate layer. This can prevent an increase in the network size.

[0096] The method of this embodiment is not limited to the above, and various modifications are possible. For example, each object distance included in the optical system information 40 may be determined based on the difference between the corresponding MTFs. For example, the training image group 32G is assumed to be composed of a first training image 32-1 obtained by performing step S200-1 based on the transfer function or point spread function of the first object distance, and a second training image 32-2 obtained by performing step S200-2 based on the transfer function or point spread function of the second object distance. The first object distance is assumed to have a larger defocus amount than the second object distance. In this case, the spatial frequency dependence of the MTF is qualitatively illustrated as A0 in FIG. 26 , with the MTF based on the second object distance as shown, and the MTF based on the first object distance as shown, A1. Then, for example, by determining a predetermined spatial frequency indicated by B0, the difference between the MTFs is determined as indicated by C0. Therefore, the first object distance and the second object distance are determined so that the difference between the MTFs indicated by C0 is smaller than a predetermined value.

[0097] The MTF difference here refers to the difference between MTFs at adjacent object distances. For example, assume that the training image group 32G includes a first training image 32-1, a second training image 32-2, and a third training image 32-3. Assume that the object distances with the largest defocus amounts are the first, second, and third object distances. In this case, A10 in FIG. 27 indicates the frequency characteristics of the MTF at the third object distance, A11 indicates the frequency characteristics of the MTF at the second object distance, and A12 indicates the frequency characteristics of the MTF at the first object distance. At a predetermined frequency indicated by B0, both the difference between the MTFs of A10 and A11 indicated by C10 and the difference between the MTFs of A11 and A12 indicated by C11 are lower than a predetermined value. In other words, at the predetermined frequency indicated by B0, the difference between the MTFs of A10 and A12 is not considered a predetermined value. For the above reasons, in the information processing system 100 of this embodiment, the object distance is set so that the difference in MTF values ​​between adjacent object distances is equal to or less than a predetermined value at a predetermined spatial frequency of the MTF of the first imaging system 101. This allows for an appropriate combination of datasets in machine learning. As described above, the machine-learned trained model 120 performs a correction process (step S30) so that both the first training image 32-1 and the second training image 32-2 can be corrected to the correct image 36. Furthermore, to correct a processing target image captured at an object distance between the first object distance and the second object distance to the correct image 36 by the correction process (step S30), it is preferable that the difference in the influence of blur added to the first training image 32-1 and the second training image 32-2 be within a certain range. In this regard, by applying the method of this embodiment, the object distance of each training image is defined based on the MTF indicating the degree of influence of blur simulated on the predetermined subject image 30, and therefore an appropriate training image group 32G can be generated. This allows for an appropriate dataset to be obtained in machine learning.

[0098] The optical system information 40 may also include the object distance under the best focus condition of the first imaging system 101. Specifically, the object distance under the best focus condition is, for example, the distance shown as D3 in FIG. 9. For example, the learning device processing unit 16 may generate the correct image 36 by performing a best focus simulation process (step S300) on the predetermined subject image 30 using a transfer function or a point spread function that uses the object distance under the best focus condition. In other words, in the information processing system 100 of this embodiment, the object distance at which focus is achieved is the object distance under the best focus condition. In this way, an appropriate correct image 36 can be generated.

[0099] In this embodiment, it is assumed that there is a one-to-one correspondence between the transfer function or point spread function based on the object distance and the training image 32. More specifically, for example, in the defocus simulation process (step S200), a process of generating a third training image 32-3 using both a transfer function or point spread function based on a first object distance and a transfer function or point spread function based on a second object distance for one predetermined object image 30 is not performed. In other words, in the information processing system 100 of this embodiment, each training image 32 is an image generated by performing the defocus simulation process (step S200) on a predetermined object image 30 based on a transfer function or point spread function at any one of a plurality of object distances. This makes it possible to clarify the relationship between each training image 32 in the training image group 32G.

[0100] In a typical optical system, the MTF decreases and changes periodically as the spatial frequency increases. However, since the MTF is an absolute value, the MTF is displayed while being aliased in the high spatial frequency region shown in B1 of FIG. 27. Therefore, in the high spatial frequency region, it is not possible to uniquely determine which object distance corresponds to an MTF of 1. Note that, for example, the MTF for an object distance shorter than the object distance at the near point of the target extended depth of field shown in P2 of FIG. 9 may be 0 at the spatial frequency shown in B0. For example, assuming that A12 in FIG. 27 is the MTF at the object distance at the near point of the target extended depth of field, the spatial frequency shown in B0 should be lower than the lowest spatial frequency at which aliasing occurs. This is because the transfer function or point spread function for object distances outside the target depth of field are not used in the machine learning of this embodiment. Note that, as described above, the target extended depth of field here does not indicate a constant value. From the above, in the information processing system 100 of this embodiment, the processing unit 130 uses the trained model 120 to correct blur caused by defocusing of the first imaging system 101 for the processing target image, thereby estimating an image in which the depth of field of the first imaging system 101 is extended to a target extended depth of field that is wider than the depth of field. Furthermore, the predetermined spatial frequency is a spatial frequency lower than the lowest spatial frequency at which the MTF value at the near point of the target extended depth of field becomes zero. This makes it possible to appropriately determine the range of predetermined spatial frequencies required to establish a one-to-one correspondence between spatial frequencies and MTFs.

[0101] More specifically, the predetermined spatial frequency indicated by B0 is preferably 0.1 as a normalized frequency, for example. That is, in the information processing system 100 of this embodiment, the predetermined spatial frequency is a spatial frequency that is 1 / 5 the Nyquist frequency of the image sensor of the first imaging system 101. By doing so, it is possible to establish a one-to-one correspondence between spatial frequency and MTF for many optical systems. This allows the method of this embodiment to be applied to processing target images captured by many types of optical systems.

[0102] Furthermore, the optical system information 40 of this embodiment may be a combination of an object distance within the depth of field and an object distance outside the depth of field. Specifically, for example, the optical system information 40 may include a first object distance outside the depth of field as shown in D1 of FIG. 9 and a second object distance as shown in D2. In other words, in the information processing system 100 of this embodiment, the first object distance among the multiple object distances is an object distance outside the depth of field, and the second object distance among the multiple object distances is an object distance within the depth of field. By doing so, a data set can be created by combining the first training image 32-1, which simulates a large effect of blur, and the second training image 32-2, which simulates a small effect of blur, with the correct image 36 through the defocus simulation process (step S200). This allows the trained model 120, which has been machine-learned using these data sets, to correct a target image affected by blur over a wide range through the correction process (step S30).

[0103] The predetermined value may also be determined based on the number of training images 32 constituting the training image group 32G. For example, in FIG. 26, the MTF indicated by A0 is the MTF at the object distance corresponding to the best focus condition, and the MTF indicated by A1 is the MTF at the object distance corresponding to the near point of the target depth of field. In this case, for example, if the spatial frequency is determined to be the spatial frequency indicated by B0, the MTF range having the range indicated by C0 as the maximum is uniquely determined. The range indicated by C0 is then divided based on the number of desired training images 32, and the predetermined value is determined. For these reasons, in the information processing system 100 of this embodiment, the predetermined value is determined based on the number of object distances that can be set to two or more. In this way, the number of datasets required for machine learning can be determined taking into account the load of machine learning.

[0104] As mentioned above, since the range of the MTF is uniquely determined when the spatial frequency is fixed, a predetermined value may be determined in advance and the number of training images 32 may be determined based on that predetermined value, and the user may decide the machine learning policy according to the circumstances.

[0105] The predetermined value is preferably 0.2 or less. That is, in the information processing system 100 of this embodiment, the predetermined value is set to 0.2 or less. In a typical optical system, if the spatial frequency described above is determined to be within a desirable range, the possible range of MTF is considered to be approximately 0.2. Therefore, for example, if the predetermined value is set to 0.2, the number of training images 32 constituting the training image group 32G will be 2. In this case, the first object distance is considered to be an object distance outside the depth of field, and the second object distance is considered to be an object distance within the depth of field.

[0106] Furthermore, the predetermined value is preferably 0.1 or less. That is, in the information processing system 100 of this embodiment, the predetermined value is set to 0.1 or less. Furthermore, the predetermined value is preferably 0.05 or less. That is, in the information processing system 100 of this embodiment, the predetermined value is set to 0.05 or less. By doing so, the number of training images 32 constituting the training image group 32G can be further increased. As a result, when a processing target image captured at an object distance other than the object distance not used in machine learning is input, the trained model 120 is more likely to be able to output a corrected image in which the effects of blur have been appropriately removed. That is, the accuracy of the correction process (step S30) of the trained model 120 can be further improved. Note that, as the number of training images 32 constituting the training image group 32G increases, the processing load of machine learning increases. Therefore, the appropriate number of training images 32 constituting the training image group 32G is determined appropriately depending on the circumstances.

[0107] Next, a specific method in which the learning device processing unit 16 performs the defocus simulation process (step S200) using the point spread function will be described. For example, when generating a first learning image 32-1 in step S200-1, as shown in FIG. 28, the learning device processing unit 16 performs a convolution operation on the predetermined object image 30 using the PSF of the first object distance of the first imaging system 101. Note that convolution can also be called a convolution integral. The PSF of the first object distance here is a PSF consisting of the region shown as FC22-1 in FIG. 12. That is, in the case of the method shown in FIG. 28, the convolution operation of the PSF corresponds to step S210 in FIG. 13. Similarly, when generating an Nth learning image 32-N in step S200-N, the learning device processing unit 16 performs a convolution operation on the predetermined object image 30 using the PSF of the Nth object distance of the first imaging system 101. The defocus simulation process based on the convolution calculation process of the PSF can be referred to as step S200-A. From the above, in the information processing system 100 of this embodiment, the defocus simulation process (step S200) is a process of performing convolution calculation of the PSF at each object distance of the first imaging system 101 for the predetermined subject image 30. In this way, it is possible to generate a trained model 120 that has been machine-learned using a dataset of the training images 32 and the ground truth images 36 using the PSF.

[0108] Next, a specific method in which the learning device processing unit 16 performs the defocus simulation process (step S200) using a transfer function will be described. For example, when generating a first learning image 32-1, as shown in FIG. 29, the learning device processing unit 16 performs a Fourier transform on the predetermined object image 30, a process of multiplying the frequency characteristics resulting from the Fourier transform by the OTF of the first object distance of the first imaging system 101, and a process of performing an inverse Fourier transform on the frequency characteristics obtained by the multiplication. The OTF for the first object distance here is an OTF consisting of the region indicated by FC22-1 in FIG. 12. In other words, in the case of the method shown in FIG. 29, the multiplication of the OTF corresponds to step S210 in FIG. 13. Similarly, when generating an Nth learning image 32-N by step S200-N, the learning device processing unit 16 performs a Fourier transform on the predetermined object image 30, a process of multiplying the frequency characteristics resulting from the Fourier transform by the OTF of the Nth object distance of the first imaging system 101, and a process of performing an inverse Fourier transform on the multiplied frequency characteristics. Note that the defocus simulation process based on the multiplication of the OTF can be referred to as step S200-B. From the above, in the information processing system 100 of this embodiment, the defocus simulation process (step S200) is a process of performing a Fourier transform on the predetermined object image 30, multiplying the frequency characteristics of the predetermined object image 30 resulting from the Fourier transform by the OTF at each object distance of the first imaging system 101, and performing an inverse Fourier transform on the multiplied frequency characteristics. In this way, it is possible to generate a trained model 120 that has been machine-learned using a dataset of the training images 32 and the ground truth images 36 using the OTF.

[0109] Since the relationship between PSF and OTF is as described above, the calculation processing result related to the processing in Fig. 28 is mathematically equivalent to the calculation processing result related to the processing in Fig. 29. In the defocus simulation processing (step S200), the user can select either PSF or OTF to use as appropriate.

[0110] Similarly, the learning device processing unit 16 may perform a best focus simulation process (step S300) using a point spread function. For example, as shown in Fig. 30, the learning device processing unit 16 performs a convolution calculation process on the predetermined subject image 30 using the PSF of the object distance at which the first imaging system 101 is in focus, thereby generating a ground truth image 36. Note that the best focus simulation process based on the convolution calculation process of the PSF can also be referred to as step S300-A.

[0111] The learning device processing unit 16 may also perform a best focus simulation process (step S300) using a transfer function. For example, as shown in Fig. 31, the learning device processing unit 16 performs a Fourier transform on the predetermined object image 30, a process of multiplying the frequency characteristics resulting from the Fourier transform by the OTF of the object distance at which the first imaging system 101 is in focus, and a process of performing an inverse Fourier transform on the frequency characteristics resulting from the multiplication, thereby generating a ground truth image 36. The best focus simulation process based on the multiplication of the OTF may also be referred to as step S300-B.

[0112] In the following explanation, a representative example will be given in which a method using a PSF is applied to generate the training image 32 and the correct image 36, but this does not preclude the application of a method using an OTF.

[0113] Furthermore, for example, the first imaging system 101 of this embodiment may have a retrofocus type lens configuration. The retrofocus type is also called a reverse telephoto type. For example, a retrofocus type lens configuration is realized by arranging a lens with negative refractive power and a lens with positive refractive power from the subject side. In the following description, the lens group on the subject side will be called the front lens group, and the lens group on the image side will be called the rear lens group.

[0114] A variety of well-known lens configurations can be employed for the specific lens configuration of a retrofocus type. For example, in the case of the optical system shown in FIG. 32, the optical system is configured, in order from the subject side, with a front lens group indicated by G1, an aperture stop indicated by S1, a rear lens group indicated by G2, and a cover glass indicated by CG1. Note that in FIG. 32, for the sake of convenience, the spacing between the lenses and other components of the optical system is not precisely illustrated. For example, in FIG. 32, the positive lens indicated by L6 and the cover glass indicated by CG1 are actually cemented together, but are illustrated with a gap between them for the sake of convenience. The same applies to FIGS. 33 and 35, which will be described later.

[0115] 32, the front lens group shown in G1 includes an object-side negative lens shown in L1 and a positive lens shown in L2, and has a negative refractive power as a whole. The rear lens group shown in G2 includes a positive lens shown in L3, a lens formed by cementing a positive lens and a negative lens shown in L4 and a positive lens shown in L5, and has a positive refractive power as a whole.

[0116] The front lens group or the rear lens group may be composed of multiple lens groups. For example, in the first imaging system 101 shown in Fig. 33, the lens group indicated by G11 functions as the front lens group, and the lens groups indicated by G12 and G13 function as the rear lens groups. For example, the lens group indicated by G11 includes, in order from the subject side, a plano-concave lens with its concave surface facing the image side as indicated by L11, and a negative meniscus lens as indicated by L12, and has negative refractive power as a whole.

[0117] Furthermore, for example, the lens group shown in G12 includes an object-side positive lens shown in L13 and an image-side positive lens shown in L14. Note that an aperture stop shown in S11 may be further disposed between the lens shown in L13 and the lens shown in L14. By doing so, the optical system is configured so that the refractive power is symmetrical on either side of the aperture stop, thereby enabling good correction of coma and astigmatism.

[0118] The lens group indicated by G13 has a positive refractive power as a whole. The lens group indicated by G13 may also include a cemented lens composed of a positive lens indicated by L15 and a negative lens indicated by L16. This allows for good correction of spherical aberration and coma. The lens group indicated by G13 may also include a plano-convex lens indicated by L17. This allows for a wide field of view. Although the plano-convex lens indicated by L17 and the cover glass indicated by CG11 are shown separated in FIG. 33, they are actually cemented together. The cover glass indicated by CG11 is provided on an image sensor (not shown), and the plano-convex lens indicated by L17 is used to position the image sensor.

[0119] Furthermore, for example, the first imaging system 101 may further include a parallel plate. The parallel plate is also called a filter. The parallel plate is disposed, for example, at position F1 in FIG. 32 and position F11 in FIG. 33, but it can also be disposed at other positions. The parallel plate is used, for example, to adjust the position of the image point.

[0120] Furthermore, in the first imaging system 101 including the retrofocus lens configuration described above, it is desirable that the amount of distortion at the maximum angle of view be −30% or less. Specifically, for example, assume that the object shown in E1 of FIG. 34 is captured as the image shown in E2 of FIG. 34 by the first imaging system 101. In this case, the value of the amount of distortion (%) at the maximum angle of view can be expressed as (AD−PD) / PD×100, using the length shown in PD of the object shown in E1 and the length shown in AD of the image shown in E2. It is desirable that this value be negative and less than −30. From the above, in the information processing system 100 of this embodiment, the first imaging system 101 has a retrofocus lens configuration, and the amount of distortion at the maximum angle of view is −30% or less. This configuration reduces the magnification of the periphery compared to the center of the image, thereby reducing the transfer function or point spread function of regions other than on the optical axis. Furthermore, it is possible to reduce the difference between the transfer function or point spread function of the region on the optical axis and the transfer function or point spread function of the region other than on the optical axis. This allows the generation of training images 32 that simulate the effects of blur more accurately.

[0121] The front lens group or the rear lens group may be composed of a single lens. For example, the first imaging system 101 shown in FIG. 35 includes a lens group indicated by G21, a lens group indicated by G22, an aperture stop indicated by S21, a lens group indicated by G23, and a cover glass indicated by CG21. The lens group indicated by G21 includes a single negative lens indicated by L21 and has negative refractive power. In other words, the lens group indicated by G21 functions as part of the front lens group. The lens group indicated by G23 includes a positive lens indicated by L23, a lens formed by cementing a positive lens indicated by L24 and a negative lens indicated by L25, and a positive lens indicated by L26, and has positive refractive power as a whole. In other words, the lens group indicated by G23 functions as the rear lens group.

[0122] Furthermore, the first imaging system 101 of this embodiment may further include a phase modulation element. For example, the second lens group G2 in Fig. 35 includes a positive lens indicated by L22, an aperture stop indicated by S21, and a phase modulation element indicated by PM. The phase modulation element indicated by PM is disposed at the position of the pupil of the first imaging system 101. The phase modulation element indicated by PM is an element to which wavefront coding (WFC) is applied, and has, for example, a phase modulation surface indicated by PMS. Note that wavefront coding is a well-known technique used in extended depth of field (EDOF), and therefore a detailed description thereof will be omitted.

[0123] In Fig. 35, the phase modulation surface indicated by PMS is shown as being expressed by a predetermined cubic function using coordinates orthogonal to the optical axis, but the surface shape of the phase modulation surface is not limited to this, and other surface shapes may be adopted. Also, in Fig. 35, the phase modulation surface is shown on the image side, but the same effect can be obtained even if it is located on the subject side. Furthermore, the lens group indicated by G22 has positive refractive power as a whole and also functions as part of a front lens group of a retrofocus type.

[0124] Furthermore, by including the phase modulation element shown in PM, the MTF of the first imaging system 101 changes less rapidly with defocus. In other words, by including the phase modulation element, the MTF of the first imaging system 101 acts to match the change in object distance. More specifically, for example, the difference between the MTF at the first object distance and the MTF at the second object distance in the first imaging system 101 that includes the phase modulation element is smaller than the difference between the MTF at the first object distance and the MTF at the second object distance in the first imaging system 101 that does not include the phase modulation element.

[0125] For example, in the relationship between MTF and spatial frequency shown in FIG. 36, A20 is the MTF of the first imaging system 101 at an in-focus object distance, A21 is the MTF at an object distance with a larger defocus amount than the object distance associated with A20, and A22 is the MTF at an object distance with a larger defocus amount than the object distance associated with A21. Also, A20 to A22 are the MTFs of the first imaging system 101 that does not include a phase modulation element. When the predetermined spatial frequency indicated by B0 is determined, the difference between the MTFs of A20 and A21 is the difference indicated by C20, and the difference between the MTFs of A21 and A22 is the difference indicated by C21. Note that in FIG. 36, the MTFs at frequencies higher than the spatial frequency indicated by B0 are partially omitted.

[0126] Here, by including the phase modulation element shown in PM in the first imaging system 101, the MTF shown in A20 changes to the MTF shown in A30, the MTF shown in A21 changes to the MTF shown in A31, and the MTF shown in A22 changes to the MTF shown in A32. Furthermore, the difference in the MTFs shown in C20 becomes smaller as shown in C30, and the difference in the MTFs shown in C21 becomes smaller as shown in C31. From the above, in the information processing system 100 of this embodiment, the first imaging system 101 further includes an optical wavefront modulation element that changes the transfer function or point spread function. This shortens the distance required for machine learning, thereby reducing the number of data sets required for machine learning.

[0127] The above-described example of the defocus simulation process (step S200) is an example of a process for generating a predetermined object image 30 based on optical information from the first imaging system 101 for the predetermined object image 30 captured by an arbitrary imaging system 104, but the method of this embodiment is not limited to this. For example, the learning device processing unit 16 may perform a defocus simulation process that further includes a process for simulating the removal of the influence of imaging by the arbitrary imaging system 104 from the predetermined object image 30.

[0128] 37 shows an example of image data generation processing in the case where a process for simulating the removal of the influence of the first imaging system 101 on the predetermined object image 30-1 captured by the first imaging system 101 is further included. The image data generation processing shown in FIG. 37 can also be referred to as step S122. Comparing step S122 in FIG. 37 with step S120-2 in FIG. 11, the content of the defocus simulation processing is different. FIG. 37 is similar to FIG. 11 in that the best focus simulation processing (step S300) is not performed and the target image 36 is the predetermined object image 30-1 itself. This is because the predetermined object image 30-1 is an image captured under the best focus conditions of the first imaging system 101, and therefore there is no need to perform processing similar to step S202.

[0129] 38 shows an example of the defocus simulation process (step S202-1) in the image data generation process (step S122). For example, when generating the first learning image 32-1, the learning device processing unit 16 performs a process (step S220-1) on the predetermined object image 30-1 to simulate the removal of the influence of the first imaging system 101 when capturing the predetermined object image 30-1. Step S220-1 is performed based on the transfer function or point spread function at the object distance at which the first imaging system 101 is in focus and the transfer function or point spread function of the first imaging system 101 at the first object distance.

[0130] More specifically, the learning device processing unit 16 performs an appropriate combination of, for example, a calculation process for deconvolving the PSF at an object distance where the first imaging system 101 is focused on the predetermined subject image 30 and a calculation process for convolving the PSF at the first object distance of the first imaging system 101 (step S200-A). The appropriate combination of calculation processes refers to a calculation process in which one calculation process and part or all of the other calculation process are combined in any order. However, this does not prevent one calculation process from being performed separately from the other calculation process, and the order is determined appropriately according to predetermined circumstances. The same applies to the following explanation. Note that predetermined circumstances include, for example, the processing time required for machine learning, the processing load on the processor, etc. In other words, by performing step S220-1, it is possible to obtain a calculation result that reflects both the effect of the calculation process that deconvolves the PSF at the object distance at which the first imaging system 101 is focused on the specified subject image 30-1, and the effect of the calculation process (step S200-A) that convolves the PSF at the first object distance of the first imaging system 101.

[0131] From the above, in the information processing system 100 of this embodiment, the arbitrary imaging system 104 is the first imaging system 101. The defocus simulation process (step S202) further includes a process (step S212) of removing the influence of the first imaging system 101 from the predetermined object image 30-1 based on the transfer function or point spread function at the object distance where the first imaging system 101 is in focus and the transfer functions or point spread functions at multiple object distances of the first imaging system 101. This allows for the generation of a more accurate training image 32. The training image 32 and the correct image 36 obtained by the methods shown in FIGS. 10 and 11 are influenced by both the arbitrary imaging system 104 and the first imaging system 101 on the predetermined object, whereas the training image 32 and the correct image 36 obtained by the methods shown in FIGS. 37 and 38 are influenced by only the first imaging system 101 on the predetermined object. This allows for machine learning using a more appropriate dataset.

[0132] Similarly, an example of image data generation processing including processing that simulates the removal of the influence of imaging by an arbitrary imaging system 104 is shown in Fig. 39. In Fig. 39, the second imaging system 102 is illustrated as a representative of the arbitrary imaging system 104. The second imaging system 102 is also assumed to be an imaging system having an imaging element with a higher resolution than the first imaging system 101. The image data generation processing shown in Fig. 39 can also be referred to as step S124, and the image that is the source of step S124 can also be referred to as a predetermined subject image 30-2.

[0133] 10, step S126 in FIG. 39 differs from step S120-1 in that image sensor information 50 is further read, and then a defocus simulation process (step S204) and a best focus simulation process (step S304) are performed. The image sensor information 50 is information related to the resolution of the image sensors of the first image sensor system 101 and any image sensor system 104. That is, in the example of FIG. 39, the learning device storage unit 18 further stores image sensor information 50, which is not shown in FIG. 4. The image sensor information 50 is also used in the calculation processes of the defocus simulation process (step S204) and the best focus simulation process (step S304).

[0134] FIG. 40 shows an example of defocus simulation processing in the image data generation processing (step S124) shown in FIG. 39. The defocus simulation processing shown in FIGS. 39 and 40 can also be referred to as step S204. For example, when generating a first learning image 32-1, the learning device processing unit 16 performs arithmetic processing on a predetermined object image 30-2, appropriately combining a process of simulating the difference between the second imaging system 102 and the first imaging system 101 (step S230-1), a process of reducing the predetermined object image 30 (step S240), and arithmetic processing based on image sensor information 50 (not shown in FIG. 40). Step S230-1 is performed based on the transfer function or point spread function at the object distance at which the second imaging system 102 is in focus and the transfer function or point spread function of the first imaging system 101 at the first object distance. That is, by performing step S230-1, it is possible to obtain a calculation processing result that reflects both the effect of the calculation processing that deconvolves the PSF at the object distance where the second imaging system 102 is focused on the predetermined subject image 30-2, and the effect of the calculation processing (step S200-A) that convolves the PSF at the first object distance of the first imaging system 101. Furthermore, by performing step S204-1, it is possible to obtain a calculation processing result that reflects the effect of the calculation processing of step S230-1, the effect of the calculation processing of step S240, and the effect of the calculation processing based on the image sensor information 50.

[0135] FIG. 41 shows an example of the best focus simulation process shown in FIG. 39. The best focus simulation process shown in FIGS. 39 and 41 can also be referred to as step S304. For example, the learning device processing unit 16 performs a process that appropriately combines, for the predetermined object image 30-2, a process of simulating the difference between the second imaging system 102 and the first imaging system 101 (step S330), a process of reducing the predetermined object image 30-2 (step S340), and a calculation process based on the image sensor information 50 (not shown in FIG. 41). This allows the learning device processing unit 16 to generate the correct image 36. Step S330 in FIG. 41 is performed based on the transfer function or point spread function at the object distance at which the second imaging system 102 is in focus and the transfer function or point spread function at the object distance at which the first imaging system 101 is in focus. That is, by performing step S330, it is possible to obtain a calculation processing result that reflects both the effect of the calculation processing of deconvolving the PSF at the object distance at which the second imaging system 102 focuses on the predetermined object image 30-2 and the effect of the calculation processing (step S300-A) of convolving the PSF at the object distance at which the first imaging system 101 focuses. Furthermore, step S340 in FIG. 41 is a calculation processing similar to step S240 in FIG. 40. Furthermore, by performing step S304, it is possible to obtain a calculation processing result that reflects the effect of the calculation processing of step S330, the effect of the calculation processing of step S340, and the effect of the calculation processing based on the image sensor information 50. Note that the reference image 36 may be generated by a process that omits step S330 from the best focus simulation process (step S304) in FIG. 41. In other words, the reference image 36 may be generated by performing a process equivalent to step S340 on the predetermined object image 30-2. If the specified subject image 30-2 is an image captured at an object distance at which any imaging system 104 is in focus, it may be possible to make the specified subject image 30-2 into the correct image 36 by changing the number of pixels in step S340.

[0136] From the above, in the information processing system 100 of this embodiment, the defocus simulation process (step S204) further includes a process (step S230) of simulating the difference between the arbitrary imaging system 104 and the first imaging system 101, and a process (step S240) of reducing the predetermined object image 30-2. The target image 36 is an image generated by performing the best focus simulation process (step S304), or an image generated by performing a process of reducing the predetermined object image 30-2. The process (step S230) of simulating the difference between the arbitrary imaging system 104 and the first imaging system 101 in the defocus simulation process (step S204) is based on a transfer function or a point spread function at an object distance at which the arbitrary imaging system 104 is in focus, and a transfer function or a point spread function at a plurality of object distances of the first imaging system 101. The best focus simulation process (step S304) further includes a process (step S330) of simulating the difference between the arbitrary imaging system 104 and the first imaging system 101, and a process (step S340) of reducing the predetermined subject image 30-2. The process (step S330) of simulating the difference between the arbitrary imaging system 104 and the first imaging system 101 in the best focus simulation process (step S304) is based on a transfer function or a point spread function at an object distance where the arbitrary imaging system 104 is in focus, and a transfer function or a point spread function at an object distance where the first imaging system 101 is in focus.

[0137] The method of this embodiment can also be applied when the imaging system 104 and the first imaging system 101 have different imaging methods. For example, as shown in FIG. 42, the first imaging system 101 includes a simultaneous imaging element 106. Also, as shown in FIG. 43, the imaging system 104 includes a monochrome imaging element 108. The image data generation process in this case will be described with reference to FIG. 43. The image data generation process in FIG. 43 can also be referred to as step S126, and the image used as the source of step S126 can also be referred to as a predetermined subject image 30-3. FIG. 43 differs from FIG. 39 in the contents of the defocus simulation process (step S206) and the best focus simulation process (step S306), and in the fact that a color shift determination process (step S190) is performed before steps S206 and S306. In FIG. 43, the second imaging system 102 is illustrated as a representative example of the imaging system 104, just like the example in FIG. 39. The color shift determination process (S190) compares the amount of color shift around saturated areas in the predetermined subject image 30-3 with a predetermined threshold. Note that color shift refers to the shift that occurs between the R, G, and B images due to differences in image capture timing when capturing an image of a subject using the monochrome image sensor 108. Note that color shift does not occur in the processing target image captured using the simultaneous image sensor 106. The amount of color shift around saturated areas in the predetermined subject image 30-3 refers to the amount of color shift that occurs around areas that appear white in the predetermined subject image 30-3. In other words, steps S206 and S306 in FIG. 43 use the predetermined subject image 30-3 for which the amount of color shift around saturated areas is determined to be below the predetermined threshold in step S190. Thus, by performing step S206, a training image 32 with reduced effects of color shift can be generated. Similarly, by performing step S306, a target image 36 with reduced effects of color shift can be generated. This makes it possible to generate a data set consisting of appropriate training images 32 and correct images 36 when the imaging methods of the arbitrary imaging system 104 and the first imaging system 101 are different.

[0138] FIG. 44 shows an example of defocus simulation processing in the image data generation processing (step S126) shown in FIG. 43. The defocus simulation processing shown in FIGS. 43 and 44 can also be referred to as step S206. FIG. 44 differs from FIG. 40 in that it further includes processing for generating a mosaic image from the predetermined object image 30-3 (step S250) and processing for demosaicing the mosaic image (step S252). For example, when generating the first learning image 32-1, the learning device processing unit 16 performs arithmetic processing on the predetermined object image 30-3, which is an appropriate combination of the above-described steps S230-1, S240, S250, and S252, as well as arithmetic processing based on image sensor information 50 (not shown in FIG. 44). In other words, by performing step S206-1, it is possible to obtain a calculation result that reflects the effect of the calculation processing of step S230-1, the effect of the calculation processing of step S240, the effect of the calculation processing of step S250, the effect of the calculation processing of step S252, and the effect of the calculation processing based on the image sensor information 50.

[0139] Steps S250 and S252 will be described in detail. The predetermined object image 30-3 is a frame-sequential image obtained by a process of synthesizing a plurality of images captured by the monochrome image sensor 108 at the timing when light of a plurality of wavelength bands is sequentially irradiated. Then, as shown in FIG. 45, for example, in the above-mentioned step S206-1, a mosaic image is generated by a process including step S250. Then, a frame-sequential image is generated again from the mosaic image by a process including step S252, thereby generating a first learning image 32-1. Note that in step S206-1 of FIG. 45, processes other than steps S250 and S252 are not shown.

[0140] FIG. 46 shows an example of best focus simulation processing in the image data generation processing (step S126) shown in FIG. 43. The best focus simulation processing shown in FIGS. 43 and 46 can also be referred to as step S306. FIG. 46 differs from FIG. 41 in that it further includes processing for generating a mosaic image from the predetermined object image 30-3 (step S350) and processing for demosaicing the mosaic image (step S352). Step S350 in FIG. 46 is similar to step S250 in FIG. 44, and step S352 in FIG. 46 is similar to step S252 in FIG. 44. For example, the learning device processing unit 16 performs arithmetic processing that appropriately combines the above-described step S330-1, the above-described step S340, step S350, step S352, and arithmetic processing based on image sensor information 50 (not shown in FIG. 46). This allows the learning device processing unit 16 to generate the correct image 36. As a result, by performing step S306, it is possible to obtain a calculation processing result that reflects the effect of the calculation processing of step S330, the effect of the calculation processing of step S340, the effect of the calculation processing of step S350, the effect of the calculation processing of step S352, and the effect of the calculation processing based on the image sensor information 50. Note that the reference image 36 may be generated by a process that omits steps S330, S350, and S352 from the best focus simulation process (step S306). In other words, the reference image 36 may be generated by performing a process equivalent to step S340 on the predetermined subject image 30-3.

[0141] From the above, in the information processing system 100 of this embodiment, the arbitrary imaging system 104 includes a monochrome imaging element 108. The predetermined object image 30-3 is a frame sequential image obtained by combining multiple images captured by the monochrome imaging element 108 at the timing when light of multiple wavelength bands is sequentially irradiated. The first imaging system 101 includes a simultaneous imaging element 106 having multiple pixels of different colors, each pixel assigned a single color. The defocus simulation process (step S206) further includes a process of generating a mosaic image from the predetermined object image 30-3, in which each pixel is assigned a single color, a process of demosaicing the mosaic image, a process of simulating the difference between the arbitrary imaging system 104 and the first imaging system 101, and a process of reducing the predetermined object image 30-3. The process of simulating the difference between the arbitrary imaging system 104 and the first imaging system 101 in the defocus simulation process (step S206) is based on the transfer function or point spread function at the object distance at which the arbitrary imaging system 104 is in focus and the transfer function or point spread function at multiple object distances of the first imaging system 101. The ground truth image 36 is an image generated by performing the best focus simulation process (step S306) or an image generated by performing a process of reducing the predetermined object image 30-3. The best focus simulation process (step S306) further includes a process of generating a mosaic image, a process of demosaicing the mosaic image, a process of simulating the difference between the arbitrary imaging system 104 and the first imaging system 101, and a process of reducing the predetermined object image 30-3. The process of simulating the difference between the arbitrary imaging system 104 and the first imaging system 101 in the best focus simulation process (step S306) is based on the transfer function or point spread function at the object distance at which the arbitrary imaging system 104 is in focus, and the transfer function or point spread function at the object distance at which the first imaging system 101 is in focus. In this way, even if the imaging method of the predetermined subject image 30 and the imaging method of the processing target image are different, it is possible to generate a more appropriate data set of training images 32 and ground truth images 36.

[0142] Moreover, different trained models 120 may be used depending on the imaging method. That is, in the information processing system 100 of this embodiment, the storage unit 110 may store a first trained model 121 and a second trained model 122, for example, as shown in FIG.

[0143] When the storage unit 110 stores the first trained model 121 and the second trained model 122, the flow shown in FIG. 3 may be, for example, the flow shown in FIG. 48. After reading the image to be processed (step S10), the processing unit 130 performs a process of checking the imaging method of the first imaging system 101 (step S12). If the imaging method is the frame sequential method, the processing unit 130 reads the first trained model (step S21), performs correction processing (step S31), and outputs the corrected image (step S41). On the other hand, if the imaging method is the Bayer simultaneous imaging method, the processing unit 130 reads the second trained model (step S22), performs correction processing (step S32), and outputs the corrected image (step S42). Note that steps S21 and S22 in FIG. 48 correspond to step S20 in FIG. 3. 3. Similarly, steps S31 and S32 in FIG. 48 correspond to step S30 in FIG. 3, and steps S41 and S42 in FIG. 48 correspond to step S40 in FIG.

[0144] Also, in this case, step S100 in Figure 7 may be replaced by step S101 in Figure 49 and step S102 in Figure 50. Specifically, the first trained model creation process (step S101) in Figure 49 may be replaced by step S100 in Figure 7, with the image data generation process being replaced by step S124 in Figure 39. Similarly, the second trained model creation process (step S102) in Figure 50 may be replaced by step S100 in Figure 7, with the image data generation process being replaced by step S126 in Figure 43.

[0145] The method of this embodiment can also be applied when the observation methods are different between the arbitrary imaging system 104 and the first imaging system 101. A method of image data generation processing when the observation methods are different will be described using FIG. 51 . The image data generation processing in FIG. 51 can also be referred to as step S128, and the image that serves as the source of step S128 can also be referred to as the predetermined subject image 30-4. Step S128 in FIG. 51 differs from step S124 in FIG. 39 in the contents of the defocus simulation processing (step S208) and the best focus simulation processing (step S308), and in that observation method information 60 is read before steps S206 and S306 are performed. The observation method information 60 is, for example, information regarding the observation method of the first imaging system 101. In other words, in the example of FIG. 51 , the learning device storage unit 18 further stores observation method information 60, which is not shown in FIG. 4. 51, the second imaging system 102 is illustrated as a representative example of an arbitrary imaging system 104, just like the example in FIG.

[0146] The observation method can also be called an observation mode. Different observation methods may be, for example, when different light sources are used for observation, or when different image processing techniques are used between the time the user captures an image of the subject and the time the predetermined subject image 30-4 is acquired. Observation methods include, for example, a white light imaging (WLI) mode that uses white illumination light and a special light observation mode that uses special light other than white light. The special light observation mode includes a narrow band imaging (NBI) mode that uses two narrow band lights. The two narrow band lights are narrow band light included in the blue wavelength band and narrow band light included in the green wavelength band. WLI and NBI differ in the image processing used to generate a color image from the image signal output by the image sensor. For example, the content of the demosaicing or the parameters used in the image processing differ. Furthermore, a red dichromatic imaging (RDI) mode, for example, can be used as a special light observation mode. The RDI mode is an observation mode that uses narrowband light included in the amber wavelength band, narrowband light included in the green wavelength band, and narrowband light included in the red wavelength band, and uses technology disclosed in, for example, U.S. Patent No. 9,775,497 B2.

[0147] Fig. 52 shows an example of defocus simulation processing (step S208-1) for generating a first learning image 32-1 from a predetermined object image 30-4. Step S208-1 in Fig. 52 differs from step S204-1 in Fig. 50 in that it further includes processing for WLI mode (step S262), processing for NBI mode (step S264), processing for RDI mode (step S266), and processing for TXI mode (step S268). TXI stands for Texture and Color Enhancement Imaging, and details will be described later.

[0148] Note that step S128 in FIG. 51 is an example in which the above-described different processing is added to step S124 in FIG. 39, but this is not limiting. For example, the above-described different processing may be added to step S126 in FIG. 43. In this case, although not shown, the color shift determination processing (step S190) in FIG. 43 is further performed before steps S208 and S308 are performed. Also, in this case, step S208 in FIG. 52 further includes steps S240, S250, and S252 in FIG. 44. Similarly, step S308 in FIG. 53 in this case further includes steps S340, S350, and S352 in FIG. 46. Hereinafter, explanations of points that overlap with step S124 in FIG. 39 and step S126 in FIG. 43 will be omitted as appropriate.

[0149] For example, although not shown in the flowchart, the learning device processing unit 16 reads the observation method information 60 and acquires the observation method used in the first imaging system 101. Then, the learning device processing unit 16 selects one of steps S262, S264, S266, and S268 as the process corresponding to the acquired observation method.

[0150] For example, if the first imaging system 101 captures images in TXI mode, information to that effect is stored in the learning device storage unit 18 as observation method information 60. The learning device processing unit 16 then reads the observation method information 60 and performs a defocus simulation process (step S208) including a TXI mode process (step S368) on the predetermined object image 30-4. Specifically, for example, the learning device processing unit 16 performs a process of decomposing the predetermined object image 30-4 into a texture image portion, which is an image portion related to the surface structure of the predetermined object image 30-4, and a base image portion other than the texture image portion. The learning device processing unit 16 then performs a first process of emphasizing the surface structure related to the texture image portion, a second process of optimizing the brightness of the base image portion, and a third process of optimizing the color tone of an image obtained by combining the image related to the first process and the image related to the second process. This process allows a learning image 32 to be obtained that simulates the effect of capturing images in TXI mode for the predetermined object image 30-4. This allows machine learning to be performed on a dataset including more accurate learning images 32.

[0151] Furthermore, for example, although not shown, when the first imaging system 101 is capturing images in WLI mode or NBI mode, information to that effect is stored as observation method information 60 in the learning device storage unit 18. Then, the learning device processing unit 16 reads the observation method information 60 and performs color interpolation on the predetermined subject image 30-4 to correspond to the light source. Color interpolation may also be performed, for example, in conjunction with step S252 in FIG. 44. For example, when the learning device processing unit 16 selects WLI mode processing (step S262), it performs processing to interpolate the R image and the B image using the G image in conjunction with step S252. For example, when the learning device processing unit 16 selects NBI mode processing (step S264), it performs processing to independently interpolate the G image and the B image in conjunction with step S252.

[0152] FIG. 53 shows an example of a best focus simulation process (step S308) for generating a ground truth image 36 from a predetermined object image 30-4 in the image data generation process (step S128). Step S308 in FIG. 53 differs from step S304 in FIG. 41 in that it further includes processing for WLI mode (step S362), processing for NBI mode (step S364), processing for RDI mode (step S366), and processing for TXI mode (step S368). Step S362 in FIG. 53 is the same as step S262 in FIG. 52, step S364 in FIG. 53 is the same as step S264 in FIG. 52, step S366 in FIG. 53 is the same as step S266 in FIG. 52, and step S368 in FIG. 53 is the same as step S268 in FIG. 52. The correct image 36 may be generated by a process that omits steps S308 to S330 in the best focus simulation process of FIG.

[0153] 25 can be created based on the evaluation results, for example, by evaluating the frequency characteristics of the contrast or amplitude of various image data for each object distance. For example, in FIG. 54, input data indicated by K1 is input to the trained model 1120, and output data indicated by K11 is output from the trained model 1120. Similarly, input data indicated by K9 is input to the trained model 1120, and output data indicated by K19 is output from the trained model 1120.

[0154] In FIG. 54, the image data related to the input data is exemplified by the chart image shown in K2, but it may also be the aforementioned image of biological tissue or an image of a subject simulating biological tissue, as long as it is an image corresponding to the training image 32. As described above, the contrast of the chart image may be the same as the contrast of the biological tissue. The input data in FIG. 54 may be an image of a real chart or an image created by simulation. K3 in FIG. 54 conceptually illustrates a spread function corresponding to the chart shown in K2. In the output data shown in K11, the chart image shown in K12 corresponds to the output image 134, and K13 conceptually illustrates a spread function corresponding to the chart shown in K12.

[0155] In Figure 54, the input data at the top of the page includes a chart image with a low frequency, while the input data at the bottom of the page includes a chart image with a high frequency. For example, the frequency of the chart image shown in K2 associated with the input data K1 is lower than the frequency of the chart image associated with the input data K9. Due to the effect of the correction process (step S1030), the chart image associated with the output data K11 is sharper than the chart image shown in K2. Therefore, by analyzing the spread function shown in K13 and the spread function shown in K3, the user can quantitatively evaluate the difference in contrast between the training image 32 and the output image 134, or the difference in amplitude between the training image 32 and the output image 134.

[0156] On the other hand, for example, when comparing the input data shown in K9 of FIG. 54 with the output data shown in K19, the change in contrast or amplitude due to the correction process (step S1030) is very small. The frequency region related to the input data shown in K9 of FIG. 54 is the frequency region where the MTF value drops to 0, for example, as shown in B1 of FIG. 27. As such, when the frequency increases, it becomes difficult to accurately estimate the significant difference due to the effect of the correction process (step S1030). Therefore, in such a frequency region, it is not necessary to perform blur correction in the blur adjustment process (step S1032).

[0157] By performing the frequency characteristic evaluation shown in FIG. 54 for each object distance, a frequency characteristic diagram shown, for example, as K20 or K30 in FIG. 55 is created. Note that although the vertical axis is not specified, it may represent contrast or amplitude. The graph of K20 shows the frequency characteristic when the object distance is the best focus distance. In the graph of K20, K21 is the first frequency characteristic, which is the frequency characteristic of the output image 134 obtained by the correction process (step S1030). Furthermore, K23 is the frequency characteristic of the training image 32 before the correction process (step S1030) is performed.

[0158] K22 in the graph of K20 in Fig. 55 is the target frequency characteristic of the effect of the blur adjustment process (step S1032), or in other words, the frequency characteristic of the correct image 36. More specifically, K22 is a graph of the result of frequency characteristic evaluation performed on an image generated by performing the best focus simulation process (step S300) on the predetermined object image 30 described above in Fig. 10. Alternatively, K22 may be a graph of the result of frequency characteristic evaluation performed on the predetermined object image 30 itself captured in a focused state by any imaging system 104 described above in Fig. 11.

[0159] In the frequency characteristic diagram of K20, since the object distance was captured at the best focus distance, the training image 32 is substantially the same image as the ground truth image 36, and the graphs shown in K21, K22, and K23 overlap. For convenience, the graph of K20 is shown with the graphs shown in K21, K22, and K23 slightly shifted from each other.

[0160] In the frequency characteristic diagram of K30, K31 is the first frequency characteristic, which is the frequency characteristic of the output image 134 when the object distance is other than the best focus distance, and K33 is the frequency characteristic of the training image 32. Meanwhile, K32 is a display obtained by converting the data of the second frequency characteristic shown in K22 into the frequency characteristic diagram of K30.

[0161] For example, at the frequency indicated by K40, the correction coefficient value is determined based on the vertical axis value indicated by K41 and the vertical axis value indicated by K42, etc., so that the correction coefficient at the desired object distance and frequency, that is, record 1 in correction table 1400 of Fig. 25, can be obtained. Therefore, by repeating the same method as many times as the number of combinations of the required number of object distance data and number of frequency data, correction table 1400 can be created.

[0162] From the above, in the information processing system 1000 of this embodiment, the correction table 1400 is a table in which filter characteristics of the blur adjustment process for matching the first frequency characteristic, which is the frequency characteristic of the output image 134, with the second frequency characteristic, which is the target frequency characteristic during the blur adjustment process, are associated with each of a plurality of object distances. Furthermore, the frequency characteristic is a function that quantitatively indicates the relationship between the contrast or amplitude and the frequency.

[0163] In this way, it is possible to perform blur adjustment processing (step S1032) using correction table 1400 including appropriate correction coefficients. Specifically, blur adjustment processing (step S1032) is performed on output image 134 based on correction table 1400 created in this way, and the frequency characteristics based on the obtained blur-adjusted image 1034 become frequency characteristics like those shown in K32. As described above, the frequency characteristics shown in K32 are frequency characteristics based on the correct image 36, so by further performing blur adjustment processing (step S1032) on output image 134, it is possible to correct output image 134 to an image equivalent to correct image 36.

[0164] Furthermore, in the information processing system 1000 of this embodiment, the second frequency characteristic may be created by converting information on an image captured by an arbitrary imaging system with the subject in focus or information on an image generated by best focus simulation processing into frequency characteristics at an object distance other than the object distance at which the arbitrary imaging system is in focus. In this way, it is possible to appropriately set target frequency characteristics at object distances other than the object distance at which the subject is in focus. This allows the values ​​of the correction coefficients in the correction table 1400 to be appropriately set.

[0165] Furthermore, in the information processing system 1000 of this embodiment, the training image 32 may be an image of biological tissue or an image of a subject simulating biological tissue. In this way, it is possible to evaluate the frequency characteristics of a subject with contrast similar to that of the subject to be actually captured. This allows appropriate blur correction to be performed on a processing target image with contrast similar to that of the subject to be actually captured.

[0166] Furthermore, in the correction table 1400 in the information processing system 1000 of this embodiment, the filter characteristic for blur correction associated with the object distance at which the MTF becomes zero at frequencies equal to or lower than the Nyquist frequency may be a filter characteristic that does not perform blur correction for frequencies equal to or higher than the frequency at which the MTF becomes zero. By doing so, it is possible to perform appropriate blur correction processing on an image made up of frequency components that clearly produce the blur correction effect, among the processing target images.

[0167] In the above example, the correction process (step S1030) is described as being performed on the entire image to be processed, but the method of this embodiment is not limited to this. For example, the image to be processed may be divided, and the correction process (step S1030) may be performed on each divided image to be processed.

[0168] Specifically, for example, it is assumed that the processing target image shown in J0 in Fig. 56 is input to the input unit 1140 of the information processing system 1000. In the correction process (step S1030), the processing unit 1130 performs a process of dividing the processing target image shown in J0 into four images: an image shown in J11, an image shown in J12, an image shown in J13, and an image shown in J14. The image shown in J11 corresponds to the region shown in J12 of the original processing target image, the image shown in J13 corresponds to the region shown in J3 of the original processing target image, and the image shown in J14 corresponds to the region shown in J4 of the original processing target image. Note that while Fig. 56 shows an example in which the processing target image is divided into four images, the number of images to be divided is not limited to four and can be determined appropriately by the user.

[0169] Then, the processing unit 1130 performs correction processing (step S1030) on the images shown in J11, J12, J13, and J14 in order. That is, the processing unit 1130 reads out the trained model 1120 from the storage unit 1110, and inputs the image shown in J11 into the trained model 1120. As a result, the image shown in J21 and the estimated object distance label 1070 indicating the estimated object distance ED1 are output from the trained model 1120. Similarly, the processing unit 1130 reads out the trained model 1120 from the storage unit 1110, and inputs the image shown in J12 into the trained model 1120. As a result, the image shown in J22 and the estimated object distance label 1070 indicating the estimated object distance ED2 are output from the trained model 1120. Similarly, the processing unit 1130 reads out the trained model 1120 from the storage unit 1110, and inputs the image shown in J13 into the trained model 1120. As a result, the image shown in J23 and the estimated object distance label 1070 indicating the estimated object distance ED3 are output from the trained model 1120. Similarly, the processing unit 1130 reads the trained model 1120 from the storage unit 1110, and inputs the image shown in J14 to the trained model 1120. As a result, the image shown in J24 and the estimated object distance label 1070 indicating the estimated object distance ED4 are output from the trained model 1120.

[0170] The images shown in J21, J22, J23, and J24 can be considered to be images obtained by dividing the output image 134 obtained by performing the correction process (step S1030) on the subject image shown in J0. Therefore, the processing unit 1130 performs a process of combining the images shown in J21, J22, J23, and J24 into the output image 134 shown in J30. The area shown in J31 of the output image 134 shown in J30 corresponds to the image shown in J21. Similarly, the area shown in J32 of the output image 134 shown in J30 corresponds to the image shown in J22, the area shown in J33 corresponds to the image shown in J23, and the area shown in J34 corresponds to the image shown in J24.

[0171] Also, for example, information on estimated object distance ED1 corresponding to image J21 may be superimposed and displayed in the area indicated by J31. Similarly, information on estimated object distance ED2 corresponding to image J22 may be superimposed and displayed in the area indicated by J32, information on estimated object distance ED3 corresponding to image J23 may be superimposed and displayed in the area indicated by J33, and information on estimated object distance ED4 corresponding to image J24 may be superimposed and displayed in the area indicated by J34.

[0172] Then, in addition to the correction process (step S1030), blur adjustment process (step S1032) may be further performed for each divided processing target image. In Fig. 57, as in Fig. 56, the processing unit 1130 performs the correction process (step S1030) to divide the processing target image shown as J0 into images shown as J11, J12, J13, and J14. Then, the processing unit 1130 performs the correction process (step S1030) for each of the images shown as J11, J12, J13, and J14.

[0173] From the above, in the information processing system 100 of this embodiment, the processing unit 1130 estimates the object distance for each predetermined divided area of ​​the image to be processed, acquires the filter characteristics associated with the object distance estimated for each divided area from the correction table 1400, and performs blur adjustment processing on the output image 134 using the acquired filter characteristics. In this way, it is possible to perform blur adjustment processing according to the object distance estimated for each divided area. This makes it possible to perform blur adjustment processing that takes into account the flatness of the subject.

[0174] The processing of step S1034 in FIG. 24 will be described in more detail. FIG. 57 is an example conceptually illustrating the processing of step S1034. In FIG. 57, K50 conceptually simplifies the filter characteristics as a real-space filter obtained by inverse Fourier transforming the frequency response function shown in H12 in FIG. 25. That is, step S1034 includes processing for inverse Fourier transforming the frequency response function and processing for convolving the output image 134 with the real-space filter obtained by the inverse Fourier transform. Since the image obtained by the convolution corresponds to the blur-adjusted image 1034, the processing unit 1130 outputs the blur-adjusted image 1034 from the output unit 1150 in step S1040 in FIG. 18 as a corrected image. As such, in the information processing system 1000 of this embodiment, the filter characteristics are real-space filters, and the processing unit 1130 performs blur adjustment processing by convolving the output image 134 with the real-space filter. In this way, blur adjustment processing can be performed in real space.

[0175] Furthermore, the processing unit 1130 may perform step S1034 in Fig. 24 in frequency space. Fig. 58 is a conceptually illustrated example of the processing of step S1034 performed in frequency space. In Fig. 58, K60 is a conceptually simplified illustration of a frequency signal obtained by Fourier transforming the output image 134. Furthermore, K61 in Fig. 58 is a conceptually simplified illustration of a frequency signal obtained by multiplying the frequency signal of K60 by the frequency response function shown in H12 in Fig. 25. In other words, step S1034 includes a process of Fourier transforming the output image 134, a process of multiplying the frequency signal obtained by the Fourier transform by the frequency response function as the filter characteristic acquired in step S1033, and a process of inverse Fourier transforming the frequency signal obtained by multiplication. Since the image obtained by the inverse Fourier transform corresponds to the blur-adjusted image 1034, the processing unit 1130 outputs the blur-adjusted image 1034 as a corrected image from the output unit 1150 in step S1040 of FIG.

[0176] From the above, in the information processing system 1000 of this embodiment, the filter characteristic is a frequency response function, and the processing unit 1130 performs a Fourier transform on the output image 134, multiplies the frequency signal obtained by the Fourier transform by the frequency response function, and performs an inverse Fourier transform on the frequency signal obtained by the multiplication, thereby performing blur adjustment processing. In this way, blur adjustment processing can be performed in frequency space.

[0177] Note that the method of Fig. 57 and the method of Fig. 58 are mathematically equivalent. Therefore, the blur adjustment processed image 1034 obtained by the method of Fig. 57 is substantially the same as the blur adjustment processed image 1034 obtained by the method of Fig. 58. Whether to perform step S1034 using the method of Fig. 57 or the method of Fig. 58 can be determined by the user as appropriate.

[0178] Furthermore, for example, the trained model 1120 of this embodiment may be composed of two or more neural networks. Specifically, for example, as shown in FIG. 59, the trained model 1120 may include a first input layer, a second input layer, a first hidden layer, a second hidden layer, a first output layer, and a second output layer. For example, a processing target image is input to the first input layer. The first hidden layer performs an operation on the output of the first input layer. The first output layer then generates an output image 134 from the output of the first hidden layer. The processing target image is input to the second input layer. The second hidden layer performs an operation on the output of the second input layer. The second output layer then generates an estimated object distance label 1070 from the output of the second hidden layer. Whether the trained model 1120 is configured as shown in FIG. 23 or as shown in FIG. 59 can be determined appropriately by the user.

[0179] Although the present embodiment has been described in detail above, those skilled in the art will readily understand that many modifications are possible without substantially departing from the novel features and advantages of the present disclosure. Therefore, all such modifications are intended to be included within the scope of the present disclosure. For example, a term described at least once in the specification or drawings together with a different term having a broader or equivalent meaning may be replaced with that different term anywhere in the specification or drawings. Furthermore, all combinations of the present embodiment and modifications are also included within the scope of the present disclosure. Furthermore, the configurations and operations of the information processing system, endoscope system, image processing method, program, etc. are not limited to those described in the present embodiment, and various modifications are possible. [Explanation of symbols]

[0180] 10...Learning device, 12...Communication unit, 16...Learning device processing unit, 18...Learning device memory unit, 20, 1020...Training model, 30...Specified subject image, 32...Learning image, 32-1...First learning image, 32-2...Second learning image, 32-N...Nth learning image, 32G...Learning image group, 34, 134...Output image, 34-1...First output image, 34-N...Nth output image, 36...Correct image, 40...Optical system information, 50...Image sensor information, 60...Observation method information, 100, 1000...Information processing system, 101...First imaging system, 102...Second imaging system, 104...Arbitrary imaging system, 106...Homogeneous imaging element, 108...Monochrome imaging element, 110, 1110...Memory unit, 120, 1120...Learned model, 121...First learned model, 122...second trained model, 130, 260, 1130...processing unit, 140, 240, 1140...input unit, 150, 250, 1150...output unit, 160...storage interface, 170, 210...storage unit, 200...processor unit, 220...control unit, 230...imaging data receiving unit, 270...display interface, 300...endoscopic system, 310...endoscopic scope, 320...operation unit, 330...display unit, 1034...blur adjustment processed image, 1070...estimated object distance label, 1070-1...first estimated object distance label, 1070-N...Nth estimated object distance label, 1076...correct object distance label, ED1, ED2, ED3, ED4...estimated object distance, NN...neural network

Claims

1. a storage unit that stores a trained model that has been machine-learned using a dataset including training images, ground truth images, and object distance labels, and a correction table that associates object distance, which is the distance between the imaging system and the subject, with filter characteristics for blur correction; a processing unit; An information processing system comprising: The training image is the image is generated by performing a defocus simulation process on a focused image of a predetermined object captured by an arbitrary imaging system, simulating the effect of blur due to defocus of the first imaging system, based on a transfer function or a point spread function of the first imaging system at the predetermined object distance; The defocus simulation process includes: a region on the optical axis of the first imaging system and a region other than the optical axis in the learning image are subjected to the transfer function or the point spread function on the optical axis, The correct image is an image generated by performing a best focus simulation process for simulating a state in which the first imaging system is in focus on the predetermined subject image based on the transfer function or the point spread function at the object distance in which the first imaging system is in focus, or the predetermined subject image itself; The trained model is The training image is machine-learned to become the correct image, machine learning is performed by assigning the object distance of the transfer function or point spread function of the first imaging system used in the defocus simulation process as the object distance label; The processing unit Using the trained model, an output image is generated in which blur of a processing target image, which is an image captured by the first imaging system, is corrected; Estimating the object distance of the processing target image; an information processing system that obtains the filter characteristics associated with the estimated object distance from the correction table, and performs blur adjustment processing on the output image using the obtained filter characteristics;

2. 2. The information processing system according to claim 1, The correction table is a table in which the filter characteristics of the blur adjustment processing for matching a first frequency characteristic that is a frequency characteristic of the output image with a second frequency characteristic that is a target frequency characteristic during the blur adjustment processing are associated with each of a plurality of object distances; The frequency characteristics are An information processing system characterized in that the function quantitatively indicates the relationship between contrast or amplitude and frequency.

3. 3. The information processing system according to claim 2, The second frequency characteristic is An information processing system characterized in that the information of an image obtained by capturing an image with the subject in focus using the arbitrary imaging system, or an image generated by the best focus simulation process, is created by converting the information into the frequency characteristics at an object distance other than the object distance at which the arbitrary imaging system is in focus.

4. 3. The information processing system according to claim 2, The filter characteristics are: is the frequency response function, The processing unit performing a Fourier transform on the output image; multiplying the frequency signal obtained by the Fourier transform by the frequency response function; The information processing system is characterized in that the blur adjustment process is performed by performing an inverse Fourier transform on the frequency signal obtained by the multiplication.

5. 3. The information processing system according to claim 2, The filter characteristics are: is a real-space filter, The processing unit an information processing system, characterized in that the blur adjustment processing is performed by convolving the output image with the real space filter.

6. 2. The information processing system according to claim 1, The trained model is It consists of a single neural network, The neural network an input layer to which the processing target image is input; an intermediate layer that performs calculations on the output of the input layer; a first output layer that generates the output image from the output of the intermediate layer; a second output layer that estimates the object distance from the output of the intermediate layer; An information processing system comprising:

7. 2. The information processing system according to claim 1, The processing unit estimating the object distance for each predetermined divided region of the processing target image; an information processing system characterized in that the filter characteristics associated with the object distance estimated for each of the divided areas are obtained from the correction table, and the blur adjustment processing is performed on the output image using the obtained filter characteristics.

8. 2. The information processing system according to claim 1, The training image is An information processing system characterized in that the image is an image of biological tissue or an image of a subject that resembles said biological tissue.

9. 2. The information processing system according to claim 1, the information processing system, wherein in the correction table, the filter characteristic for blur correction associated with the object distance at which the MTF becomes zero at frequencies equal to or lower than a Nyquist frequency is the filter characteristic that does not perform blur correction for frequencies equal to or higher than the frequency at which the MTF becomes zero.

10. An information processing system according to any one of claims 1 to 9; an endoscope that captures the processing target image; An endoscope system comprising:

11. An image processing method using a trained model that has been machine-learned using a dataset including training images, ground truth images, and object distance labels, and a correction table that associates object distance, which is the distance between an imaging system and a subject, with filter characteristics for blur correction, The training image is the image is generated by performing a defocus simulation process on a focused image of a predetermined object captured by an arbitrary imaging system, simulating the effect of blur due to defocus of the first imaging system, based on a transfer function or a point spread function of the first imaging system at the predetermined object distance; The defocus simulation process includes: a region on the optical axis of the first imaging system and a region other than the optical axis in the learning image are subjected to the transfer function or the point spread function on the optical axis, The correct image is an image generated by performing a best focus simulation process for simulating a state in which the first imaging system is in focus on the predetermined subject image based on the transfer function or the point spread function at the object distance in which the first imaging system is in focus, or the predetermined subject image itself; The trained model is The training image is machine-learned to become the correct image, machine learning is performed by assigning the object distance of the transfer function or point spread function of the first imaging system used in the defocus simulation process as the object distance label; generating an output image by correcting blur of a processing target image, which is an image captured by the first imaging system, using the trained model; estimating the object distance of the processing target image; obtaining the filter characteristics associated with the estimated object distance from the correction table, and performing blur adjustment processing on the output image using the obtained filter characteristics; An image processing method comprising:

12. A program causing a computer to execute the image processing method according to claim 11.

Images