Information processing system, endoscope system, trained model, information storage medium, and information processing method

Abstract

An information processing system (100) includes: a storage unit (110) for storing a trained model (120) that has been subjected to machine learning using a dataset including a training image group (32G) and a correct answer image (36); and a processing unit (130) which uses the trained model (120) to adjust blurring, resulting from defocusing of a first imaging system (101), of a processing target image, which is an image captured by the first imaging system (101). A region of each training image (32), among a plurality of training images (32), that is on an optical axis of the first imaging system (101) and a region thereof that is not on the optical axis are subjected to defocus simulation processing on the basis of a transfer function or a point spread function on the optical axis. The trained model (120) is subjected to machine learning such that each training image (32) becomes the correct answer image (36).

Description

[Technical Field]

[0001] The present invention relates to an information processing system, an endoscope system, a trained model, an information storage medium, an information processing method, etc. [Background technology]

[0002] In endoscopic observation and the like, it is desirable to be able to observe a subject at a closer distance and in a magnified manner. However, optically, the depth of field narrows as resolution increases due to miniaturization of pixels, so there is a demand for a technology to extend the depth of field using image processing technology. Patent Document 1 discloses a technology to correct optical degradation of an imaging system using deep learning. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2018 / 037521 Summary of the Invention [Problem to be solved by the invention]

[0004] In Patent Document 1, a reference image captured in advance to which optical degradation information has been added is used as a training image. However, since there are an infinite number of pieces of optical degradation information to be learned depending on the object distance and the image height, a huge number of training images are required, and the scale of the network required for processing becomes large, which raises concerns about a decrease in processing capacity, an increase in implementation costs, etc. [Means for solving the problem]

[0005] Book DisclosureOne aspect of the present invention is an information processing system including: a storage unit that stores a trained model that has been machine-learned using a dataset that includes a training image group and a ground truth image; and a processing unit that uses the trained model to correct blur due to defocus of a first imaging system in a processing target image that is an image captured by the first imaging system, wherein the training image group is a plurality of training images that are generated by performing a defocus simulation process on a predetermined subject image captured by an arbitrary imaging system, the predetermined subject image being focused by the arbitrary imaging system, based on a transfer function or a point spread function of the first imaging system at a plurality of object distances. the defocus simulation process is performed on a region on the optical axis of the first imaging system and a region other than the optical axis in each of the plurality of training images based on the transfer function or the point spread function on the optical axis, and the target image is an image generated by performing a best focus simulation process that simulates a state in which the first imaging system is in focus on the specified subject image based on the transfer function or the point spread function at an object distance at which the first imaging system is in focus, or the specified subject image itself, and the trained model relates to an information processing system in which each of the training images is machine-learned to become the target image.

[0006] Also a book Disclosure Another aspect relates to an endoscope system including a processor unit having the information processing system described above, and an endoscope connected to the processor unit and configured to capture the image to be processed.

[0007] Also a book DisclosureYet another aspect of the present invention is a trained model used in an information processing system including a storage unit that stores a trained model, an input unit, a processing unit, and an output unit, the trained model being machine-learned using a dataset including a training image group and a ground truth image, the training image group including a plurality of training images generated by performing a defocus simulation process on a predetermined subject image in which the arbitrary imaging system is focused on a predetermined subject captured by the arbitrary imaging system, based on a transfer function or a point spread function of a first imaging system at a plurality of object distances, to simulate the effect of blur due to defocus of the first imaging system, and the defocus simulation process is performed on a region on the optical axis of the first imaging system and a region other than the optical axis in each of the plurality of training images, based on the transfer function or the point spread function on the optical axis. A simulation process is performed, and the correct image is an image generated by performing a best focus simulation process that simulates the state in which the first imaging system is in focus for the specified subject image based on the transfer function or the point spread function at the object distance at which the first imaging system is in focus, or the specified subject image itself, and the trained model is machine-learned so that each training image becomes the correct image, the input unit inputs a processing target image, which is an image captured by the first imaging system, to the trained model, the processing unit uses the trained model to perform a correction process that corrects blur due to defocusing of the first imaging system in the processing target image, and the output unit is related to the trained model that outputs a corrected image obtained by the correction process.

[0008] Also a book Disclosure Yet another aspect relates to an information storage medium storing the trained model described above.

[0009] Also a book DisclosureYet another aspect of the present invention is an information processing method for correcting blur caused by defocus of a first imaging system of a processing target image, which is an image captured by a first imaging system, by using a trained model machine-learned using a dataset including a training image group and a correct answer image, wherein the training image group includes a plurality of training images generated by performing a defocus simulation process for simulating the influence of blur caused by defocus of the first imaging system based on a transfer function or a point spread function of the first imaging system at a plurality of object distances on a predetermined subject image captured by the arbitrary imaging system, the predetermined subject being focused on the arbitrary imaging system, and The defocus simulation process is performed on the area on the optical axis of the first imaging system and the area other than on the optical axis in each training image based on the transfer function or the point spread function on the optical axis, and the correct image is an image generated by performing a best focus simulation process that simulates the focused state of the first imaging system on the specified subject image based on the transfer function or the point spread function at the object distance where the first imaging system is focused, or the specified subject image itself, and the trained model relates to an information processing method in which each training image is machine-learned to become the correct image. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a block diagram illustrating an example of the configuration of an information processing system. [Figure 2] FIG. 2 is a block diagram illustrating a more detailed configuration example of the information processing system. [Figure 3] 10 is a flowchart illustrating an example of processing by the information processing system. [Figure 4] FIG. 1 is a block diagram illustrating an example of the configuration of a learning device. [Figure 5] A diagram illustrating a training model. [Figure 6] A diagram explaining a neural network. [Figure 7] 10 is a flowchart illustrating a trained model creation process. [Figure 8]FIG. 2 is a diagram illustrating an example of machine learning according to the present embodiment. [Figure 9] FIG. 4 is a diagram illustrating the relationship between the depth of field and the target depth of field. [Figure 10] 5A to 5C are diagrams illustrating an example of image data generation processing. [Figure 11] 10A and 10B are diagrams for explaining another example of image data generation processing. [Figure 12] FIG. 1 is a diagram illustrating a transfer function or a point spread function. [Figure 13] 5A to 5C are diagrams for explaining defocus simulation processing according to the present embodiment. [Figure 14] FIG. 1 is a block diagram illustrating an example of an endoscope system. [Figure 15] FIG. 10 is a block diagram illustrating another example of an endoscope system. [Figure 16] 10A and 10B are diagrams illustrating the relationship between the object distance and the MTF in the defocus simulation process. [Figure 17] FIG. 10 is another diagram illustrating the relationship between the object distance and the MTF in the defocus simulation process. [Figure 18] 10A and 10B are diagrams for explaining a specific calculation method of defocus simulation processing. [Figure 19] FIG. 10 is another diagram illustrating a specific calculation method for the defocus simulation process. [Figure 20] 5A and 5B are diagrams for explaining a specific calculation method of best focus simulation processing. [Figure 21] FIG. 10 is another diagram for explaining a specific calculation method for the best focus simulation process. [Figure 22] FIG. 3 is a diagram illustrating an example of a lens configuration of a first imaging system. [Figure 23] FIG. 4 is another diagram illustrating an example of the lens configuration of the first imaging system. [Figure 24] FIG. 10 is a diagram illustrating the amount of distortion. [Figure 25] FIG. 2 is a diagram illustrating a lens configuration including a phase modulation element. [Figure 26] 10A and 10B are diagrams illustrating an example of a change in MTF due to the inclusion of a phase modulation element. [Figure 27] 10A and 10B are diagrams for explaining another example of image data generation processing. [Figure 28] 10A and 10B are diagrams for explaining another example of the defocus simulation process. [Figure 29] 10A and 10B are diagrams for explaining another example of image data generation processing. [Figure 30] 10A and 10B are diagrams for explaining another example of the defocus simulation process. [Figure 31] 10A and 10B are diagrams for explaining another example of the best focus simulation process. [Figure 32] FIG. 10 is a diagram illustrating another example of the configuration of an information processing system. [Figure 33] 10A and 10B are diagrams for explaining another example of image data generation processing. [Figure 34] 10A and 10B are diagrams for explaining another example of the defocus simulation process. [Figure 35] 1A and 1B are diagrams illustrating the relationship between mosaic processing and demosaic processing. [Figure 36] 10A and 10B are diagrams for explaining another example of the best focus simulation process. [Figure 37] FIG. 10 is a diagram illustrating another example of the configuration of an information processing system. [Figure 38] 10 is a flowchart illustrating another example of processing by the information processing system. [Figure 39] 10 is a flowchart illustrating a first trained model creation process. [Figure 40] 10 is a flowchart illustrating a second trained model creation process. [Figure 41] 10A and 10B are diagrams for explaining another example of image data generation processing. [Figure 42] 10A and 10B are diagrams for explaining another example of the defocus simulation process. [Figure 43] 10A and 10B are diagrams for explaining another example of the best focus simulation process. DETAILED DESCRIPTION OF THE INVENTION

[0011] The present embodiment will be described below. Note that the present embodiment described below does not unduly limit the content of the present invention as defined in the claims. Furthermore, not all of the configurations described in the present embodiment are necessarily essential components of the present invention. For example, the following description will be given of an example in which the information processing system is applied to a medical endoscope, but the present invention is not limited to this, and the information processing system of the present invention can be applied to various imaging systems or image display systems. For example, the information processing system of the present invention can be applied to a still camera, a video camera, a television receiver, a microscope, or an industrial endoscope.

[0012] FIG. 1 is a block diagram illustrating an example configuration of an information processing system 100 according to this embodiment. The information processing system 100 includes a storage unit 110 and a processing unit 130. The storage unit 110 stores a trained model 120 that has been machine-learned. The trained model 120 is a program module that outputs a corrected image in which blur caused by defocusing of a processing target image is corrected, and is generated or updated through machine learning, as described below. The processing target image is, for example, image data captured by a first imaging system 101 as shown in FIG. 1, but is not limited to this and details will be described later. Note that in this embodiment, image data that can be processed as digital data may simply be referred to as an image. The training image group 32G is a collection of training images 32 consisting of a first training image 32-1, a second training image 32-2, ..., an Nth training image 32-N, and details of the training image 32, along with a target image 36, will be described later. That is, the processing unit 130 of this embodiment corrects blur caused by defocus of the first imaging system 101 in the processing target image, which is an image captured by the first imaging system 101, using the trained model 120. The storage unit 110 and the processing unit 130 are also referred to as a storage device and a processing device, respectively.

[0013] In this embodiment, machine learning refers to, for example, supervised learning. Training data in supervised learning is a dataset in which input data is associated with correct labels. Specifically, the trained model 120 in this embodiment is generated by supervised learning based on a dataset in which input data consisting of training images 32 simulating the effects of various blurs is associated with correct labels consisting of focused correct images 36.

[0014] The processing unit 130 of this embodiment is configured with the following hardware. The hardware can include at least one of a circuit for processing digital signals and a circuit for processing analog signals. For example, the hardware can be configured with one or more circuit devices or one or more circuit elements mounted on a circuit board. The one or more circuit devices are, for example, ICs, etc. The one or more circuit elements are, for example, resistors, capacitors, etc.

[0015] The processing unit 130 may also be implemented by the following processor. The processing unit 130 of this embodiment includes a memory that stores information and a processor that operates based on the information stored in the memory. The memory is, for example, the storage unit 110. The information is, for example, a program and various data. The processor includes hardware. Various processors can be used as the processor, such as a central processing unit (CPU), a graphics processing unit (GPU), or a digital signal processor (DSP). The memory may be a semiconductor memory such as a static random access memory (SRAM) or a dynamic random access memory (DRAM), a register, a magnetic storage device such as a hard disk drive, or an optical storage device such as an optical disk drive. For example, the memory stores computer-readable instructions, and the functions of each unit of the processing unit 130 are realized as processing by the processor executing the instructions. The instructions here may be instructions from an instruction set that constitutes a program, or instructions that instruct the processor's hardware circuitry to operate.

[0016] The trained model 120 of this embodiment may also be used in the information processing system 100 shown in the configuration example of Fig. 2. That is, the trained model 120 of this embodiment is used in the information processing system 100 including a storage unit 110 that stores the trained model 120, an input unit 140, a processing unit 130, and an output unit 150, and is machine-trained using a dataset including a training image group 32G and a master image 36.

[0017] The input unit 140 is an interface that receives a processing target image from the outside. Specifically, for example, as shown in FIGS. 1 and 2, it is an image data interface that receives image data from the first imaging system 101 as the processing target image. For example, the input unit 140 functions as the input unit 140 by using the received processing target image as input data to the trained model 120 and having the processing unit 130 perform the processing described below. That is, in the trained model 120 of this embodiment, the input unit 140 inputs the processing target image, which is an image captured by the first imaging system 101, to the trained model 120.

[0018] The output unit 150 is an interface that transmits the above-mentioned corrected image to the outside. For example, the output unit 150 functions as the output unit 150 by using output data from the trained model 120 as the corrected image transmitted by the output unit 150. The destination of the corrected image is, for example, a predetermined display device connected to the information processing system 100, and the output unit 150 functions as the output unit 150 by, for example, being an interface connectable to the predetermined display device, whereby the corrected image is displayed on the display device. Note that the output destination of the corrected image may be a storage device of an external device, etc.

[0019] FIG. 3 is a flowchart illustrating a method performed by the information processing system 100 of this embodiment. The processing unit 130 reads a processing target image (step S10) and a trained model (step S20), and then performs a correction process (step S30). Specifically, for example, the processing unit 130 inputs the processing target image received via the input unit 140 into the trained model 120 read from the storage unit 110. If the trained model 120 determines that the processing target image, which is input data, is common to the training image 32, it estimates that the data to be output is the correct image 36, and therefore outputs the correct image 36 when the processing target image is input. Comparing the processing target image and the correct image 36, it is found that the correct image 36 is an image in which blurring caused by defocusing of the first imaging system 101 in the processing target image has been corrected. In other words, the processing unit 130 uses the trained model 120 to perform a correction process (step S30) to correct blurring caused by defocusing of the first imaging system 101 in the processing target image.

[0020] Thereafter, the processing unit 130 outputs the corrected image (step S40). Specifically, the output unit 150 functions as described above, and the corrected image is output to a desired output destination. In other words, the output unit 150 outputs the corrected image obtained by the correction process.

[0021] Next, machine learning of the trained model 120 will be described. Machine learning is performed, for example, by a learning device 10. FIG. 4 is a block diagram showing an example configuration of the learning device 10. The learning device 10 includes, for example, a communication unit 12, a learning device processing unit 16, and a learning device storage unit 18.

[0022] The communication unit 12 is a communication interface capable of communicating with the information processing system 100 via a predetermined communication method. The predetermined communication method is, for example, a communication method conforming to a wireless communication standard such as Wi-Fi (registered trademark), but is not limited to this and may also be a communication method conforming to a wired communication standard such as USB. This allows the learning device 10 to transmit a trained model 120 that has been machine-learned using a method described below to the information processing system 100, and the information processing system 100 to update the trained model 120. Note that while FIG. 4 shows an example in which the learning device 10 and the information processing system 100 are separate, this does not preclude a configuration example in which the information processing system 100 includes a learning server equivalent to the learning device 10.

[0023] The learning device processing unit 16 controls the input and output of data between each functional unit, such as the communication unit 12 and the learning device storage unit 18. The learning device processing unit 16 can be realized by a processor similar to the processing unit 130 in FIG. 1. The learning device processing unit 16 executes various arithmetic processes based on a predetermined program read from the learning device storage unit 18 and operation input signals from an operation unit (not shown in FIG. 4), and controls data output operations to the information processing system 100, etc. The predetermined program here includes a machine learning program. In other words, the learning device processing unit 16 performs machine learning functions by reading and executing the machine learning program and necessary data, etc. from the learning device storage unit 18.

[0024] The learning device storage unit 18 stores a machine learning program (not shown), as well as a training model 20, a predetermined subject image 30, and optical system information 40. The learning device storage unit 18 can be realized by a semiconductor memory similar to the storage unit 110 described above. The learning device storage unit 18 may further include other information. The other information may be, for example, image sensor information 50 described below.

[0025] The predetermined subject image 30 is an image of a subject related to the processing target image, and a learning image 32 and a correct answer image 36, which will be described later, are created based on the predetermined subject image 30. That is, the learning device storage unit 18 pre-stores as many predetermined subject images 30 as the number of types of subjects that can become the processing target image. To give a more specific example, when the information processing system 100 is used in an endoscope system 300, which will be described later, an image of an inner cavity or the like captured by an endoscope 310, which will be described later, becomes the predetermined subject image 30. In the following description, for convenience, an imaging system that does not particularly matter for capturing the predetermined subject image 30 will be referred to as an arbitrary imaging system 104. A case in which the predetermined subject image 30 is captured using a limited imaging system will be described later.

[0026] The training model 20 is a model that is the target of machine learning by the learning device processing unit 16. The model here is information that derives the correspondence between estimation target data and estimation result data. More specifically, it is information that derives an output image 34, which is estimation result data, from a training image 32, which is estimation target data. In the training model 20 of this embodiment, a neural network NN is included in at least a part of the model. Details of the neural network NN will be described later with reference to FIG. 6. Note that, as described above, when the information processing system 100 and the learning device 10 are integrated, machine learning may be performed on the trained model 120.

[0027] For example, when a first training image 32-1 is input to the training model 20, the training model 20 outputs a first output image 34-1. Similarly, when an Nth training image 32-N is input to the training model 20, the training model 20 outputs an Nth output image 34-N. That is, as shown in FIG. 5, in the learning device 10 of this embodiment, N images consisting of the first training image 32-1 to the Nth training image 32-N are input to the training model 20 as a training image group 32G.

[0028] FIG. 6 is a schematic diagram illustrating a neural network (NN). The neural network (NN) has an input layer to which data is input, a hidden layer that performs calculations based on the output from the input layer, and an output layer that outputs data based on the output from the hidden layer. While FIG. 6 illustrates a network with two hidden layers, the hidden layer may be one or three or more. The number of nodes included in each layer is not limited to the example shown in FIG. 6, and various modifications are possible. As shown in FIG. 6, nodes in a given layer are connected to nodes in adjacent layers. Each connection is assigned a weighting coefficient. Each node multiplies the output of the previous node by the weighting coefficient and calculates the sum of the multiplication results. Each node then adds a bias to the sum and applies an activation function to the sum to calculate the output of that node. This process is performed sequentially from the input layer to the output layer to calculate the output of the neural network (NN). Various activation functions, such as the sigmoid function and the ReLU function, are known, and these can be widely applied in this embodiment.

[0029] Various types of neural network NN models are known, and these can be widely applied in this embodiment. For example, the neural network NN may be a CNN (Convolutional Neural Network), an RNN (Recurrent Neural Network), or other models.

[0030] FIG. 7 is a flowchart illustrating an example of the trained model creation process (step S100). The trained model creation process (step S100) is a process for creating or updating a trained model 120 through machine learning. The learning device processing unit 16 reads a predetermined object image (step S110) and then performs image data generation process (step S120). For example, the learning device processing unit 16 reads a predetermined object image 30 from the learning device storage unit 18 and performs predetermined processing to generate a training image 32 and a correct answer image 36 using the predetermined object image 30. The predetermined processing includes a defocus simulation process (step S200), a best focus simulation process (step S300), etc., which will be described in detail later.

[0031] Thereafter, the learning device processing unit 16 performs a correction learning process (step S130). For example, the learning device processing unit 16 reads out the training model 20 from the learning device storage unit 18, inputs the training image 32 generated in the image data generation process (step S120) into the training model 20, and performs machine learning processing based on the output image 34 output from the training model 20 and the correct image 36.

[0032] The machine learning process based on the output image 34 and the correct image 36 is, for example, as shown in FIG. 8, a process of changing the network parameters of the neural network NN so that the first output image 34-1 to the Nth output image 34-N become the correct image 36. Specifically, the process of changing the network parameters of the neural network NN is, for example, a process of updating appropriate weighting coefficients in the neural network NN. The weighting coefficients here include biases. In updating the weighting coefficients, for example, an error backpropagation method can be used, in which weighting coefficients are updated from the output layer to the input layer. That is, the learning device 10 inputs input data from the training data into a model and determines the output by performing a forward calculation in accordance with the model configuration using the weighting coefficients at that time. An error function is calculated based on the output and the correct label, and the weighting coefficients are updated to reduce the error function.

[0033] More specifically, for example, the learning device processing unit 16 inputs the first training image 32-1 as input data to the neural network NN included in the training model 20, and performs forward calculations using the weighting coefficients at that time to output the first output image 34-1 as output data. The learning device processing unit 16 calculates an error function based on the first output image 34-1 and the correct answer image 36, which is the correct label. Then, it performs processing to update the weighting coefficients so as to reduce the error function. The learning device processing unit 16 also repeatedly performs similar processing on the second output image 34-2 to the Nth output image 34-N. In this way, the training model 20 is machine-learned so that one correct answer image 36 can be output for multiple types of training images 32. As a result, the machine-learned training model 20 is output as the trained model 120 to the information processing system 100, and the trained model 120 stored in the storage unit 110 is updated. 4 shows the learning device 10 and the information processing system 100 as being communicatively connected via the communication unit 12, but the learning device 10 and the information processing system 100 do not have to be communicatively connected. In this case, for example, a user can update the trained model 120 by causing the learning device 10 to temporarily store the training model 20 in an information storage medium as the trained model 120, carrying the information storage medium and moving to the location of the information processing system 100, and then causing the information processing system 100 to update the trained model 120 based on the information storage medium.

[0034] FIG. 9 is a diagram illustrating the relationship between the depth of focus and the depth of field for the first imaging system 101 of this embodiment, with the optical axis as the horizontal axis. Note that FIG. 9 is a diagram for convenience and does not depict the specific lens configuration of the first imaging system 101. For example, in FIG. 9, the range indicated by DP1 is the depth of field corresponding to the depth of focus in the optical design of the first imaging system 101. Therefore, for example, if the distance between the subject and the first imaging system 101 is a first object distance indicated by D1, the subject is located outside the depth of field, and when captured by the first imaging system 101, an image to be processed that includes the effects of blur due to defocus is obtained. Also, for example, if the distance between the subject and the first imaging system 101 is a second object distance indicated by D2, the subject is located within the depth of field, and the image to be processed is in focus. Note that, for example, if the distance between the subject and the first imaging system 101 is an object distance indicated by D3, that is, the position indicated by P1 on the optical axis in the depth of field is a position that satisfies the best focus condition. 9, the first object distance indicated by D1 and the second object distance indicated by D2 are shown on the near point side from the position indicated by P1, but are not limited to the near point side and may be on the far point side. In the following description and illustrations, the method of this embodiment will be explained using an object distance on the near point side as an example, but this does not prevent the method of this embodiment from being applicable even when an object distance on the far point side is used.

[0035] For example, in a system equipped with the first imaging system 101, miniaturizing pixels to increase resolution narrows the depth of field, and therefore extending the depth of field is desirable. Also, for example, when the first imaging system 101 is used in an endoscope 310 of an endoscope system 300 (described later), it is difficult to adjust the endoscope 310 to the best focus position for a desired subject, and therefore extending the depth of field is desirable.

[0036] Therefore, in this embodiment, a trained model 120, which has undergone the machine learning described above with reference to FIG. 8 and the like as a data set in which an image obtained by simulating the effect of blur on a predetermined subject image 30 captured in advance is used as a training image 32 and an image in focus is used as a ground truth image 36, is incorporated into the information processing system 100. In this way, a captured image to which the effect of blur due to defocus has been added is used as a processing target image, and by performing the processing of FIG. 3, a corrected image in focus is output from the information processing system 100. This makes it possible to substantially expand the range of the depth of field of the first imaging system 101.

[0037] More specifically, the depth of field can be substantially expanded from the range shown in DP1 in FIG. 9 to the range shown in DP2. Substantially expanding refers to expanding the apparent depth of field to a range where the depth of field is not optically expanded, but the image processing performed by the information processing system 100 allows a subject actually located outside the depth of field to be captured as if it were located within the depth of field. That is, when a subject is located at a position away from the first imaging system 101 by the object distance shown in D1, a blurred processing target image is output from the first imaging system 101. However, since this position is located within the effective depth of field shown in DP2, the processing target image is corrected to a focused corrected image and output from the information processing system 100. In the following description, the effective depth of field shown in DP2 in FIG. 9 expanded using the trained model 120 of this embodiment is referred to as the target depth of field. Note that the focused corrected image here does not necessarily mean that the entire image is strictly focused. For example, even if a portion of the output corrected image is blurred, the user may determine that the function of the information processing system 100 is sufficient as long as a procedure using the endoscope 310 can be performed. In other words, the target depth of field distance in this embodiment is a distance wider than the optically determined depth of field distance, but is a distance that can vary depending on the user's tolerance level, etc. Therefore, DP2 shown in FIG. 9 is shown merely for convenience and does not indicate a fixed length. The same applies to the following explanation.

[0038] The trained model 120 of this embodiment is machine-learned so that a blurred image obtained by capturing an image of a subject located in the range shown in DP10 in Fig. 9 can be corrected to an in-focus image, using the difference between the target depth of field shown in DP2 and the depth of field shown in DP1. In other words, the distance shown in DP10 is the distance required for machine learning.

[0039] The method of image data generation processing (step S120) for generating training images 32 and ground truth images 36 required for the machine learning will be described with reference to Fig. 10. Note that the method of image data generation processing is not limited to that shown in Fig. 10, and various modifications are possible as will be described later. Therefore, the image data generation processing shown in Fig. 10 can also be referred to as step S120-1.

[0040] In any example, the predetermined subject image 30 of this embodiment is assumed to be captured at an object distance at which the imaging system that captured the image is in focus.

[0041] The learning device processing unit 16 generates a learning image 32 by performing a defocus simulation process (step S200) on a predetermined object image 30 captured by an arbitrary imaging system 104. In the following description, for example, the defocus simulation process for generating the first learning image 32-1 may also be referred to as step S200-1, and similarly, the defocus simulation process for generating the Nth learning image 32-N may also be referred to as step S200-N. The same applies to steps S202, S204, S206, S208, S210, S220, and S230 described below. For example, when generating the first learning image 32-1 by the defocus simulation process (step S200-1), the learning device processing unit 16 selects information on the first object distance from the loaded optical system information 40. Similarly, when generating the second learning image 32-2 by step S200-2, the learning device processing unit 16 selects information on the second object distance from the loaded optical system information 40. That is, in this embodiment, the optical system information 40 corresponding to the Nth learning image 32-N is the Nth object distance, and when generating the Nth learning image 32-N, the learning device processing unit 16 can select and expand information on the corresponding Nth object distance from the optical system information 40. In the following explanation, the defocus simulation process will be exemplified as a process for generating the first learning image 32-1, but the same process is also performed when generating the second learning image 32-2 to the Nth learning image 32-N.

[0042] The learning device processing unit 16 also generates a ground truth image 36 by performing a best focus simulation process (step S300) on the predetermined subject image 30. For example, the learning device processing unit 16 selects information on the object distance at which the first imaging system 101 is in focus from the read optical system information 40. The information on the object distance at which the first imaging system 101 is in focus is, for example, as shown in D3, the design distance from the first imaging system 101 to the point shown in P1 in FIG. 9, which is the object distance that corresponds to the so-called best focus condition.

[0043] The image data generation process of this embodiment may be as shown in Fig. 11. The image data generation process shown in Fig. 11 may also be called step S120-2. Furthermore, a description of the same processes as those in Fig. 10 will be omitted as appropriate.

[0044] 11 differs from step S120-1 in Fig. 10 in that the best focus simulation process (step S300) is not performed, and the predetermined subject image 30 itself is used as the target image 36. This is because the predetermined subject image 30 can be used as the target image 36 if it is an image captured at an object distance at which any imaging system 104 is in focus.

[0045] The defocus simulation process (step S200) will be described using FIGS. 12 and 13. Optical system information 40, which is read when performing the defocus simulation process (step S200), includes information on a transfer function or a point spread function. The transfer function or the point spread function varies depending on the amount of defocus in the optical axis direction and the image height in a plane perpendicular to the optical axis. For example, at a first object distance, a region perpendicular to the optical axis and having the same size as a predetermined object image 30 is divided into regions FC11-1, FC12-1, FC13-1, FC21-1, FC22-1, FC23-1, FC31-1, FC32-1, and FC33-1. In this case, the transfer function or the point spread function at the first object distance may exhibit different values ​​for each divided region. Similarly, for example, at the Nth object distance, a region perpendicular to the optical axis and the same size as the predetermined object image 30 is divided into regions FC11-N, FC12-N, FC13-N, FC21-N, FC22-N, FC23-N, FC31-N, FC32-N, and FC33-N. In this case, the transfer function or point spread function at the Nth object distance may exhibit different values ​​for each divided region. Furthermore, the transfer function or point spread function of region FC11-1 may exhibit different values ​​from the transfer function or point spread function of region FC11-N. The same applies to regions FC12-1 and FC12-N, ..., and regions FC33-1 and FC33-N. Thus, if the training image group 32G is a collection of N training images 32, as shown in FIG. 12, the amount of information on the transfer function or point spread function required for machine learning becomes enormous.

[0046] In this regard, in this embodiment, a transfer function or a point spread function on the optical axis is used for machine learning. In this embodiment, the region FC22-1 is the region through which the optical axis of the first imaging system 101 passes. In other words, the transfer function or the point spread function in the region FC22-1 is the transfer function or the point spread function on the optical axis of the first imaging system 101 at the first object distance. Similarly, in the region FC22-N in The transfer function or point spread function at the Nth object distance is At the Nth object distanceThis is a transfer function or point spread function on the optical axis of the first imaging system 101. Note that although the transfer function or point spread function is divided into nine in Fig. 12, this is merely an example, and the same applies to Fig. 13. For example, the regions FC22-1 to FC22-N in Fig. 12 are each divided into predetermined regions in the vertical and horizontal directions. Number of pixels However, it may be a single pixel. In other words, the transfer function or point spread function on the optical axis in this embodiment is a transfer function or point spread function in at least one of an area of ​​one pixel passing through the optical axis or an area of ​​a predetermined number of pixels including that pixel.

[0047] As shown in FIG. 13, in the defocus simulation process (step S200), a process (step S210) is performed on the predetermined object image 30 to simulate the effect of blur based on the on-axis transfer function or on-axis point spread function of the first imaging system 101. Details of step S210 will be described later. That is, step S210 is also performed on regions other than those on the optical axis of the predetermined object image 30 based on the on-axis transfer function or on-axis point spread function of the first imaging system 101. For example, suppose the predetermined object image 30 is divided into nine regions AR11, AR12, AR13, AR21, AR22, AR23, AR31, AR32, and AR33, as in FIG. 12. For example, when generating a first training image 32-1, the training device processing unit 16 performs the calculation of step S210-1 on the region AR11 using the on-axis transfer function or on-axis point spread function shown as FC22-1 in FIG. 12. In the following explanation and in the illustration of FIG. 13, this calculation will be abbreviated as AR11*FC22-1. The same applies to calculations such as step S210 that use other regions. Furthermore, the "*" here indicates convolution when, for example, a PSF is used as the point spread function, as will be described in detail later. Furthermore, when, for example, an OTF is used as the transfer function, the "*" indicates that the frequency characteristic obtained by Fourier transforming the region AR11 is multiplied by the OTF of the region FC22-1.

[0048] Furthermore, the learning device processing unit 16 also performs step S210-1 for regions AR12 to AR33 using the transfer function or point spread function on the optical axis shown in FC22-1. That is, although some parts are omitted in FIG. 13, the learning device processing unit 16 performs AR12*FC22-1, AR13*FC22-1, AR21*FC22-1, AR22*FC22-1, AR23*FC22-1, AR31*FC22-1, AR32*FC22-1, and AR33*FC22-1. In this way, the learning device processing unit 16 divides the same region as the predetermined object image 30 into a desired number of regions, and performs step S210 using the transfer function or point spread function of one of the divided regions.

[0049] Similarly, suppose the generated first learning image 32-1 is divided into nine regions: region BR11-1, region BR12-1, region BR13-1, region BR21-1, region BR22-1, region BR23-1, region BR31-1, region BR32-1, and region BR33-1. Region BR11-1 corresponds to the result of performing step S210-1 on the aforementioned region AR11. That is, as shown in FIG. 13, BR11-1 = AR11 * FC22-1. Similarly, BR12-1 = AR12*FC22-1, BR13-1 = AR13*FC22-1, BR21-1 = AR21*FC22-1, BR22-1 = AR22*FC22-1, BR23-1 = AR23*FC22-1, BR31-1 = AR31*FC22-1, BR32-1 = AR32*FC22-1, BR33-1 = AR33*FC22-1.

[0050] This method is also used when generating the Nth training image 32-N. That is, although not shown, the training device processing unit 16 performs the following: BR11-N=AR11*FC22-N, BR12-N=AR12*FC22-N, ..., BR22-N=AR22*FC22-N, ..., BR32-N=AR32*FC22-N, BR33-N=AR33*FC22-N. From the above, the area on the optical axis of the first imaging system 101 (BR22) and the areas off the optical axis (BR11, ..., BR21, BR23, ...) in each training image 32 are BR33), a defocus simulation process (step S200) is performed based on the transfer function on the optical axis or the point spread function (FC22).

[0051] The transfer function in this embodiment can also be called an optical transfer function or OTF. OTF stands for Optical Transfer Function. The point spread function in this embodiment can also be called a point spread function or PSF. PSF stands for Point Spread Function. OTF is the result of a Fourier transform of the PSF. In other words, PSF is the result of an inverse Fourier transform of the OTF. OTF is a complex function, and the absolute value of the OTF is called a modulation transfer function, amplitude transfer function, or MTF. MTF stands for Modulation Transfer Function.

[0052] As described above, the information processing system 100 of this embodiment includes a storage unit 110 that stores a trained model 120 that has been machine-learned using a dataset including a training image group 32G and a gold standard image 36, and a processing unit 130 that uses the trained model 120 to correct blur due to defocus of the first imaging system 101 in a processing target image, which is an image captured by the first imaging system 101. The training image group 32G includes multiple training images 32 generated by performing a defocus simulation process (step S200) on a predetermined object image 30, in which the predetermined object captured by the arbitrary imaging system 104 is focused on the predetermined object, based on the transfer function or point spread function of the first imaging system 101 at multiple object distances. The defocus simulation process is performed on an area on the optical axis of the first imaging system 101 and an area off the optical axis in each of the multiple training images 32, based on the transfer function or point spread function on the optical axis. The correct image 36 is an image generated by performing a best focus simulation process (step S300) that simulates the state in which the first imaging system 101 is in focus for the predetermined object image 30 based on the transfer function or point spread function at the object distance at which the first imaging system 101 is in focus, or is the predetermined object image 30 itself. The trained model 120 is machine-learned so that each training image 32 becomes the correct image 36.

[0053] As described above, the information processing system 100 of this embodiment includes the storage unit 110 that stores the trained model 120 and the processing unit 130. Therefore, even if the processing target image captured by the first imaging system 101 includes the influence of blur due to defocus, a corrected image in which the influence of blur is corrected can be output. This allows the depth of field of the first imaging system 101 to be substantially expanded. Furthermore, since the training image group 32G and the ground truth image 36 are created in advance based on the predetermined subject image 30 captured by an arbitrary imaging system 104, when the subject related to the processing target image is a subject being captured for the first time by the first imaging system 101, the trained model 120 that has been machine-learned in advance can be used. Furthermore, the defocus simulation process (step S200) is performed on the on-axis region and off-axis region of the first imaging system 101 in each training image 32 based on the on-axis transfer function or point spread function. This reduces the amount of information required for the defocus simulation process (step S200). This makes it possible to create a trained model 120 with an appropriate size for the neural network NN required for machine learning. This makes it possible to easily implement the trained model 120 in the information processing system 100.

[0054] The method of this embodiment can also be realized as a trained model 120. That is, the trained model 120 of this embodiment is used in an information processing system 100 including a storage unit 110 that stores the trained model 120, an input unit 140, a processing unit 130, and an output unit 150, and is machine-learned using a dataset including a training image group 32G and a ground truth image 36. The training image group 32G includes a plurality of training images 32 generated by performing a defocus simulation process on a predetermined object image 30, in which the predetermined object image is captured by an arbitrary imaging system 104 and the arbitrary imaging system 104 is focused on the predetermined object, based on the transfer function or point spread function of the first imaging system 101 at a plurality of object distances. The defocus simulation process is performed on an area on the optical axis of the first imaging system 101 and an area off the optical axis in each of the plurality of training images 32, based on the transfer function or point spread function on the optical axis. The correct image 36 is an image generated by performing a best focus simulation process that simulates the state in which the first imaging system 101 is in focus on the predetermined object image 30 based on a transfer function or a point spread function at an object distance where the first imaging system 101 is in focus, or is the predetermined object image 30 itself. The trained model 120 is machine-trained so that each training image 32 becomes the correct image 36. The input unit 140 inputs a processing target image, which is an image captured by the first imaging system 101, to the trained model 120. The processing unit 130 uses the trained model 120 to perform a correction process that corrects blurring of the processing target image due to defocusing of the first imaging system 101. The output unit 150 outputs the corrected image obtained by the correction process. In this manner, the same effect as described above can be obtained.

[0055] The technique of this embodiment can also be realized as an information processing method. That is, the information processing method of this embodiment corrects blur due to defocus of the first imaging system 101 in a processing target image, which is an image captured by the first imaging system 101, using a trained model 120 that has been machine-learned using a dataset including a training image group 32G and a gold standard image 36. The training image group 32G includes a plurality of training images 32 that are generated by performing a defocus simulation process on a predetermined object image 30, in which an arbitrary imaging system 104 focuses on a predetermined object captured by the arbitrary imaging system 104, to simulate the effect of blur due to defocus of the first imaging system 101 based on the transfer function or point spread function of the first imaging system 101 at a plurality of object distances. The defocus simulation process is performed on an area on the optical axis of the first imaging system 101 and an area off the optical axis in each of the plurality of training images 32, based on the transfer function or point spread function on the optical axis. The correct image 36 is an image generated by performing a best focus simulation process that simulates the state in which the first imaging system 101 is in focus on the predetermined object image 30 based on the transfer function or point spread function at the object distance at which the first imaging system 101 is in focus, or is the predetermined object image 30 itself. The trained model 120 is machine-learned so that each training image 32 becomes the correct image 36. In this way, the same effect as above can be obtained.

[0056] The method of this embodiment can also be realized as an information storage medium storing the trained model 120. In this way, the training model 20 obtained by machine learning in the learning device 10 can be stored in the information storage medium. As a result, by connecting the information storage medium to the information processing system 100, the training model 20 can be updated to the latest trained model 120. This makes it possible to obtain the same effect as above even under certain circumstances. Examples of certain circumstances include a situation where the location of the learning device 10 is far from the location of the information processing system 100, or a situation where data communication between the learning device 10 and the information processing system 100 is not possible.

[0057] The method of the present embodiment may also be realized as an endoscope system 300. For example, the endoscope system 300 of the present embodiment includes a processor unit 200 including the above-described information processing system 100, and an endoscope 310 connected to the processor unit 200 and configured to capture an image to be processed. In this way, it is possible to construct an endoscope system 300 including the information processing system 100 having the above-described effects.

[0058] More specifically, the endoscope system 300 can have a configuration example as shown in Fig. 14. The endoscope system 300 includes an endoscope 310, an operation unit 320, a display unit 330, and a processor unit 200. The processor unit 200 includes a storage unit 210, a control unit 220, and an information processing system 100. The information processing system 100 in Fig. 14 further includes a storage interface 160 in addition to the configuration described above in Fig. 2. Note that descriptions of the same configuration as in Fig. 2 will be omitted as appropriate.

[0059] The endoscope 310 includes an imaging device at its tip (not shown). The imaging device includes a first imaging system 101. The tip of the endoscope 310 is inserted into a body cavity, the imaging device captures images of the abdominal cavity, and the captured image data is transmitted from the endoscope 310 to the processor unit 200. The operation unit 320 is a device that allows a user to operate the endoscope system 300, and is, for example, a button, a dial, a foot switch, or a touch panel. The display unit 330 is a device that displays images captured by the endoscope 310, and is, for example, a liquid crystal display, but may also be hardware integrated with the operation unit 320, for example, as a touch panel.

[0060] The processor unit 200 performs various processes such as control and image processing in the endoscope system 300. For example, the control unit 220 performs mode switching, zooming, display switching, etc. of the endoscope system 300 based on information input from the operation unit 320, thereby realizing the functions of the processor unit 200. The storage unit 210 records images captured by the endoscope 310. The storage unit 210 is, for example, a semiconductor memory, a hard disk drive, or an optical drive.

[0061] In the configuration example shown in Figure 14, the function of receiving imaging data from the endoscope 310 is realized by making the connector to which the cable of the endoscope 310 is connected or the interface circuit for receiving imaging data, etc., the input unit 140, but the processor unit 200 may also be provided with an interface circuit for receiving imaging data.

[0062] The storage interface 160 is an interface for accessing the storage unit 210. The storage interface 160 records image data received by the input unit 140 in the storage unit 210. When playing back the recorded image data, the storage interface 160 reads the image data from the storage unit 210 and transmits the image data to the processing unit 130. The processing unit 130 performs the processing described above in FIG. 3 on the image data from the input unit 140 or the storage interface 160 as the image to be processed. As a result, the processing unit 130 outputs the corrected image via the output unit 150, and the corrected image in focus is displayed on the display unit 330.

[0063] The endoscope system 300 of this embodiment may have a configuration example shown in FIG. 15, for example. The configuration example of FIG. 15 differs from the configuration example of FIG. 14 in that the information processing system 100 and the processor unit 200 are provided separately. The information processing system 100 and the processor unit 200 may be connected via device-to-device communication such as USB, or via network communication such as LAN or WAN. The information processing system 100 is configured by one or more information processing devices. When the information processing system 100 is configured by multiple information processing devices, the information processing system 100 may be a cloud system in which multiple PCs or multiple servers connected via a network perform parallel processing. The storage unit 170 in FIG. 15 corresponds to the storage unit 210 in FIG. 14.

[0064] The processor unit 200 includes a control unit 220, an imaging data receiving unit 230, an input unit 240, an output unit 250, a processing unit 260, and a display interface 270. The imaging data receiving unit 230 is configured with an interface circuit and the like similar to the input unit 140 in FIG. 14 , and receives imaging data from the endoscope 310. The processing unit 260 transmits the image data received by the imaging data receiving unit 230 to the information processing system 100 via the output unit 250. The information processing system 100 performs the processing of FIG. 3 on the received image data as a processing target image to generate a corrected image. The input unit 240 receives the corrected image transmitted from the information processing system 100 via the output unit 150 and outputs the corrected image to the processing unit 260. The processing unit 260 outputs the corrected image to the display unit 330 via the display interface 270. As a result, the corrected image is displayed on the display unit 330. Display interface 270 in Fig. 15 is configured with the same hardware as output unit 150 in Fig. 14, and realizes the same functions as output unit 150 in Fig. 14. Note that in Fig. 15, input unit 140 and output unit 150 of information processing system 100 may be configured with separate interfaces, or the functions of input unit 140 and output unit 150 may be realized with a single input / output interface. The same applies to input unit 240 and output unit 250 of processor unit 200.

[0065] The method of this embodiment is not limited to the above, and various modifications are possible. For example, each object distance included in the optical system information 40 may be determined based on the difference between the corresponding MTFs. For example, the training image group 32G is assumed to be composed of a first training image 32-1 obtained by performing step S200-1 based on the transfer function or point spread function of the first object distance, and a second training image 32-2 obtained by performing step S200-2 based on the transfer function or point spread function of the second object distance. The first object distance is assumed to have a larger defocus amount than the second object distance. In this case, the spatial frequency dependence of the MTF is qualitatively illustrated as A0 in FIG. 16 , with the MTF based on the second object distance as shown by A1. Then, for example, by determining a predetermined spatial frequency indicated by B0, the difference between the MTFs is determined as indicated by C0. Therefore, the first object distance and the second object distance are determined so that the difference between the MTFs indicated by C0 is smaller than a predetermined value.

[0066] The MTF difference here refers to the difference between MTFs at adjacent object distances. For example, assume that the training image group 32G includes a first training image 32-1, a second training image 32-2, and a third training image 32-3. Assume that the object distances with the largest defocus amounts are the first, second, and third object distances. In this case, A10 in FIG. 17 indicates the frequency characteristics of the MTF at the third object distance, A11 indicates the frequency characteristics of the MTF at the second object distance, and A12 indicates the frequency characteristics of the MTF at the first object distance. At a predetermined frequency indicated by B0, both the difference between the MTFs of A10 and A11 indicated by C10 and the difference between the MTFs of A11 and A12 indicated by C11 are lower than a predetermined value. In other words, at the predetermined frequency indicated by B0, the difference between the MTFs of A10 and A12 is not considered a predetermined value. For the above reasons, in the information processing system 100 of this embodiment, the object distance is set so that the difference in MTF values ​​between adjacent object distances is equal to or less than a predetermined value at a predetermined spatial frequency of the MTF of the first imaging system 101. This allows for an appropriate combination of datasets in machine learning. As described above, the machine-learned trained model 120 performs a correction process (step S30) so that both the first training image 32-1 and the second training image 32-2 can be corrected to the correct image 36. Furthermore, to correct a processing target image captured at an object distance between the first object distance and the second object distance to the correct image 36 by the correction process (step S30), it is preferable that the difference in the influence of blur added to the first training image 32-1 and the second training image 32-2 be within a certain range. In this regard, by applying the method of this embodiment, the object distance of each training image is defined based on the MTF indicating the degree of influence of blur simulated on the predetermined subject image 30, and therefore an appropriate training image group 32G can be generated. This allows for an appropriate dataset to be obtained in machine learning.

[0067] The optical system information 40 may also include the object distance under the best focus condition of the first imaging system 101. Specifically, the object distance under the best focus condition is, for example, the distance shown as D3 in FIG. 9. For example, the learning device processing unit 16 may generate the correct image 36 by performing a best focus simulation process (step S300) on the predetermined subject image 30 using a transfer function or a point spread function that uses the object distance under the best focus condition. In other words, in the information processing system 100 of this embodiment, the object distance at which focus is achieved is the object distance under the best focus condition. In this way, an appropriate correct image 36 can be generated.

[0068] In this embodiment, it is assumed that there is a one-to-one correspondence between the transfer function or point spread function based on the object distance and the training image 32. More specifically, for example, in the defocus simulation process (step S200), a process of generating a third training image 32-3 using both a transfer function or point spread function based on a first object distance and a transfer function or point spread function based on a second object distance for one predetermined object image 30 is not performed. In other words, in the information processing system 100 of this embodiment, each training image 32 is an image generated by performing the defocus simulation process (step S200) on a predetermined object image 30 based on a transfer function or point spread function at any one of a plurality of object distances. This makes it possible to clarify the relationship between each training image 32 in the training image group 32G.

[0069] In a typical optical system, the MTF decreases and changes periodically as the spatial frequency increases. However, since the MTF is an absolute value, the MTF is displayed while being aliased in the high spatial frequency region shown in B1 of FIG. 17. Therefore, in the high spatial frequency region, it is not possible to uniquely determine which object distance an MTF of 1 corresponds to. Note that, for example, the MTF for an object distance shorter than the object distance at the near point of the target extended depth of field shown in P2 of FIG. 9 may be 0 at the spatial frequency shown in B0. For example, assuming that A12 in FIG. 17 is the MTF at the object distance at the near point of the target extended depth of field, the spatial frequency shown in B0 should be lower than the lowest spatial frequency at which aliasing occurs. This is because the transfer function or point spread function for object distances outside the target depth of field are not used in the machine learning of this embodiment. Note that, as described above, the target extended depth of field here does not indicate a constant value. From the above, in the information processing system 100 of this embodiment, the processing unit 130 uses the trained model 120 to correct blur caused by defocusing of the first imaging system 101 for the processing target image, thereby estimating an image in which the depth of field of the first imaging system 101 is extended to a target extended depth of field that is wider than the depth of field. Furthermore, the predetermined spatial frequency is a spatial frequency lower than the lowest spatial frequency at which the MTF value at the near point of the target extended depth of field becomes zero. This makes it possible to appropriately determine the range of predetermined spatial frequencies required to establish a one-to-one correspondence between spatial frequencies and MTFs.

[0070] More specifically, the predetermined spatial frequency indicated by B0 is preferably 0.1 as a normalized frequency, for example. That is, in the information processing system 100 of this embodiment, the predetermined spatial frequency is a spatial frequency that is 1 / 5 the Nyquist frequency of the image sensor of the first imaging system 101. By doing so, it is possible to establish a one-to-one correspondence between spatial frequency and MTF for many optical systems. This allows the method of this embodiment to be applied to processing target images captured by many types of optical systems.

[0071] Furthermore, the optical system information 40 of this embodiment may be a combination of an object distance within the depth of field and an object distance outside the depth of field. Specifically, for example, the optical system information 40 may include a first object distance outside the depth of field as shown in D1 of FIG. 9 and a second object distance as shown in D2. In other words, in the information processing system 100 of this embodiment, the first object distance among the multiple object distances is an object distance outside the depth of field, and the second object distance among the multiple object distances is an object distance within the depth of field. By doing so, a data set can be created by combining the first training image 32-1, which simulates a large effect of blur, and the second training image 32-2, which simulates a small effect of blur, with the correct image 36 through the defocus simulation process (step S200). This allows the trained model 120, which has been machine-learned using these data sets, to correct a target image affected by blur over a wide range through the correction process (step S30).

[0072] The predetermined value may also be determined based on the number of training images 32 constituting the training image group 32G. For example, in FIG. 16, the MTF indicated by A0 is the MTF at the object distance corresponding to the best focus condition, and the MTF indicated by A1 is the MTF at the object distance corresponding to the near point of the target depth of field. In this case, for example, if the spatial frequency is determined to be the spatial frequency indicated by B0, the MTF range having the range indicated by C0 as the maximum is uniquely determined. The range indicated by C0 is then divided based on the number of desired training images 32, and the predetermined value is determined. For these reasons, in the information processing system 100 of this embodiment, the predetermined value is determined based on the number of object distances that can be set to two or more. In this way, the number of datasets required for machine learning can be determined taking into account the load of machine learning.

[0073] As mentioned above, since the range of the MTF is uniquely determined when the spatial frequency is fixed, a predetermined value may be determined in advance and the number of training images 32 may be determined based on that predetermined value, and the user may decide the machine learning policy according to the circumstances.

[0074] The predetermined value is preferably 0.2 or less. That is, in the information processing system 100 of this embodiment, the predetermined value is set to 0.2 or less. In a typical optical system, if the spatial frequency described above is determined to be within a desirable range, the possible range of MTF is considered to be approximately 0.2. Therefore, for example, if the predetermined value is set to 0.2, the number of training images 32 constituting the training image group 32G will be 2. In this case, the first object distance is considered to be an object distance outside the depth of field, and the second object distance is considered to be an object distance within the depth of field.

[0075] Furthermore, the predetermined value is preferably 0.1 or less. That is, in the information processing system 100 of this embodiment, the predetermined value is set to 0.1 or less. Furthermore, the predetermined value is preferably 0.05 or less. That is, in the information processing system 100 of this embodiment, the predetermined value is set to 0.05 or less. By doing so, the number of training images 32 constituting the training image group 32G can be further increased. As a result, when a processing target image captured at an object distance other than the object distance not used in machine learning is input, the trained model 120 is more likely to be able to output a corrected image in which the effects of blur have been appropriately removed. That is, the accuracy of the correction process (step S30) of the trained model 120 can be further improved. Note that, as the number of training images 32 constituting the training image group 32G increases, the processing load of machine learning increases. Therefore, the appropriate number of training images 32 constituting the training image group 32G is determined appropriately depending on the circumstances.

[0076] Next, a specific method in which the learning device processing unit 16 performs the defocus simulation process (step S200) using the point spread function will be described. For example, when generating a first learning image 32-1 in step S200-1, as shown in FIG. 18, the learning device processing unit 16 performs a convolution operation on the predetermined object image 30 using the PSF of the first object distance of the first imaging system 101. Note that convolution can also be called a convolution integral. The PSF of the first object distance here is a PSF consisting of the region shown as FC22-1 in FIG. 12. That is, in the case of the method shown in FIG. 18, the convolution operation of the PSF corresponds to step S210 in FIG. 13. Similarly, when generating an Nth learning image 32-N in step S200-N, the learning device processing unit 16 performs a convolution operation on the predetermined object image 30 using the PSF of the Nth object distance of the first imaging system 101. The defocus simulation process based on the convolution calculation process of the PSF can be referred to as step S200-A. From the above, in the information processing system 100 of this embodiment, the defocus simulation process (step S200) is a process of performing convolution calculation of the PSF at each object distance of the first imaging system 101 for the predetermined subject image 30. In this way, it is possible to generate a trained model 120 that has been machine-learned using a dataset of the training images 32 and the ground truth images 36 using the PSF.

[0077] Next, the learning device processing unit 16 butA specific method for performing the defocus simulation process (step S200) using a transfer function will be described. For example, when generating a first learning image 32-1, as shown in FIG. 19, the learning device processing unit 16 performs a Fourier transform on the predetermined object image 30, multiplies the frequency characteristics resulting from the Fourier transform by the OTF of the first object distance of the first imaging system 101, and performs an inverse Fourier transform on the frequency characteristics obtained by the multiplication. The OTF for the first object distance here is an OTF consisting of the region indicated by FC22-1 in FIG. 12. In other words, in the case of the method shown in FIG. 19, the multiplication of the OTF corresponds to step S210 in FIG. 13. Similarly, when generating an Nth learning image 32-N by step S200-N, the learning device processing unit 16 performs a Fourier transform on the predetermined object image 30, multiplies the frequency characteristics resulting from the Fourier transform by the OTF of the Nth object distance of the first imaging system 101, and performs an inverse Fourier transform on the multiplied frequency characteristics. Note that the defocus simulation process based on the multiplication of the OTF can be referred to as step S200-B. From the above, in the information processing system 100 of this embodiment, the defocus simulation process (step S200) is a process of performing a Fourier transform on the predetermined object image 30, multiplying the frequency characteristics of the predetermined object image 30 resulting from the Fourier transform by the OTF at each object distance of the first imaging system 101, and performing an inverse Fourier transform on the multiplied frequency characteristics. In this way, it is possible to generate a trained model 120 that has been machine-learned using a dataset of the training images 32 and the ground truth images 36 using the OTF.

[0078] Since the relationship between PSF and OTF is as described above, the calculation processing result related to the processing in Fig. 18 is mathematically equivalent to the calculation processing result related to the processing in Fig. 19. In the defocus simulation processing (step S200), the user can select either PSF or OTF to use as appropriate.

[0079] Similarly, the learning device processing unit 16 may perform a best focus simulation process (step S300) using a point spread function. For example, as shown in Fig. 20, the learning device processing unit 16 performs a convolution calculation process on the predetermined subject image 30 using a PSF at an object distance at which the first imaging system 101 is in focus, thereby generating a ground truth image 36. Note that the best focus simulation process based on the convolution calculation process of the PSF can also be referred to as step S300-A.

[0080] The learning device processing unit 16 may also perform a best focus simulation process (step S300) using a transfer function. For example, as shown in Fig. 21, the learning device processing unit 16 performs a Fourier transform on the predetermined object image 30, a process of multiplying the frequency characteristics resulting from the Fourier transform by the OTF of the object distance at which the first imaging system 101 is in focus, and a process of performing an inverse Fourier transform on the frequency characteristics resulting from the multiplication, thereby generating a ground truth image 36. The best focus simulation process based on the multiplication of the OTF may also be referred to as step S300-B.

[0081] In the following explanation, a representative example will be given in which a method using a PSF is applied to generate the training image 32 and the correct image 36, but this does not preclude the application of a method using an OTF.

[0082] Furthermore, for example, the first imaging system 101 of this embodiment may have a retrofocus type lens configuration. The retrofocus type is also called a reverse telephoto type. For example, a retrofocus type lens configuration is realized by arranging a lens with negative refractive power and a lens with positive refractive power from the subject side. In the following description, the lens group on the subject side will be called the front lens group, and the lens group on the image side will be called the rear lens group.

[0083] A variety of well-known configurations can be employed for the specific lens configuration of a retrofocus type. For example, in the case of the optical system shown in FIG. 22, from the subject side, the optical system is configured with a front lens group indicated by G1, an aperture stop indicated by S1, a rear lens group indicated by G2, and a cover glass indicated by CG1. Note that for the sake of convenience, the spacing between the lenses and other components of the optical system is not precisely illustrated in FIG. 22. For example, in FIG. 22, the positive lens indicated by L6 and the cover glass indicated by CG1 are actually cemented together, but are illustrated with a gap between them for the sake of convenience. The same applies to FIGS. 23 and 25, which will be described later.

[0084] 22, the front lens group shown in G1 includes an object-side negative lens shown in L1 and a positive lens shown in L2, and has a negative refractive power as a whole. The rear lens group shown in G2 includes a positive lens shown in L3, a lens formed by cementing a positive lens and a negative lens shown in L4 and a positive lens shown in L5, and has a positive refractive power as a whole.

[0085] The front lens group or the rear lens group may be composed of multiple lens groups. For example, in the first imaging system 101 shown in Fig. 23, the lens group indicated by G11 functions as the front lens group, and the lens groups indicated by G12 and G13 function as the rear lens groups. For example, the lens group indicated by G11 includes, in order from the subject side, a plano-concave lens with its concave surface facing the image side as indicated by L11, and a negative meniscus lens as indicated by L12, and has negative refractive power as a whole.

[0086] Furthermore, for example, the lens group shown in G12 includes an object-side positive lens shown in L13 and an image-side positive lens shown in L14. Note that an aperture stop shown in S11 may be further disposed between the lens shown in L13 and the lens shown in L14. By doing so, the optical system is configured so that the refractive power is symmetrical on either side of the aperture stop, thereby enabling good correction of coma and astigmatism.

[0087] The lens group shown in G13 has a positive refractive power as a whole. The lens group shown in G13 may also include a cemented lens composed of a positive lens shown in L15 and a negative lens shown in L16. This allows for good correction of spherical aberration and coma aberration. The lens group shown in G13 may also include a plano-convex lens shown in L17. This allows for a wide field of view. Although FIG. 23 shows the plano-convex lens shown in L17 and the cover glass shown in CG11 separated from each other, they are actually cemented together. The cover glass shown in CG11 is provided on an image sensor (not shown), and the plano-convex lens shown in L17 is used to position the image sensor.

[0088] Furthermore, for example, the first imaging system 101 may further include a parallel plate. The parallel plate is also called a filter. The parallel plate is disposed, for example, at position F1 in FIG. 22 and position F11 in FIG. 23, but it can also be disposed at other positions. The parallel plate is used, for example, to adjust the position of the image point.

[0089] Furthermore, in the first imaging system 101 including the retrofocus lens configuration described above, it is desirable that the amount of distortion at the maximum angle of view be −30% or less. Specifically, for example, assume that the object shown in E1 of FIG. 24 is captured as the image shown in E2 of FIG. 24 by the first imaging system 101. In this case, the value of the amount of distortion (%) at the maximum angle of view can be expressed as (AD−PD) / PD×100, using the length shown in PD of the object shown in E1 and the length shown in AD of the image shown in E2. It is desirable that this value be negative, less than −30. From the above, in the information processing system 100 of this embodiment, the first imaging system 101 has a retrofocus lens configuration, and the amount of distortion at the maximum angle of view is −30% or less. This configuration reduces the magnification in the periphery compared to the center of the image, thereby reducing the transfer function or point spread function in regions other than on the optical axis. Furthermore, it is possible to reduce the difference between the transfer function or point spread function in the region on the optical axis and the transfer function or point spread function in the region other than on the optical axis. This allows the generation of training images 32 that simulate the effects of blur more accurately.

[0090] The front lens group or the rear lens group may be composed of a single lens. For example, the first imaging system 101 shown in FIG. 25 includes a lens group indicated by G21, a lens group indicated by G22, an aperture stop indicated by S21, a lens group indicated by G23, and a cover glass indicated by CG21. The lens group indicated by G21 includes a single negative lens indicated by L21 and has negative refractive power. In other words, the lens group indicated by G21 functions as part of the front lens group. The lens group indicated by G23 includes a positive lens indicated by L23, a lens formed by cementing a positive lens indicated by L24 and a negative lens indicated by L25, and a positive lens indicated by L26, and has positive refractive power as a whole. In other words, the lens group indicated by G23 functions as the rear lens group.

[0091] Furthermore, the first imaging system 101 of this embodiment may further include a phase modulation element. For example, the second lens group G2 in Fig. 25 includes a positive lens indicated by L22, an aperture stop indicated by S21, and a phase modulation element indicated by PM. The phase modulation element indicated by PM is disposed at the position of the pupil of the first imaging system 101. The phase modulation element indicated by PM is an element to which wavefront coding (WFC) is applied, and has, for example, a phase modulation surface indicated by PMS. Note that wavefront coding is a well-known technique used in extended depth of field (EDOF), and therefore a detailed description thereof will be omitted.

[0092] In Fig. 25, the phase modulation surface indicated by PMS is shown as being expressed by a predetermined cubic function using coordinates orthogonal to the optical axis, but the surface shape of the phase modulation surface is not limited to this, and other surface shapes may be adopted. Also, in Fig. 25, the phase modulation surface is shown on the image side, but the same effect can be obtained even if it is located on the subject side. Furthermore, the lens group indicated by G22 has positive refractive power as a whole and also functions as part of a retrofocus type front lens group.

[0093] Furthermore, by including the phase modulation element shown in PM, the MTF of the first imaging system 101 changes less rapidly with defocus. In other words, by including the phase modulation element, the MTF of the first imaging system 101 acts to match the change in object distance. More specifically, for example, the difference between the MTF at the first object distance and the MTF at the second object distance in the first imaging system 101 that includes the phase modulation element is smaller than the difference between the MTF at the first object distance and the MTF at the second object distance in the first imaging system 101 that does not include the phase modulation element.

[0094] For example, in the relationship between MTF and spatial frequency shown in FIG. 26, A20 is the MTF of the first imaging system 101 at an in-focus object distance, A21 is the MTF at an object distance with a larger defocus amount than the object distance associated with A20, and A22 is the MTF at an object distance with a larger defocus amount than the object distance associated with A21. Also, A20 to A22 are the MTFs of the first imaging system 101 that does not include a phase modulation element. When the predetermined spatial frequency indicated by B0 is determined, the difference between the MTFs of A20 and A21 is the difference indicated by C20, and the difference between the MTFs of A21 and A22 is the difference indicated by C21. Note that in FIG. 26, the MTFs at frequencies higher than the spatial frequency indicated by B0 are partially omitted.

[0095] Here, by including the phase modulation element shown in PM in the first imaging system 101, the MTF shown in A20 changes to the MTF shown in A30, the MTF shown in A21 changes to the MTF shown in A31, and the MTF shown in A22 changes to the MTF shown in A32. Furthermore, the difference in the MTFs shown in C20 becomes smaller as shown in C30, and the difference in the MTFs shown in C21 becomes smaller as shown in C31. From the above, in the information processing system 100 of this embodiment, the first imaging system 101 further includes an optical wavefront modulation element that changes the transfer function or point spread function. This shortens the distance required for machine learning, thereby reducing the number of data sets required for machine learning.

[0096] The above-described example of the defocus simulation process (step S200) is performed by using the optical information of the first imaging system 101 for the predetermined object image 30 captured by the arbitrary imaging system 104. to Based on Training image 32 However, the method of this embodiment is not limited to these examples. For example, the learning device processing unit 16 may perform a defocus simulation process that further includes a process of simulating the removal of the influence of imaging by an arbitrary imaging system 104 from the predetermined subject image 30.

[0097] 27 shows an example of image data generation processing in the case where a process for simulating the removal of the influence of the first imaging system 101 on the predetermined object image 30-1 captured by the first imaging system 101 is further included. The image data generation processing shown in FIG. 27 can also be referred to as step S122. Comparing step S122 in FIG. 27 with step S120-2 in FIG. 11, the content of the defocus simulation processing is different. FIG. 27 is similar to FIG. 11 in that the best focus simulation processing (step S300) is not performed and the target image 36 is the predetermined object image 30-1 itself. This is because the predetermined object image 30-1 is an image captured under the best focus conditions of the first imaging system 101, and therefore there is no need to perform processing similar to step S202.

[0098] 28 shows an example of the defocus simulation process (step S202-1) in the image data generation process (step S122). For example, when generating the first learning image 32-1, the learning device processing unit 16 performs a process (step S220-1) on the predetermined object image 30-1 to simulate the removal of the influence of the first imaging system 101 when capturing the predetermined object image 30-1. Step S220-1 is performed based on the transfer function or point spread function at the object distance at which the first imaging system 101 is in focus and the transfer function or point spread function of the first imaging system 101 at the first object distance.

[0099] More specifically, the learning device processing unit 16 performs an appropriate combination of, for example, a calculation process for deconvolving the PSF at an object distance where the first imaging system 101 is focused on the predetermined subject image 30 and a calculation process for convolving the PSF at the first object distance of the first imaging system 101 (step S200-A). The appropriate combination of calculation processes refers to a calculation process in which one calculation process and part or all of the other calculation process are combined in any order. However, this does not prevent one calculation process from being performed separately from the other calculation process, and the order is determined appropriately according to predetermined circumstances. The same applies to the following explanation. Note that predetermined circumstances include, for example, the processing time required for machine learning, the processing load on the processor, etc. In other words, by performing step S220-1, it is possible to obtain a calculation result that reflects both the effect of the calculation process that deconvolves the PSF at the object distance at which the first imaging system 101 is focused on the specified subject image 30-1, and the effect of the calculation process (step S200-A) that convolves the PSF at the first object distance of the first imaging system 101.

[0100] From the above, in the information processing system 100 of this embodiment, the arbitrary imaging system 104 is the first imaging system 101. The defocus simulation process (step S202) further includes a process (step S212) of removing the influence of the first imaging system 101 from the predetermined object image 30-1 based on the transfer function or point spread function at the object distance where the first imaging system 101 is in focus and the transfer functions or point spread functions at multiple object distances of the first imaging system 101. This allows for the generation of a more accurate training image 32. The training image 32 and the correct image 36 obtained by the methods shown in FIGS. 10 and 11 are influenced by both the arbitrary imaging system 104 and the first imaging system 101 on the predetermined object, whereas the training image 32 and the correct image 36 obtained by the methods shown in FIGS. 27 and 28 are influenced by only the first imaging system 101 on the predetermined object. This allows for machine learning using a more appropriate dataset.

[0101] Similarly, Fig. 29 shows an example of image data generation processing including processing that simulates the removal of the influence of imaging by an arbitrary imaging system 104. In Fig. 29, the second imaging system 102 is illustrated as a representative of the arbitrary imaging system 104. The second imaging system 102 is also assumed to be an imaging system having an imaging element with a higher resolution than the first imaging system 101. The image data generation processing shown in Fig. 29 can also be referred to as step S124, and the image that is the source of step S124 can also be referred to as a predetermined subject image 30-2.

[0102] 10, step S126 in FIG. 29 differs from step S120-1 in that image sensor information 50 is further read, and then a defocus simulation process (step S204) and a best focus simulation process (step S304) are performed. The image sensor information 50 is information related to the resolution of the image sensors of the first image sensor system 101 and any image sensor system 104. That is, in the example of FIG. 29, the learning device storage unit 18 further stores image sensor information 50, which is not shown in FIG. 4. The image sensor information 50 is also used in the calculation processes of the defocus simulation process (step S204) and the best focus simulation process (step S304).

[0103] FIG. 30 shows an example of defocus simulation processing in the image data generation processing (step S124) shown in FIG. 29. The defocus simulation processing shown in FIGS. 29 and 30 can also be referred to as step S204. For example, when generating a first learning image 32-1, the learning device processing unit 16 performs arithmetic processing on a predetermined object image 30-2, appropriately combining a process of simulating the difference between the second imaging system 102 and the first imaging system 101 (step S230-1), a process of reducing the predetermined object image 30 (step S240), and arithmetic processing based on image sensor information 50 (not shown in FIG. 30). Step S230-1 is performed based on the transfer function or point spread function at the object distance at which the second imaging system 102 is in focus and the transfer function or point spread function of the first imaging system 101 at the first object distance. That is, by performing step S230-1, it is possible to obtain a calculation processing result that reflects both the effect of the calculation processing that deconvolves the PSF at the object distance where the second imaging system 102 is focused on the predetermined subject image 30-2, and the effect of the calculation processing (step S200-A) that convolves the PSF at the first object distance of the first imaging system 101. Furthermore, by performing step S204-1, it is possible to obtain a calculation processing result that reflects the effect of the calculation processing of step S230-1, the effect of the calculation processing of step S240, and the effect of the calculation processing based on the image sensor information 50.

[0104] FIG. 31 shows an example of the best focus simulation process shown in FIG. 29. The best focus simulation process shown in FIGS. 29 and 31 can also be referred to as step S304. For example, the learning device processing unit 16 performs processing that appropriately combines, for the predetermined object image 30-2, a process of simulating the difference between the second imaging system 102 and the first imaging system 101 (step S330), a process of reducing the predetermined object image 30-2 (step S340), and a calculation process based on the image sensor information 50 (not shown in FIG. 31). This allows the learning device processing unit 16 to generate the correct image 36. Step S330 in FIG. 31 is performed based on the transfer function or point spread function at the object distance at which the second imaging system 102 is in focus and the transfer function or point spread function at the object distance at which the first imaging system 101 is in focus. That is, by performing step S330, it is possible to obtain a calculation processing result that reflects both the effect of the calculation processing of deconvolving the PSF at the object distance at which the second imaging system 102 focuses on the predetermined object image 30-2 and the effect of the calculation processing (step S300-A) of convolving the PSF at the object distance at which the first imaging system 101 focuses. Furthermore, step S340 in FIG. 31 is a calculation processing similar to step S240 in FIG. 30. Furthermore, by performing step S304, it is possible to obtain a calculation processing result that reflects the effect of the calculation processing of step S330, the effect of the calculation processing of step S340, and the effect of the calculation processing based on the image sensor information 50. Note that the reference image 36 may be generated by a process that omits step S330 from the best focus simulation process (step S304) in FIG. 31. In other words, the reference image 36 may be generated by performing a process equivalent to step S340 on the predetermined object image 30-2. If the specified subject image 30-2 is an image captured at an object distance at which any imaging system 104 is in focus, it may be possible to make the specified subject image 30-2 into the correct image 36 by changing the number of pixels in step S340.

[0105] From the above, in the information processing system 100 of this embodiment, the defocus simulation process (step S204) further includes a process (step S230) of simulating the difference between the arbitrary imaging system 104 and the first imaging system 101, and a process (step S240) of reducing the predetermined object image 30-2. The target image 36 is an image generated by performing the best focus simulation process (step S304), or an image generated by performing a process of reducing the predetermined object image 30-2. The process (step S230) of simulating the difference between the arbitrary imaging system 104 and the first imaging system 101 in the defocus simulation process (step S204) is based on a transfer function or a point spread function at an object distance at which the arbitrary imaging system 104 is in focus, and a transfer function or a point spread function at a plurality of object distances of the first imaging system 101. The best focus simulation process (step S304) further includes a process (step S330) of simulating the difference between the arbitrary imaging system 104 and the first imaging system 101, and a process (step S340) of reducing the predetermined subject image 30-2. The process (step S330) of simulating the difference between the arbitrary imaging system 104 and the first imaging system 101 in the best focus simulation process (step S304) is based on a transfer function or a point spread function at an object distance where the arbitrary imaging system 104 is in focus, and a transfer function or a point spread function at an object distance where the first imaging system 101 is in focus.

[0106] The method of this embodiment can also be applied when the imaging system 104 and the first imaging system 101 have different imaging methods. For example, as shown in FIG. 32, the first imaging system 101 includes a simultaneous imaging element 106. Also, as shown in FIG. 33, the imaging system 104 includes a monochrome imaging element 108. The image data generation process in this case will be described with reference to FIG. 33. The image data generation process in FIG. 33 can also be referred to as step S126, and the image used as the source of step S126 can also be referred to as a predetermined subject image 30-3. FIG. 33 differs from FIG. 29 in the contents of the defocus simulation process (step S206) and the best focus simulation process (step S306), and in the fact that a color shift determination process (step S190) is performed before steps S206 and S306. In FIG. 33, the second imaging system 102 is illustrated as a representative example of the imaging system 104, just like the example in FIG. 29. The color shift determination process (S190) compares the amount of color shift around saturated areas in the predetermined subject image 30-3 with a predetermined threshold. Note that color shift refers to the shift that occurs between the R, G, and B images due to differences in image capture timing when capturing an image of a subject using the monochrome image sensor 108. Note that color shift does not occur in the processing target image captured using the simultaneous image sensor 106. The amount of color shift around saturated areas in the predetermined subject image 30-3 refers to the amount of color shift that occurs around areas that appear white in the predetermined subject image 30-3. In other words, steps S206 and S306 in FIG. 33 use the predetermined subject image 30-3 for which the amount of color shift around saturated areas is determined to be below the predetermined threshold in step S190. Thus, by performing step S206, a training image 32 with reduced effects of color shift can be generated. Similarly, by performing step S306, a target image 36 with reduced effects of color shift can be generated. This makes it possible to generate a data set consisting of appropriate training images 32 and correct images 36 when the imaging methods of the arbitrary imaging system 104 and the first imaging system 101 are different.

[0107] FIG. 34 shows an example of defocus simulation processing in the image data generation processing (step S126) shown in FIG. 33. The defocus simulation processing shown in FIGS. 33 and 34 can also be referred to as step S206. FIG. 34 differs from FIG. 30 in that it further includes processing for generating a mosaic image from the predetermined object image 30-3 (step S250) and processing for demosaicing the mosaic image (step S252). For example, when generating the first learning image 32-1, the learning device processing unit 16 performs arithmetic processing on the predetermined object image 30-3, which is an appropriate combination of the above-described steps S230-1, S240, S250, and S252, as well as arithmetic processing based on image sensor information 50 (not shown in FIG. 34). In other words, by performing step S206-1, it is possible to obtain a calculation result that reflects the effect of the calculation processing of step S230-1, the effect of the calculation processing of step S240, the effect of the calculation processing of step S250, the effect of the calculation processing of step S252, and the effect of the calculation processing based on the image sensor information 50.

[0108] Steps S250 and S252 will be described in detail. The predetermined subject image 30-3 is a frame-sequential image obtained by a process of synthesizing a plurality of images captured by the monochrome image sensor 108 at the timing when light of a plurality of wavelength bands is sequentially irradiated. Then, as shown in FIG. 35, for example, in the above-mentioned step S206-1, a mosaic image is generated by a process including step S250. Then, a frame-sequential image is generated again from the mosaic image by a process including step S252, thereby generating a first learning image 32-1. Note that in step S206-1 of FIG. 35, processes other than steps S250 and S252 are not shown.

[0109] FIG. 36 shows an example of best focus simulation processing in the image data generation processing (step S126) shown in FIG. 33. The best focus simulation processing shown in FIGS. 33 and 36 can also be referred to as step S306. FIG. 36 differs from FIG. 31 in that it further includes processing for generating a mosaic image from a predetermined object image 30-3 (step S350) and processing for demosaicing the mosaic image (step S352). Step S350 in FIG. 36 is similar to step S250 in FIG. 34, and step S352 in FIG. 36 is similar to step S252 in FIG. 34. For example, the learning device processing unit 16 performs arithmetic processing that appropriately combines the above-described step S330-1, the above-described step S340, step S350, step S352, and arithmetic processing based on image sensor information 50 (not shown in FIG. 36). This allows the learning device processing unit 16 to generate a correct answer image 36. As a result, by performing step S306, it is possible to obtain a calculation processing result that reflects the effect of the calculation processing of step S330, the effect of the calculation processing of step S340, the effect of the calculation processing of step S350, the effect of the calculation processing of step S352, and the effect of the calculation processing based on the image sensor information 50. Note that the reference image 36 may be generated by a process that omits steps S330, S350, and S352 from the best focus simulation process (step S306). In other words, the reference image 36 may be generated by performing a process equivalent to step S340 on the predetermined subject image 30-3.

[0110] From the above, in the information processing system 100 of this embodiment, the arbitrary imaging system 104 includes a monochrome imaging element 108. The predetermined object image 30-3 is a frame sequential image obtained by combining multiple images captured by the monochrome imaging element 108 at the timing when light of multiple wavelength bands is sequentially irradiated. The first imaging system 101 includes a simultaneous imaging element 106 having multiple pixels of different colors, each pixel assigned a single color. The defocus simulation process (step S206) further includes a process of generating a mosaic image from the predetermined object image 30-3, in which each pixel is assigned a single color, a process of demosaicing the mosaic image, a process of simulating the difference between the arbitrary imaging system 104 and the first imaging system 101, and a process of reducing the predetermined object image 30-3. The process of simulating the difference between the arbitrary imaging system 104 and the first imaging system 101 in the defocus simulation process (step S206) is based on the transfer function or point spread function at the object distance at which the arbitrary imaging system 104 is in focus and the transfer function or point spread function at multiple object distances of the first imaging system 101. The ground truth image 36 is an image generated by performing the best focus simulation process (step S306) or an image generated by performing a process of reducing the predetermined object image 30-3. The best focus simulation process (step S306) further includes a process of generating a mosaic image, a process of demosaicing the mosaic image, a process of simulating the difference between the arbitrary imaging system 104 and the first imaging system 101, and a process of reducing the predetermined object image 30-3. The process of simulating the difference between the arbitrary imaging system 104 and the first imaging system 101 in the best focus simulation process (step S306) is based on the transfer function or point spread function at the object distance at which the arbitrary imaging system 104 is in focus, and the transfer function or point spread function at the object distance at which the first imaging system 101 is in focus. In this way, even if the imaging method of the predetermined subject image 30 and the imaging method of the processing target image are different, it is possible to generate a more appropriate data set of training images 32 and ground truth images 36.

[0111] Moreover, different trained models 120 may be used depending on the imaging method. That is, in the information processing system 100 of this embodiment, the storage unit 110 may store a first trained model 121 and a second trained model 122, for example, as shown in FIG.

[0112] When the storage unit 110 stores the first trained model 121 and the second trained model 122, the flow shown in FIG. 3 may be, for example, the flow shown in FIG. 38. After reading the image to be processed (step S10), the processing unit 130 performs a process of checking the imaging method of the first imaging system 101 (step S12). If the imaging method is the frame sequential method, the processing unit 130 reads the first trained model (step S21), performs correction processing (step S31), and outputs the corrected image (step S41). On the other hand, if the imaging method is the Bayer simultaneous imaging method, the processing unit 130 reads the second trained model (step S22), performs correction processing (step S32), and outputs the corrected image (step S42). Note that steps S21 and S22 in FIG. 38 correspond to step S20 in FIG. 3. 38. Similarly, steps S31 and S32 in FIG. 38 correspond to step S30 in FIG. 3, and steps S41 and S42 in FIG. 38 correspond to step S40 in FIG.

[0113] In this case, step S100 in FIG. 7 may be replaced by step S101 in FIG. 39 and step S102 in FIG. 40. Specifically, the first trained model creation process (step S101) in FIG. 39 may be replaced by step S100 in FIG. 7, with the image data generation process being replaced by step S124 in FIG. 29. Similarly, the second trained model creation process (step S10 2 ) can be achieved by replacing step S100 in FIG. 7 with image data generation processing in step S126 in FIG.

[0114] The method of this embodiment can also be applied when the observation methods are different between the arbitrary imaging system 104 and the first imaging system 101. A method of image data generation processing when the observation methods are different will be described using FIG. 41 . The image data generation processing in FIG. 41 can also be referred to as step S128, and the image that serves as the source of step S128 can also be referred to as the predetermined subject image 30-4. Step S128 in FIG. 41 differs from step S124 in FIG. 29 in the contents of the defocus simulation processing (step S208) and the best focus simulation processing (step S308), and in that observation method information 60 is read before steps S206 and S306 are performed. The observation method information 60 is, for example, information related to the observation method of the first imaging system 101. That is, in the example of FIG. 41 , the learning device storage unit 18 further stores observation method information 60, which is not shown in FIG. 4. 41, the second imaging system 102 is illustrated as a representative example of an arbitrary imaging system 104, just like the example in FIG.

[0115] The observation method can also be called an observation mode. Different observation methods may be, for example, when different light sources are used for observation, or when different image processing techniques are used between the time the user captures an image of the subject and the time the predetermined subject image 30-4 is acquired. Observation methods include, for example, a white light imaging (WLI) mode that uses white illumination light and a special light observation mode that uses special light other than white light. The special light observation mode includes a narrow band imaging (NBI) mode that uses two narrow band lights. The two narrow band lights are narrow band light included in the blue wavelength band and narrow band light included in the green wavelength band. WLI and NBI differ in the image processing used to generate a color image from the image signal output by the image sensor. For example, the content of the demosaicing or the parameters used in the image processing differ. Furthermore, a red dichromatic imaging (RDI) mode, for example, can be used as a special light observation mode. The RDI mode is an observation mode that uses narrowband light included in the amber wavelength band, narrowband light included in the green wavelength band, and narrowband light included in the red wavelength band, and uses technology disclosed in, for example, U.S. Patent No. 9,775,497 B2.

[0116] Fig. 42 shows an example of defocus simulation processing (step S208-1) for generating a first learning image 32-1 from a predetermined object image 30-4. Step S208-1 in Fig. 42 differs from step S204-1 in Fig. 30 in that it further includes processing for WLI mode (step S262), processing for NBI mode (step S264), processing for RDI mode (step S266), and processing for TXI mode (step S268). TXI stands for Texture and Color Enhancement Imaging, and details will be described later.

[0117] Note that step S128 in FIG. 41 is an example in which the above-described different processing is added to step S124 in FIG. 29, but this is not limiting. For example, the above-described different processing may be added to step S126 in FIG. 33. In this case, although not shown, the color shift determination processing (step S190) in FIG. 33 is further performed before steps S208 and S308 are performed. Also, in this case, step S208 in FIG. 42 further includes steps S240, S250, and S252 in FIG. 34. Similarly, step S308 in FIG. 43 in this case further includes steps S340, S350, and S352 in FIG. 36. Hereinafter, explanations of points that overlap with step S124 in FIG. 29 and step S126 in FIG. 33 will be omitted as appropriate.

[0118] For example, although not shown in the flowchart, the learning device processing unit 16 reads the observation method information 60 and acquires the observation method used in the first imaging system 101. Then, the learning device processing unit 16 selects one of steps S262, S264, S266, and S268 as the process corresponding to the acquired observation method.

[0119] For example, if the first imaging system 101 captures images in TXI mode, information to that effect is stored in the learning device storage unit 18 as observation method information 60. The learning device processing unit 16 then reads the observation method information 60 and performs a defocus simulation process (step S208) including a TXI mode process (step S368) on the predetermined object image 30-4. Specifically, for example, the learning device processing unit 16 performs a process of decomposing the predetermined object image 30-4 into a texture image portion, which is an image portion related to the surface structure of the predetermined object image 30-4, and a base image portion other than the texture image portion. The learning device processing unit 16 then performs a first process of emphasizing the surface structure related to the texture image portion, a second process of optimizing the brightness of the base image portion, and a third process of optimizing the color tone of an image obtained by combining the image related to the first process and the image related to the second process. This process allows a learning image 32 to be obtained that simulates the effect of capturing images in TXI mode for the predetermined object image 30-4. This allows machine learning to be performed on a dataset including more accurate learning images 32.

[0120] Furthermore, for example, although not shown, when the first imaging system 101 is capturing images in WLI mode or NBI mode, information to that effect is stored as observation method information 60 in the learning device storage unit 18. Then, the learning device processing unit 16 reads the observation method information 60 and performs color interpolation on the predetermined subject image 30-4 to correspond to the light source. Color interpolation may also be performed, for example, in conjunction with step S252 in FIG. 34. For example, when the learning device processing unit 16 selects WLI mode processing (step S262), it performs processing to interpolate the R image and the B image using the G image in conjunction with step S252. For example, when the learning device processing unit 16 selects NBI mode processing (step S264), it performs processing to independently interpolate the G image and the B image in conjunction with step S252.

[0121] FIG. 43 shows an example of a best focus simulation process (step S308) for generating a ground truth image 36 from a predetermined object image 30-4 in the image data generation process (step S128). Step S308 in FIG. 43 differs from step S304 in FIG. 31 in that it further includes processing for WLI mode (step S362), processing for NBI mode (step S364), processing for RDI mode (step S366), and processing for TXI mode (step S368). Step S362 in FIG. 43 is the same as step S262 in FIG. 42, step S364 in FIG. 43 is the same as step S264 in FIG. 42, step S366 in FIG. 43 is the same as step S266 in FIG. 42, and step S368 in FIG. 43 is the same as step S268 in FIG. 42. The correct image 36 may be generated by a process that omits steps S308 to S330 in the best focus simulation process of FIG.

[0122] Although the present embodiment has been described in detail above, those skilled in the art will readily understand that many modifications are possible without substantially departing from the novel features and effects of the present embodiment. Therefore, all such modifications are intended to be included within the scope of the present disclosure. For example, a term described at least once in the specification or drawings together with a different term with a broader or equivalent meaning may be replaced with that different term anywhere in the specification or drawings. Furthermore, all combinations of the present embodiment and modifications are also included within the scope of the present disclosure. Furthermore, the configurations and operations of the information processing system, endoscope system, trained model, information storage medium, and information processing method are not limited to those described in the present embodiment, and various modifications are possible. [Explanation of symbols]

[0123] 10...Learning device, 12...Communication unit, 16...Learning device processing unit, 18...Learning device memory unit, 20...Training model, 30...Specified subject image, 32...Learning image, 32-1...First learning image, 32-2...Second learning image, 32-N...Nth learning image, 32G...Learning image group, 34...Output image, 34-1...First output image, 34-N...Nth output image, 36...Correct image, 40...Optical system information, 50...Image sensor information, 60...Observation method information, 100...Information processing system, 101...First imaging system, 102...Second imaging system, 104...Arbitrary imaging system, 106...Homogeneous image sensor , 108... monochrome image sensor, 110... memory unit, 120... trained model, 121... first trained model, 122... second trained model, 130, 260... processing unit, 140, 240... input unit, 150, 250... output unit, 160... storage interface, 170, 210... storage unit, 200... processor unit, 220... control unit, 230... imaging data receiving unit, 270... display interface, 300... endoscope system, 310... endoscope scope, 320... operation unit, 330... display unit, NN... neural network

Claims

1. a storage unit that stores a trained model that has been machine-learned using a dataset including a training image group and a correct answer image; a processing unit that corrects blur caused by defocus of a first imaging system of a processing target image, which is an image captured by the first imaging system, using the trained model; An information processing system comprising: The training image group includes: a plurality of learning images generated by performing a defocus simulation process on a predetermined subject image captured by an arbitrary imaging system, the predetermined subject being focused by the arbitrary imaging system, to simulate the effect of blur caused by defocusing of the first imaging system based on a transfer function or a point spread function of the first imaging system at a plurality of object distances; the defocus simulation process is performed on an area on the optical axis of the first imaging system and an area other than the optical axis in each of the plurality of learning images, based on the transfer function on the optical axis or the point spread function; The correct image is an image generated by performing a best focus simulation process for simulating a state in which the first imaging system is in focus on the predetermined subject image based on the transfer function or the point spread function at an object distance in focus of the first imaging system, or the predetermined subject image itself; The trained model is Each of the training images is machine-learned to become the correct image; The first imaging system It is a retrofocus type lens configuration, An information processing system characterized in that the amount of distortion at the maximum angle of view is −30% or less.

2. 2. The information processing system according to claim 1, Each of the learning images is an information processing system characterized in that the image is generated by performing the defocus simulation processing on the specified subject image based on the transfer function or the point spread function at any one of the plurality of object distances.

3. 2. The information processing system according to claim 1, The first imaging system An information processing system further comprising an optical wavefront modulation element that changes the transfer function or the point spread function.

4. 2. The information processing system according to claim 1, The object distance is an information processing system, wherein a difference in MTF values ​​at adjacent object distances is set to be equal to or less than a predetermined value at a predetermined spatial frequency of the MTF of the first imaging system;

5. 5. The information processing system according to claim 4, The processing unit using the trained model to correct blur caused by defocusing of the first imaging system for the processing target image, thereby estimating an image in which the depth of field of the first imaging system has been extended to a target extended depth of field that is wider than the depth of field; The predetermined spatial frequency is an information processing system, characterized in that the spatial frequency is lower than the lowest spatial frequency at which the MTF value at the near point of the target extended depth of field becomes zero.

6. 5. The information processing system according to claim 4, The predetermined spatial frequency is an information processing system, wherein the spatial frequency is 1 / 5 of the Nyquist frequency of the image pickup element of the first image pickup system;

7. 5. The information processing system according to claim 4, The predetermined value is An information processing system, characterized in that the object distance is determined based on the number of object distances that can be set to two or more.

8. 5. The information processing system according to claim 4, The predetermined value is An information processing system, characterized in that the ratio is set to 0.2 or less.

9. 5. The information processing system according to claim 4, The predetermined value is An information processing system, characterized in that the value is set to be 0.1 or less.

10. 5. The information processing system according to claim 4, The predetermined value is An information processing system characterized in that the ratio is set to 0.05 or less.

11. 2. The information processing system according to claim 1, The defocus simulation process includes: An information processing system, characterized in that the information processing system performs a convolution operation on the predetermined subject image with respect to a PSF at each of the object distances of the first imaging system.

12. 2. The information processing system according to claim 1, The defocus simulation process includes: an information processing system characterized by performing a Fourier transform on the specified subject image, multiplying the frequency characteristics of the specified subject image resulting from the Fourier transform by the OTF at each of the object distances of the first imaging system, and performing an inverse Fourier transform on the multiplied frequency characteristics.

13. 2. The information processing system according to claim 1, The arbitrary imaging system is the first imaging system, The defocus simulation process includes: a process of removing an influence of the first imaging system from the predetermined object image based on the transfer function or the point spread function at the focused object distance of the first imaging system and the transfer function or the point spread function at the plurality of object distances of the first imaging system; An information processing system further comprising:

14. 2. The information processing system according to claim 1, The defocus simulation process includes: a process of simulating a difference between the arbitrary imaging system and the first imaging system based on the transfer function or the point spread function at the focused object distance of the arbitrary imaging system and the transfer function or the point spread function at the plurality of object distances of the first imaging system; a process of reducing the predetermined subject image; further comprising The correct image is an image generated by performing the best focus simulation process or an image generated by performing a process of reducing the predetermined subject image, The best focus simulation process includes: a process of simulating a difference between the arbitrary imaging system and the first imaging system based on the transfer function or the point spread function of the arbitrary imaging system at the focused object distance and the transfer function or the point spread function of the first imaging system at the focused object distance; A process including a process of reducing the predetermined subject image, An information processing system further comprising:

15. 2. The information processing system according to claim 1, The arbitrary imaging system is A monochrome image sensor is included. The predetermined subject image is a frame-sequential image obtained by a process of combining a plurality of images captured by the monochrome image sensor at the timing when light of a plurality of wavelength bands is sequentially irradiated, the plurality of images being captured by the monochrome image sensor at the timing when light of each of the wavelength bands is irradiated; The first imaging system a simultaneous imaging element having a plurality of pixels of different colors, each pixel being assigned one color; The defocus simulation process includes: A process of generating a mosaic image in which one color is assigned to each of the pixels from the predetermined subject image; demosaicing the mosaic image; a process of simulating a difference between the arbitrary imaging system and the first imaging system based on the transfer function or the point spread function at the focused object distance of the arbitrary imaging system and the transfer function or the point spread function at the plurality of object distances of the first imaging system; The process of reducing the predetermined subject image is performed by Further including, The correct image is an image generated by performing the best focus simulation process or an image generated by performing a process of reducing the predetermined subject image, The best focus simulation process includes: generating the mosaic image; demosaicing the mosaic image; a process of simulating a difference between the arbitrary imaging system and the first imaging system based on the transfer function or the point spread function at the object distance where the arbitrary imaging system is in focus and the transfer function or the point spread function at the object distance where the first imaging system is in focus; A process including a process of reducing the predetermined subject image, An information processing system further comprising:

16. 2. The information processing system according to claim 1, The object distance at which the focus is achieved is An information processing system, characterized in that the object distance is under best focus conditions.

17. 2. The information processing system according to claim 1, a first object distance among the plurality of object distances is the object distance outside a depth of field; An information processing system, wherein a second object distance among the plurality of object distances is the object distance within a depth of field.

18. a processor unit having an information processing system according to any one of claims 1 to 17; an endoscope connected to the processor unit and configured to capture the image to be processed; An endoscope system comprising:

19. A trained model used in an information processing system including a storage unit that stores a trained model, an input unit, a processing unit, and an output unit, the trained model being machine-learned using a dataset including a training image group and a correct answer image, The training image group includes: a plurality of learning images generated by performing a defocus simulation process on a predetermined subject image captured by an arbitrary imaging system, the predetermined subject being focused by the arbitrary imaging system, to simulate the effect of blur caused by defocusing of the first imaging system, based on a transfer function or a point spread function of the first imaging system at a plurality of object distances; the defocus simulation process is performed on an area on the optical axis of the first imaging system and an area other than the optical axis in each of the plurality of learning images, based on the transfer function on the optical axis or the point spread function; The correct image is an image generated by performing a best focus simulation process for simulating a state in which the first imaging system is in focus on the predetermined subject image based on the transfer function or the point spread function at an object distance in focus of the first imaging system, or the predetermined subject image itself; The trained model is Each of the training images is machine-learned to become the correct image; The input unit A processing target image, which is an image captured by the first imaging system, is input to the trained model; The processing unit performing a correction process for correcting blur caused by defocus of the first imaging system in the processing target image using the trained model; The output unit outputting a corrected image obtained by the correction processing; The first imaging system It is a retrofocus type lens configuration, The distortion amount at the maximum angle of view is -30% or less, A trained model characterized in that the processing unit functions to perform the correction processing.

20. An information storage medium storing the trained model according to claim 19.

21. An information processing method for correcting blur caused by defocus of a first imaging system of a processing target image, which is an image captured by a first imaging system, using a trained model machine-learned using a dataset including a training image group and a ground truth image, The training image group includes: a plurality of learning images generated by performing a defocus simulation process on a predetermined subject image captured by an arbitrary imaging system, the predetermined subject being focused by the arbitrary imaging system, to simulate the effect of blur caused by defocusing of the first imaging system based on a transfer function or a point spread function of the first imaging system at a plurality of object distances; the defocus simulation process is performed on an area on the optical axis of the first imaging system and an area other than the optical axis in each of the plurality of learning images, based on the transfer function on the optical axis or the point spread function; The correct image is an image generated by performing a best focus simulation process for simulating a state in which the first imaging system is in focus on the predetermined subject image based on the transfer function or the point spread function at an object distance in focus of the first imaging system, or the predetermined subject image itself; The trained model is Each of the training images is machine-learned to become the correct image; The first imaging system It is a retrofocus type lens configuration, An information processing method, characterized in that the amount of distortion at the maximum angle of view is −30% or less.

Images