Subject photographing image simulation method, image generating device, and program.

Abstract

To provide a method for simulating a captured image of a subject, an image generation device, and a program which allow for accurately generating a low-noise and high-reliability simulation of a captured image of a subject in a short time even when the subject has less surface scattering.SOLUTION: The method for simulating a captured image of a subject includes: a beam tracking calculation step of calculating a beam impinging on the subject, scattered in the subject, and then emitted from a surface of the subject so as to generate beam information including at least an angular distribution of the beam; an intensity estimation function setting step of estimating an intensity of the beam reaching an imaging system or an illumination system from the surface of the subject and an intensity estimation function on the basis of the beam information; an intensity distribution calculation step of calculating an intensity distribution of the subject on the basis of the beam information and the intensity estimation function; and a generation step of generating a captured image of the subject on the basis of the intensity distribution of the subject.SELECTED DRAWING: Figure 7

Description

[Technical field]

[0001] The present disclosure relates to a subject photographing image simulation method, an image generating device, and a program. [Background technology]

[0002] Conventionally, in the design of endoscopes, simulation of images captured by endoscopes has been attempted. In addition, the organs of the subject observed by the endoscope are volume scatterers, not just the surface, so in order to accurately predict the simulated captured images, it is necessary to calculate the light scattering inside the subject and reflect it in the simulated captured images. One method is known to calculate the light scattering inside the subject using the Monte Carlo method, which calculates the light scattering inside the subject by ray tracing, but due to the enormous calculation cost, it is difficult to use this method for image generation in the simulation of captured images.

[0003] On the other hand, a surface photon mapping technique is known as a means for improving the efficiency of image generation in the simulation of a captured image (see, for example, Patent Document 1). This technique discloses a method for generating an image of the skin, which is a volume scatterer, as shown in FIG. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 6092565 Summary of the Invention [Problem to be solved by the invention]

[0005] Incidentally, when the subject is skin as in the above-mentioned Patent Document 1, scattering is strong and the surface scattering characteristics are broad, so that a large number of high-brightness configurations can be obtained in the imaging system to make image calculations more efficient.

[0006] However, the surface of a subject such as an organ observed with an endoscope is often covered with a liquid such as a mucous membrane, and scattering at the surface is weak. When this surface scattering is weak and the luminance of light rays other than those in the angular direction immediately before emission is infinitesimally close to zero (when the surface scattering characteristic is a delta function), only light rays with low luminance reach the imaging system (see FIG. 22). For this reason, even when the technology of Patent Document 1 described above is used, there is a problem that a captured image of only light rays with low luminance has a lot of noise and is unreliable, and therefore a huge number of trial calculations are required to reduce the noise, which takes a huge amount of time.

[0007] The present disclosure has been made in consideration of the above, and aims to provide a subject capture image simulation method, image generation device, and program that are capable of accurately generating a low-noise, highly reliable simulation of a subject capture image in a short period of time, even in the case of a subject with weak surface scattering. [Means for solving the problem]

[0008] In order to solve the above-mentioned problems and achieve the object, the subject photographing image simulation method according to the present disclosure is a subject photographing image simulation method for simulating a photographed image in a model composed of an illumination system, an imaging system, and a subject which is a volume scatterer, and includes a ray tracing calculation step of calculating light rays which enter the subject, are scattered inside, and then exit from the surface of the subject to generate ray information including at least an angular distribution of the light rays, a luminance estimation function setting step of estimating the luminance and luminance estimation function of the light rays which reach the imaging system or the illumination system from the surface of the subject based on the ray information, a luminance distribution calculation step of calculating the luminance distribution of the subject based on the ray information and the luminance estimation function, and a generation step of generating a photographed image of the subject based on the luminance distribution of the subject.

[0009] In the subject photographing image simulation method according to the present disclosure, in the above disclosure, the brightness estimation function setting step sets a kernel function as the brightness estimation function.

[0010] In the subject photographing image simulation method according to the present disclosure, the kernel function is a rectangle function.

[0011] In the subject photographing image simulation method according to the present disclosure, the kernel function is a Gaussian function.

[0012] In addition, in the subject photographing image simulation method according to the present disclosure, in the above disclosure, the ray tracing calculation step irradiates the subject with light rays generated from the illumination system, calculates light rays that exit the subject after the light rays that entered the subject are scattered inside the subject, and generates the light ray information, and the brightness estimation function setting step estimates the brightness of the light rays that reach the imaging system from the surface of the subject.

[0013] Furthermore, in the subject photographing image simulation method according to the present disclosure, in the above disclosure, the ray tracing calculation step traces light rays that travel from the imaging system toward the subject, are scattered inside the subject, and then are emitted from the subject, the luminance estimation function setting step estimates the luminance of light rays that reach the imaging system from a surface of the subject, and the luminance distribution calculation step calculates a contribution to the imaging system from the luminance distribution of the illumination system obtained by tracing light rays that travel from the subject, are scattered inside the subject, and then are emitted from the subject, to the illumination system.

[0014] Furthermore, in the subject photographing image simulation method according to the present disclosure, in the above disclosure, the ray tracing calculation step determines a ratio between an entrance pupil diameter of an imaging lens provided in the imaging system, an exit pupil diameter of an illumination lens provided in the illumination system, and the number of pixels to be simulated out of all pixels of an image sensor provided in the imaging system, and if a value obtained by multiplying the ratio by the entrance pupil diameter is greater than the exit pupil diameter, a forward tracing process is executed to trace rays from the illumination system, while if a value obtained by multiplying the ratio by the entrance pupil diameter is equal to or smaller than the exit pupil diameter, a backward tracing process is executed to trace rays from the imaging system.

[0015] Moreover, an image generating device according to the present disclosure is an image generating device that simulates a captured image in a model composed of an illumination system, an imaging system, and a subject that is a volume scatterer, and includes a ray tracing calculation unit that calculates light rays that enter the subject, are scattered internally, and then exit from the surface of the subject to generate ray information including at least an angular distribution of the light rays, a luminance estimation function setting unit that estimates the luminance and luminance estimation function of the light rays that reach the imaging system or the illumination system from the surface of the subject based on the ray information, a luminance distribution calculation unit that calculates the luminance distribution of the subject based on the ray information and the luminance estimation function, and a generation unit that generates a captured image of the subject based on the luminance distribution of the subject.

[0016] Further, a program according to the present disclosure is a program executed by an image generating device that simulates a captured image in a model composed of an illumination system, an imaging system, and a subject that is a volume scatterer, and includes a ray tracing calculation step of calculating light rays that enter the subject, are scattered internally, and then exit from the surface of the subject to generate ray information including at least an angular distribution of the light rays, a luminance estimation function setting step of estimating the luminance and luminance estimation function of the light rays that reach the imaging system or the illumination system from the surface of the subject based on the ray information, a luminance distribution calculation step of calculating the luminance distribution of the subject based on the ray information and the luminance estimation function, and a generation step of generating a captured image of the subject based on the luminance distribution of the subject. Effect of the Invention

[0017] According to the present invention, even in the case of a subject with weak surface scattering, it is possible to accurately generate a simulation of a captured image of the subject with low noise and high reliability in a short time. [Brief description of the drawings]

[0018] [Figure 1] FIG. 1 is a schematic diagram showing an overview of an image generating system according to an embodiment. [Diagram 2]FIG. 2 is a diagram showing an example of the relationship between the structure and optical characteristics of a phantom, which is a subject, according to one embodiment. [Diagram 3] FIG. 3 is a diagram showing an example of conditions of an illumination system according to an embodiment. [Figure 4] FIG. 4 is a diagram showing an example of conditions of an imaging system according to an embodiment. [Diagram 5] FIG. 5 is a diagram showing a schematic overview of how an image generating system according to an embodiment determines a kernel function from light ray information. [Figure 6] FIG. 6 is a diagram illustrating an outline of a distributed configuration when an image generating system according to an embodiment applies a kernel function to a light ray. [Figure 7] FIG. 7 is a flowchart showing an overview of the process of a subject photographing image simulation method executed by an image generation system according to one embodiment. [Figure 8] FIG. 8 is a diagram illustrating a schematic diagram of calculation of scattering inside a subject by a ray tracing calculation unit according to an embodiment. [Figure 9] FIG. 9 is a diagram showing a part of the light beam information recorded by the light beam information recording unit according to one embodiment. [Figure 10] FIG. 10 is a diagram illustrating an example of the Kernel function K(t) according to an embodiment. [Figure 11] FIG. 11 is a diagram illustrating a calculation method in which a luminance estimation function setting unit according to an embodiment calculates the illuminance of one pixel in an imaging element of an imaging system from a light ray reaching an illumination system. [Figure 12] FIG. 12 is a diagram illustrating a calculation method when a luminance estimation function setting unit according to an embodiment calculates the illuminance of one pixel in an imaging element of an imaging system. [Figure 13] FIG. 13 is a diagram illustrating the relationship between the error of the illuminance bias and the Kernel function angle width according to one embodiment. [Figure 14] FIG. 14 is a flowchart showing an outline of forward tracing processing according to one embodiment. [Figure 15]FIG. 15 is a diagram illustrating an outline of forward tracking processing according to an embodiment. [Figure 16] FIG. 16 is a diagram illustrating an example of a captured image generated by a generating unit according to an embodiment. [Figure 17] FIG. 17 is a flowchart showing a schematic overview of the backward tracing process according to one embodiment. [Figure 18] FIG. 18 is a diagram illustrating an outline of the backward tracing process according to one embodiment. [Figure 19] FIG. 19 is a diagram showing a comparison result of line profiles in which illuminance at an imaging element is calculated from estimated luminance of each pixel using an example (present calculation method) according to one embodiment and a comparative example (conventional calculation method). [Figure 20] FIG. 20 is a diagram illustrating another example of the Kernel function according to the first modification of the embodiment. [Figure 21] FIG. 21 is a diagram showing a schematic diagram of an example of a case where the surface scattering of a subject is strong according to the conventional technology. [Figure 22] FIG. 22 is a diagram showing a schematic diagram of an example of a case where the surface scattering of a subject is weak according to the conventional technology. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0019] Hereinafter, the embodiments for carrying out the present disclosure will be described in detail with reference to the drawings. Note that the present disclosure is not limited to the following embodiments. Furthermore, each figure referred to in the following description merely shows the shape, size, and positional relationship in a schematic manner to the extent that the contents of the present disclosure can be understood. In other words, the present disclosure is not limited to only the shape, size, and positional relationship exemplified in each figure. Furthermore, in the following description, an image generation system including an image generation device capable of executing a subject shooting image simulation method will be described.

[0020] [Image Generation System Overview] Fig. 1 is a schematic diagram showing an overview of an image generating system according to an embodiment. The image generating system 1 shown in Fig. 1 simulates an image captured in a model consisting of an illumination system, an imaging system, and a subject that is a volume scatterer. The image generating system 1 includes a subject 2 that is a simulated living body (phantom) of a volume scatterer, an illumination system 3, an imaging system 4, and an image generating device 5.

[0021] The object 2 (phantom) is a single blood vessel, which is a perfect absorber, placed in tissue, which is a uniform, non-absorbing scatterer.

[0022] Fig. 2 is a diagram showing an example of the relationship between the structure and optical characteristics of a phantom, which is the subject 2. Table T1 shown in Fig. 2 lists the structure of tissues and blood vessels in the phantom, which is the subject 2, in association with the optical characteristics of each of the tissues and blood vessels in the phantom, which is the subject 2.

[0023] The illumination system 3 is disposed at a predetermined distance from the subject 2 and irradiates light towards the subject 2. The illumination system 3 has at least a light source that emits light, and an illumination lens that collects the light emitted by the light source and irradiates it towards the subject 2.

[0024] Fig. 3 is a diagram showing an example of the conditions of the illumination system 3. In table T2 shown in Fig. 3, various illumination conditions in the illumination system 3 are associated with the observation distance. Specifically, in table T2, the shape, size, light distribution, and observation distance of the illumination system 3 are associated with each other.

[0025] The imaging system 4 is disposed at a predetermined distance from the subject 2 and receives and captures light from the subject 2. The imaging system 4 has at least an imaging lens that collects light from the subject 2 to form a subject image, and an imaging element (a Complementary Metal Oxide Semiconductor (CMOS) sensor or a Charge Coupled Device (CCD) sensor) that receives the subject image formed by the imaging lens.

[0026] Fig. 4 is a diagram showing an example of conditions of the imaging system 4. In table T3 shown in Fig. 4, various imaging conditions in the imaging system 4 are associated with the observation distance. Specifically, in table T3, the size of the imaging element, the number of pixels, and the observation distance are associated with each other.

[0027] The image generating device 5 simulates a captured image. The image generating device 5 includes a communication unit 51, a display unit 52, an input unit 53, a recording unit 54, and a control unit 55.

[0028] The communication unit 51 outputs various data input from the control unit 55 to the illumination system 3 and the imaging system 4, and outputs subject photographic image data input from the imaging system 4 to the control unit 55. The communication unit 51 is configured using a communication interface, etc.

[0029] The display unit 52 displays various information related to the image generating system 1 and captured images of a subject under the control of the control unit 55. The display unit 52 is configured using an organic electroluminescent display, a liquid crystal display, or the like.

[0030] The input unit 53 receives various operation inputs from an operator, and outputs signals corresponding to the received operations to the control unit 55. The input unit 53 is configured using a keyboard, a mouse, a touch panel, and the like.

[0031] The recording unit 54 records various information related to the image generating system 1. The recording unit 54 is configured using a RAM (Random Access Memory), a ROM (Read Only Memory), an HDD (Hard Disk Drive), an SSD (Solid State Drive), etc. The recording unit 54 has a program recording unit 541 that records various programs executed by the image generating device 5, a light ray information recording unit 542 that records light ray information described later, and an error occurrence amount table data recording unit 543 that records error occurrence amount table data.

[0032] The control unit 55 controls each unit constituting the image generating system 1. The control unit 55 is configured using a processor having hardware such as a memory, a CPU (Central Processing Unit), and a GPU (Graphics Processing Unit). The control unit 55 has a ray tracing calculation unit 551, a luminance estimation function setting unit 552, a luminance distribution calculation unit 553, and a generation unit 554.

[0033] The ray tracing calculation unit 551 calculates light rays that enter the subject, are scattered inside, and then emerge from the subject surface to generate light ray information including at least the angular distribution of the light rays. Here, the angular distribution of the light rays includes the position of the emergent light rays emerging from the surface of the subject, which is a scattering body, the direction cosine of the emergent light rays, and the luminous flux of the emergent light rays.

[0034] The luminance estimation function setting unit 552 estimates the luminance and luminance estimation function of a ray reaching the imaging system 4 or the illumination system 3 from the surface of the subject, based on the ray information generated by the ray tracing calculation unit 551. Specifically, the luminance estimation function setting unit 552 estimates a kernel function as the luminance estimation function.

[0035] The luminance distribution calculation section 553 calculates the luminance distribution of the subject based on the light ray information generated by the ray tracing calculation section 551 and the luminance estimation function estimated by the luminance estimation function setting section 552 .

[0036] The generation section 554 generates a captured image of the subject based on the luminance distribution of the subject calculated by the luminance distribution calculation section 553 .

[0037] [Overview of processing by image generation system 1] Next, an outline of the subject photographing image simulation method executed by the image generation system 1 will be described. Fig. 5 is a diagram showing an outline of how the image generation system 1 determines a kernel function from light ray information. Fig. 6 is a diagram showing an outline of a distributed configuration when the image generation system 1 applies a kernel function to light rays.

[0038] 5, the image generating system 1 determines the kernel function K(t) based on light ray information (including at least the position of the exit ray R1, the direction of the exit ray R1, and the angular distribution of the exit ray R1 including the luminous flux of the exit ray R1) which is a light ray group D1 of a light ray R1 (hereinafter referred to as an "exit ray R1") that is emitted from the surface of the subject 2 after scattering inside the subject 2, rather than on the scattering characteristics of the surface of the subject 2. Specifically, as shown in FIG. 6, the exit ray R1 that is multiple scattered inside the subject 2 becomes a uniform light ray direction for each scattering, and becomes an exit ray R2 that is appropriately dispersed when emitted from the surface of the subject 2.

[0039] Therefore, the image generation system 1 disperses and emits the exit ray R1 using a kernel function K(t) determined based on the light ray information, which is a group of rays D1 of the exit ray R1 that is scattered inside the subject 2 and then emitted from the surface, and causes the exit ray R1 to reach the imaging system 4, thereby contributing a large amount of the exit ray R2 to the captured image, thereby achieving efficient calculations.

[0040] As a result, the image generating system 1 estimates the luminance of the light reaching the imaging system 4 or the illumination system 3 from the surface of the subject 2 using the kernel function K(t) calculated from the ray information of the outgoing ray R1 that is scattered inside the subject 2 and then emitted from the surface of the subject 2, rather than the scattering characteristics (BTDF) of the surface of the subject 2. As a result, even in the case of a subject 2 that is covered with a liquid surface such as the surface of the digestive tract and has very weak scattering characteristics on the surface, the light ray information is appropriately dispersed, so that more components enter the imaging system 4, and the function width of the kernel function K(t) as a luminance estimation function becomes a value suitable for estimation.

[0041] In a typical endoscope, when illuminating the subject 2, the light source is small, so the NA of the illumination light on the surface of the subject 2 is also small, and the angle of the incident light is localized. However, since the subject 2 is a volume scatterer, the direction of the light ray becomes uniform each time the light is scattered. Therefore, according to the image generating system 1, even if the direction of the incident light ray is very strong, the light ray information on the surface of the subject 2 after scattering at the subject 2 is appropriately dispersed by repeating scattering inside, so that the function width of the kernel function K(t) becomes a value suitable for estimation with a distribution. As a result, the brightness of the outgoing light ray R2 incident on the imaging system 4 is increased, and the calculation efficiency can be improved.

[0042] In this way, according to the image generation system 1, even if the surface scattering of the subject 2 is very weak, by performing image generation using the brightness estimation function setting method described above, it is possible to accurately and efficiently simulate a captured image of a living body where the surface of an organ or the like is covered in liquid and the scattering characteristics of the surface of the subject 2 are very weak.

[0043] [Processing Executed by the Image Generation System] Next, a description will be given of the process of the subject photographing image simulation method executed by the image generation system 1. Fig. 7 is a flowchart showing an overview of the process of the subject photographing image simulation method executed by the image generation system 1.

[0044] 7, first, the ray tracing calculation unit 551 determines the illumination system 3, the imaging system 4, and the subject model (step S101). Specifically, the ray tracing calculation unit 551 determines the illumination system 3, the imaging system 4, and the subject model in Tables T1 to T3 in the above-mentioned FIGS. 2 to 4.

[0045] Next, the ray tracing calculation unit 551 performs ray tracing from one pixel on the axis of the imaging element in the imaging system 4 toward the exit pupil direction (step S102), calculates scattering inside the subject 2 (step S103), and records the outgoing rays emitted from the surface of the scatterer that is the subject 2 as ray information in the ray information recording unit 542 (step S104). Specifically, the ray tracing calculation unit 551 records ray information including the angular distribution of the outgoing rays emitted from the surface of the scatterer that is the subject 2 in the ray information recording unit 542. Here, the angular distribution of the outgoing rays includes the position (x, y, z) of the outgoing rays emitted from the surface 21 of the subject 2 that is a scatterer, the direction cosines α, β, γ of the outgoing rays, and the luminous flux Φ of the outgoing rays.

[0046] 8 is a diagram showing a schematic diagram of calculation of scattering inside the subject 2 by the ray tracing calculation unit 551. As shown in FIG. 8, the ray tracing calculation unit 551 traces a ray R3 from one pixel on the axis of the imaging element in the imaging system 4 toward the exit pupil direction, calculates scattering inside the subject 2, and then generates position P1(x, y, z) of an outgoing ray R1 emitted from the surface 21 of the subject 2, which is a scattering body, direction cosines α, β, γ of the outgoing ray R1, and a light flux Φ of the outgoing ray R1 as ray information and records it in the ray information recording unit 542.

[0047] FIG. 9 is a diagram showing a part of the light ray information recorded by the light ray information recording unit 542. The light ray information T4 shown in FIG. 9 includes x, y, and z, which are the position P1 of the output light ray R1 on the surface 21 of the subject 2, the direction cosines α, β, and γ of the output light ray R1, and the light flux Φ of the output light ray R1. The number of light rays traced by the ray tracing calculation unit 551 is set to 10,000 rays (statistical error 1%) that reach the illumination system 3, which is the target (goal). For this reason, in one embodiment, the ray tracing calculation unit 551 traces 300,000 light rays from the imaging system 4. Furthermore, in one embodiment, the ray tracing by the ray tracing calculation unit 551 was performed using Ver. 8.5 of LightTools (registered trademark) (Synopsys, Inc. (registered trademark)).

[0048] After step S104, the brightness estimation function setting unit 552 calculates the illuminance of one pixel in the imaging element of the imaging system 4 based on the light rays reaching the illumination system 3 and the brightness of the illumination system 3, leaving the light ray information recorded by the light ray information recording unit 542 unchanged (step S105).

[0049] Next, the brightness estimation function setting unit 552 sets the kernel function to the initial value of the human organ scatterer recorded by the error generation amount table data recording unit 543 (step S106). Here, the method for determining the initial value of the kernel function angle width is to use a past calculation example of a human organ to convert the error generation amount for the kernel angle width into table data and record it in the error generation amount table data recording unit 543. That is, the brightness estimation function setting unit 552 refers to the error generation amount table data recorded by the error generation amount table data recording unit 543, and sets an appropriate function width for the target error amount as the initial value. This makes it possible to optimize the angle width within a range that does not deviate significantly from the target value.

[0050] Fig. 10 is a diagram showing an example of the kernel function K(t). As shown in Fig. 10, the shape of the kernel function K(t) uses a rectangular function and satisfies the following formula (1). In addition, since the kernel function K(t) is a rectangular function, the width Δ of the function is a variable to be set.

number

[0051] After step S106, the brightness estimation function setting unit 552 tracks the light rays reaching the illumination system 3 based on the light ray information recorded by the light ray information recording unit 542 without changing the light ray information, and calculates the illuminance of one pixel in the imaging element of the imaging system 4 from the light rays reaching the illumination system 3 (step S107).

[0052] Fig. 11 is a diagram illustrating a calculation method used by the luminance estimation function setting unit 552 to calculate the illuminance of one pixel in the image sensor of the imaging system 4 from a light ray reaching the illumination system 3. As shown in Fig. 11, the luminance estimation function setting unit 552 tracks a light ray R4 reaching the illumination system 3 while keeping the light ray information recorded by the light ray information recording unit 542 as it is, and calculates the illuminance of one pixel in the image sensor of the imaging system 4 from the light ray R4 reaching the illumination system 3.

[0053] After step S107, the brightness estimation function setting unit 552 assumes a certain kernel function, disperses each light ray according to the kernel function (step S108), and calculates the illuminance of one pixel in the imaging element of the imaging system 4 from the dispersed light rays reaching the illumination system 3 and the brightness of the illumination system 3 (step S109).

[0054] FIG. 12 is a diagram showing a schematic diagram of a calculation method when the brightness estimation function setting unit 552 calculates the illuminance of one pixel in the imaging element of the imaging system 4. As shown in FIG. 12, the brightness estimation function setting unit 552 assumes a certain kernel function K(t) and calculates the illuminance of one pixel in the imaging element of the imaging system 4 based on the outgoing light R2 (dispersed light) obtained by dispersing each light ray according to the kernel function K(t) and the luminance of the light ray reaching the illumination system 3. In this embodiment, for the outgoing light R2 (dispersed light), 1000 light rays are randomly generated within the range of the kernel function width, and the energy of one light ray is set to 1 / 1000. From the viewpoint of energy conservation, the number of light rays generated and the rate of reduction of the energy need to have an inverse relationship.

[0055] After step S109, the luminance estimation function setting unit 552 calculates the error (hereinafter, referred to as "illuminance bias error") between the illuminance of one pixel in the imaging element of the imaging system 4 calculated in the above-mentioned step S107 and the illuminance of one pixel in the imaging element of the imaging system 4 calculated in the above-mentioned step S109 (step S110). In this case, the luminance estimation function setting unit 552 can efficiently acquire high-luminance light rays while suppressing the illuminance bias error, even if the light rays are dispersed in a certain angle range by applying the kernel function K(t) in step S108, because each light ray is scattered multiple times inside and the emission angle becomes nearly uniform.

[0056] Next, the luminance estimation function setting unit 552 judges whether the error of the illuminance Bias is equal to or less than a preset tolerance (step S111). Specifically, the luminance estimation function setting unit 552 judges whether the error of the illuminance Bias is equal to or less than a preset tolerance of ±5%. This tolerance can be appropriately set by the user. For example, when comparing with the error Sim of the illuminance Bias, the user sets a target that can suppress the preset tolerance within the error amount of the measuring device actually used. When the luminance estimation function setting unit 552 judges that the error of the illuminance Bias is equal to or less than a preset tolerance (step S111: Yes), it determines the kernel function K(t) to be a final value (hereinafter, referred to as "kernel function K(θ)") (step S112). After step S112, the image generating system 1 proceeds to step S115 described later.

[0057] In step S111, if the brightness estimation function setting unit 552 determines that the error in the illuminance Bias exceeds a preset allowable amount (step S111: No), the image generation system 1 proceeds to step S112.

[0058] Next, the brightness estimation function setting unit 552 refers to the error generation amount table data recorded by the error generation amount table data recording unit 543, determines the amount of change in the kernel function angle width from the current error amount of the illuminance Bias (step S113), and changes and sets the kernel function (step S114). After step S114, the image generating system 1 returns to the above-mentioned step S107, and repeats steps S107 to S109 until the error of the illuminance Bias becomes equal to or less than the preset allowable amount.

[0059] Fig. 13 is a diagram showing the relationship between the error of the illuminance bias and the angular width of the kernel function. In Fig. 13, the vertical axis represents the error of the illuminance bias, and the horizontal axis represents the angular width of the kernel function [deg]. In Fig. 13, the broken line L1 represents the relationship between the error of the illuminance bias and the angular width of the kernel function.

[0060] As shown by the broken line L1 in FIG. 13, the luminance estimation function setting unit 552 sets the initial value of the kernel function angle width to 63 [deg], and reduces the angle width by 9 [deg] and performs recalculation every time an error in the illuminance bias that is greater than the allowable amount occurs. As shown by the broken line L1 in FIG. 13, the luminance estimation function setting unit 552 adopts the angle width of 45 [deg] as the kernel function angle width because the error in the illuminance bias reaches 5% of the allowable amount. In this way, the luminance estimation function setting unit 552 suppresses the kernel function width to be smaller, disperses the light beam according to the re-set kernel function, calculates the illuminance, evaluates the error in the illuminance bias again, and compares it with the allowable amount. Then, the luminance estimation function setting unit 552 finally determines the kernel function width at which the error in the illuminance bias falls within the allowable amount.

[0061] After step S111, the ray tracing calculation unit 551 calculates the entrance pupil diameter P in and the exit pupil diameter P of the illumination lens of the illumination system 3. out The ratio to is calculated (step S115).

[0062] Next, the ray tracing calculation unit 551 determines the ratio α of the number of pixels to be simulated among all the pixels of the image sensor in the image pickup system 4 (0<α≦1) (step S116).

[0063] Then, the ray tracing calculation unit 551 calculates the entrance pupil diameter P in Multiplied by (α×P in ) is the exit pupil diameter P of the illumination lens of the illumination system 3 out The ray tracing calculation unit 551 determines whether the ratio α is greater than the entrance pupil diameter P in Multiplied by (α×P in ) is the exit pupil diameter P of the illumination lens of the illumination system 3 out If it is determined that the ratio is larger than the incident pupil diameter P in Multiplied by (α×P in ) is the exit pupil diameter P of the illumination lens of the illumination system 3 out If it is determined that the value is not larger (step S117: No), the image generation system 1 proceeds to step S119, which will be described later.

[0064] In step S118, a forward tracing process is executed, which starts ray tracing from the illumination system 3. Details of the forward tracing process will be described later. After step S118, the image generation system 1 ends this process.

[0065] In step S119, a backward tracing process is executed to start tracing a light ray from the imaging system 4 to the illumination system 3. Details of the backward tracing process will be described later. After step S119, the image generating system 1 ends this process.

[0066] [Details of forward tracking process] Next, the forward tracking process described in step S118 above will be described in detail. Fig. 14 is a flowchart showing an outline of the forward tracking process. Fig. 15 is a diagram showing a schematic outline of the forward tracking process. Here, the forward tracking process is a process of a technique for tracing rays from the illumination system 3.

[0067] As shown in FIG. 14, first, the luminance estimation function setting unit 552 traces rays from the illumination system 3 based on the light ray information recorded by the light ray information recording unit 542 (step S201), and calculates internal scattering (step S202).

[0068] Next, the brightness estimation function setting unit 552 disperses the light beam according to the kernel function K(θ) determined in the above-mentioned step S112 when the light beam is scattered inside the living body of the subject 2, which is a scatterer, and then emitted from the surface of the living body of the subject 2 (step S203). Specifically, as shown in Fig. 15, the brightness estimation function setting unit 552 disperses the light beam, which is scattered inside the living body of the subject 2, which is a scatterer, and then emitted from the surface of the subject 2, by the kernel function K(θ). Note that the method of dispersing the light beam is similar to that used when setting the kernel function in the above-mentioned step S108 (see Fig. 12), and therefore detailed description thereof will be omitted.

[0069] Thereafter, the luminance distribution calculation unit 553 tracks the light rays dispersed by the luminance estimation function setting unit 552 to the imaging system 4 using the kernel function K(θ), and calculates the illuminance of each pixel of the imaging element of the imaging system 4 (step S204). Specifically, the luminance distribution calculation unit 553 tracks the dispersed light rays to the imaging system 4, and calculates the illuminance of each pixel of the imaging element of the imaging system 4, thereby calculating the luminance distribution of the subject 2. As a result, for example, in an endoscope, the light rays are imaged onto the imaging element by the imaging lens, and the luminance distribution of the illuminance (illuminance value) of each pixel can be simulated.

[0070] Next, the generation unit 554 generates a captured image of the subject 2 based on the light rays that reach the image sensor of the imaging system 4 (step S205). Specifically, the generation unit 554 generates a captured image of the subject 2 based on the light rays that reach the image sensor of the imaging system 4 corresponding to the luminance distribution of the subject calculated by the luminance distribution calculation unit 553. After step S205, the image generation system 1 returns to the main routine in FIG. 7 and ends this process.

[0071] FIG. 16 is a diagram showing an example of a captured image generated by the generating unit 554. As shown in FIG. 16, the generating unit 554 generates a captured image i1 that is a simulation of a captured image of a subject based on light rays that reach an imaging element of the imaging system 4 corresponding to the luminance distribution of the subject calculated by the luminance distribution calculating unit 553. Note that the captured image i1 in FIG. 16 shows an example in which an imaging element having a pixel count of 700×500 (Pixel) is used. Furthermore, in one embodiment, the control unit 55 having a 12-core CPU was able to generate the captured image i1 with a peak statistical error of 3% in about 8 hours. In contrast, the Monte Carlo calculation of the conventional technology required about 2000 days for the same calculation, and about 6,000 times efficiency was achieved.

[0072] [Details of reverse tracking process] Next, the reverse tracking process described in step S119 above will be described in detail. Fig. 17 is a flow chart that shows an outline of the reverse tracking process. Fig. 18 is a diagram that shows an outline of the reverse tracking process. Here, the reverse tracking process is a process of a technique of sending a light ray from the imaging system 4 side and tracing it to the illumination system 3. That is, the reverse tracking process directly utilizes information on the path that the light ray passed through, and compares the position and angle when the light ray reaches the illumination system 3 with the light distribution characteristics of the illumination system 3 to calculate the illuminance of one pixel in the imaging element of the imaging system 4.

[0073] As shown in FIG. 17, first, the brightness estimation function setting unit 552 traces rays from each pixel in the imaging element of the imaging system 4 based on the light ray information recorded by the light ray information recording unit 542 (step S301), and calculates internal scattering (step S302).

[0074] Next, the luminance estimation function setting unit 552 disperses the light beam according to the kernel function K(θ) set in the above-mentioned step S112 when the light beam is scattered inside the living body of the subject 2, which is a scatterer, and then emitted from the surface of the subject 2, which is a scatterer (step S303). Specifically, as shown in Fig. 18, the luminance estimation function setting unit 552 disperses the light beam, which is scattered inside the living body of the subject 2, which is a scatterer, and then emitted from the surface of the subject 2, using the kernel function K(θ). Note that the method of dispersing the light beam is the same as when setting the kernel function in the above-mentioned step S108 (see Fig. 12), and therefore detailed description thereof will be omitted.

[0075] Thereafter, the luminance distribution calculation unit 553 sends out a light ray from the imaging system 4 side, tracks it to the illumination system 3, and uses information on the path the light ray passed through as it is to compare the position and angle when the light ray reaches the illumination system 3 with the light distribution characteristics of the illumination system 3 to inversely calculate the illuminance of the imaging system 4 (step S304). Specifically, the luminance distribution calculation unit 553 sends out a light ray from the imaging system 4 side, tracks it to the illumination system 3, and compares the position and angle when the light ray reaches the illumination system 3 with the light distribution characteristics of the illumination system 3 to inversely calculate the illuminance of the imaging system 4, thereby calculating the luminance distribution of the subject 2. In this case, since the luminance distribution calculation unit 553 takes into account the light distribution and in-plane distribution of the illumination system 3, it is preferable that the luminance distribution calculation unit 553 previously grasps information on the incident position and incident angle of the light ray to the illumination system 3.

[0076] Next, the generation unit 554 generates an image based on the light rays that reach the image sensor of the imaging system 4 (step S305). Specifically, the generation unit 554 generates a captured image of the subject 2 based on the light rays that reach the image sensor of the imaging system 4 corresponding to the luminance distribution of the subject calculated by the luminance distribution calculation unit 553. After step S305, the image generation system 1 returns to the main routine in FIG. 7 and ends this process.

[0077] In this way, with the reverse tracking process, when the area and NA of the receiver (imaging lens and image sensor) are extremely small, as in an endoscope, a light source that is relatively larger than the receiver becomes the target (goal) of ray tracing, making it easier to capture the light rays that contribute to the captured image.

[0078] Furthermore, the backward tracking process makes it possible to selectively calculate the illuminance received by a specific pixel of the imaging element in the imaging system 4, thereby shortening the calculation time according to the required number of pixels.

[0079] (Example) Next, the above-mentioned effects will be described more specifically with reference to an embodiment and a comparative example. Fig. 19 is a diagram showing a comparison result of a line profile in which the illuminance at an image sensor is calculated from the estimated luminance of each pixel using an embodiment (present calculation method) and a comparative example (conventional calculation method). Note that Fig. 19 describes a case in which an image sensor with 200 pixels per line is used. Furthermore, in Fig. 19, curve L11 indicates the embodiment (present calculation method), and curve L12 indicates the comparative example (conventional calculation method). Note that Fig. 19 shows the result of calculation for about 20 minutes when the number of CPU cores in the control unit 55 is 12.

[0080] 19, the comparative example (conventional method) has a peak statistical error of about 9.4% (see curve L12), whereas the embodiment has a peak statistical error of about 0.5% (see curve L11), achieving an efficiency about 350 times higher than the comparative example.

[0081] According to the embodiment described above, even in the case of a subject 2 with weak surface scattering, a large amount of high-intensity light rays can be acquired without only low-intensity light rays reaching the imaging system 4, and a simulation of a low-noise and highly reliable captured image i1 of the subject 2 can be accurately generated in a short time.

[0082] (Variation 1) In addition, in the embodiment, the kernel function has a rectangular shape, but is not limited to this, and various shapes can be applied. Fig. 20 is a diagram showing another example of the kernel function according to the first modification of the embodiment. As shown in Fig. 20, even a Gaussian function can be applied as the angle function. The Gaussian function in Fig. 20 is for σ=30 deg.

[0083] (Other embodiments) Various inventions can be formed by appropriately combining multiple components disclosed in the image generation system according to the embodiment of the present disclosure described above. For example, some components may be deleted from all the components described in the image generation system according to the embodiment of the present disclosure described above. Furthermore, the components described in the image generation system according to the embodiment of the present disclosure described above may be appropriately combined.

[0084] In addition, in the image generating system according to an embodiment of the present disclosure, the above-mentioned "unit" can be read as "means" or "circuit," etc. For example, the control unit can be read as control means or a control circuit.

[0085] In addition, the program to be executed by the image generating system according to one embodiment of the present disclosure is provided as file data in an installable or executable format recorded on a computer-readable recording medium such as a CD-ROM, a flexible disk (FD), a CD-R, a DVD (Digital Versatile Disk), a USB medium, or a flash memory.

[0086] In addition, the program executed by the image generating system according to one embodiment of the present disclosure may be configured to be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network.

[0087] In the description of the flowcharts in this specification, the order of processing between steps is clearly indicated using expressions such as "first," "then," and "continue," but the order of processing required to implement the present invention is not uniquely determined by these expressions. In other words, the order of processing in the flowcharts described in this specification can be changed to the extent that there is no contradiction. In addition, programs are not limited to those consisting of simple branching processing, and branching may be performed by comprehensively determining more judgment items.

[0088] Although some of the embodiments of the present application have been described in detail above with reference to the drawings, these are merely examples, and the present invention can be embodied in other forms that incorporate various modifications and improvements based on the knowledge of those skilled in the art, including the forms described in the disclosure of the present invention. [Explanation of symbols]

[0089] 1. Image Generation System 2. Subject 3. Lighting system 4. Imaging system 5. Image generation device 51 Communications Department 52 Display section 53 Input section 54 Recording Section 55 Control section 56: Luminance estimation function setting section 541 Program Recording Section 542 Light Information Recording Unit 543 Difference generation amount table data recording section 551 Ray tracing calculation unit 552 Luminance estimation function setting unit 553 Luminance distribution calculation unit 554 Generation part

Claims

1. A subject photographic image simulation method for simulating a photographic image in a model including an illumination system, an imaging system, and a subject that is a volume scatterer, comprising: a ray tracing calculation step of calculating light rays that enter the object, are scattered inside the object, and then exit from the object surface to generate light ray information including at least an angular distribution of the light rays; a brightness estimation function setting step of estimating a brightness and a brightness estimation function of a light ray reaching the imaging system or the illumination system from the surface of the object based on the light ray information; a luminance distribution calculation step of calculating a luminance distribution of the subject based on the light ray information and the luminance estimation function; a generation step of generating a captured image of the subject based on a luminance distribution of the subject; Including, A method for simulating photographed images of a subject.

2. 2. A subject photographing image simulation method according to claim 1, The brightness estimation function setting step includes: A kernel function is set as the brightness estimation function. A method for simulating photographed images of a subject.

3. 3. A subject photographing image simulation method according to claim 2, The kernel function is The rectangular function, A method for simulating photographed images of a subject.

4. 3. A subject photographing image simulation method according to claim 2, The kernel function is The Gaussian function, A method for simulating photographed images of a subject.

5. A subject photographing image simulation method according to any one of claims 1 to 4, The ray tracing calculation step includes: irradiating the object with light rays generated from the illumination system, calculating light rays emitted from the object after the light rays incident on the object are scattered inside the object, and generating the light ray information; The brightness estimation function setting step includes: Estimating the brightness of light reaching the imaging system from the object surface; A method for simulating photographed images of a subject.

6. A subject photographing image simulation method according to any one of claims 1 to 4, The ray tracing calculation step includes: tracing light rays from the imaging system toward the object, scattering within the object, and then exiting the object; The brightness estimation function setting step includes: Estimating the brightness of a light ray reaching the imaging system from a surface of the object; the luminance distribution calculation step calculates a contribution to the imaging system from a luminance distribution of the illumination system obtained by tracing a light ray emitted from the object after being scattered inside the object to the illumination system; A method for simulating photographed images of a subject.

7. 7. A subject photographing image simulation method according to claim 5, further comprising the steps of: The ray tracing calculation step includes: determining an entrance pupil diameter of an imaging lens included in the imaging system, an exit pupil diameter of an illumination lens included in the illumination system, and a ratio of the number of pixels to be simulated among all pixels of an imaging element included in the imaging system; if the ratio multiplied by the entrance pupil diameter is greater than the exit pupil diameter, a forward tracing process is performed to trace rays from the illumination system; if a value obtained by multiplying the ratio by the entrance pupil diameter is equal to or smaller than the exit pupil diameter, a backward tracing process is executed to perform ray tracing from the imaging system. A method for simulating photographed images of a subject.

8. An image generating device that simulates an image captured in a model that includes an illumination system, an imaging system, and a subject that is a volume scatterer, comprising: a ray tracing calculation unit that calculates light rays that enter the object, are scattered inside the object, and then exit from the object surface to generate light ray information including at least an angular distribution of the light rays; a luminance estimation function setting unit that estimates a luminance and a luminance estimation function of a light ray reaching the imaging system or the illumination system from the surface of the object based on the light ray information; a luminance distribution calculation unit that calculates a luminance distribution of the subject based on the light ray information and the luminance estimation function; a generation unit that generates a captured image of the subject based on a luminance distribution of the subject; Equipped with Image generating device.

9. A program executed by an image generating device that simulates a captured image in a model including an illumination system, an imaging system, and a subject that is a volume scatterer, the program comprising: a ray tracing calculation step of calculating light rays that enter the object, are scattered inside the object, and then exit from the object surface to generate light ray information including at least an angular distribution of the light rays; a brightness estimation function setting step of estimating a brightness and a brightness estimation function of a light ray reaching the imaging system or the illumination system from the surface of the object based on the light ray information; a luminance distribution calculation step of calculating a luminance distribution of the subject based on the light ray information and the luminance estimation function; a generating step of generating a captured image of the subject based on a luminance distribution of the subject; Including, program.

Images