All-in-focus image generation method, all-in-focus image generation apparatus, and all-in-focus image generation program

Abstract

To provide an all-in-focus image generation method, an all-in-focus image generation apparatus, and an all-in-focus image generation program capable of generating an all-in-focus image simply and accurately.SOLUTION: A method MT1 includes: an original image acquisition step (Step ST1) of acquiring an original image for each focal position related to a target object; a point spread information calculation step (Step ST4) of calculating point spread information regarding the original image for each focal position; a focal position determination step (Step ST3) of determining focal positions of pixels based on the original image for each focal position; a focal position correction step (Step ST5) of correcting the focal positions of the pixels based on the point spread information; and an all-in-focus image generation step (Step ST6) of extracting pixels corresponding to the corrected focal positions of the pixels and generating an all-in-focus image by integrating the extracted pixels.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present disclosure relates to an omnifocus image generating method, an omnifocus image generating device, and an omnifocus image generating program. [Background technology]

[0002] A method is known in which the same object is photographed in the thickness direction (Z-axis direction) of the object while shifting the focal position to obtain multiple images (Z-stack images) at different focal positions, and an all-in-focus image in which each pixel is in focus is generated from the Z-stack images (for example, Patent Document 1).

[0003] In the method of Patent Document 1, an all-in-focus image is generated by combining pixels (pixels indicating the focal position) of images with the highest degree of focus at each pixel position from acquired Z-stack images. Specifically, a focus degree calculation unit calculates the degree of focus at all pixel positions of two-dimensional images constituting the Z-stack image and sends the calculated values ​​to an all-in-focus image generation unit. The all-in-focus image generation unit selects the pixel value of any two-dimensional image in the Z direction that has the highest degree of focus (extracts the focal position) and generates the all-in-focus image by combining the selected pixel values ​​at all pixel positions. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2018 / 042629 Summary of the Invention [Problem to be solved by the invention]

[0005] In the method of Patent Document 1, the focus level calculation unit uses a Laplacian pyramid that can represent high-frequency components such as edge information to calculate the focus level based on the high-frequency components. The high-frequency components may include noise components in addition to focus components.

[0006] To calculate the focus degree with high accuracy and generate an all-in-focus image with high accuracy, it is effective to separate the in-focus component from the noise component of the acquired Z-stack image or all-in-focus image, for example, as in the method of Patent Document 1. For example, it is expected that noise filtering will be performed on the Z-stack image or all-in-focus image, but manually adjusting the filter parameters depending on the imaging target is very cumbersome.

[0007] The present disclosure provides an all-in-focus image generating method, an all-in-focus image generating device, and an all-in-focus image generating program that can generate an all-in-focus image simply and accurately. [Means for solving the problem]

[0008] The gist of the present disclosure is as follows. [1] An all-in-focus image generation method comprising: an original image acquisition step of acquiring original images for each focus position of an object obtained by capturing images of the object while shifting the focus position in a predetermined direction; a point spread information calculation step of calculating point spread information for the original images for each focus position; a focus position determination step of determining a focus position for each pixel based on the original images for each focus position; a focus position correction step of correcting the focus position for each pixel based on the point spread information; and an all-in-focus image generation step of extracting pixels corresponding to the corrected focus position for each pixel from the original images for each focus position and integrating the extracted pixels to generate an all-in-focus image.

[0009] This all-in-focus image generation method uses point spread information to suppress high-frequency components that are not present in the original image for each focus position, thereby reducing noise components. As a result, while retaining the focus position information for each pixel, the pixel values ​​of each pixel are smoothed to reduce noise components, allowing for accurate calculation of the focus position. Furthermore, by using point spread information for the original image as a correction parameter, there is no need to manually adjust the correction parameters to match the imaging target. An all-in-focus image can be easily generated by extracting pixels corresponding to the corrected focus position from the original image for each focus position and integrating the extracted pixels.

[0010] [2] The all-in-focus image generating method according to [1], wherein the point spread information includes a point spread function. In this case, the point spread function is a function in the spatial domain. Since the focal position of each pixel is data in image space, it is easy to directly apply the point spread function. This allows for efficient generation of an all-in-focus image.

[0011] [3] The all-in-focus image generating method according to [2], wherein the point spread information calculation step includes a step of frequency-transforming the original image for each focus position to obtain an amplitude spectrum, and a step of inverse-frequency-transforming the amplitude spectrum to calculate the point spread function. In the amplitude spectrum, frequency components change with increasing distance from the focus position. By performing an inverse frequency transform on this amplitude spectrum, it is possible to simply calculate the distribution of frequency components in a spatial domain based on the focus position.

[0012] [4] The all-in-focus image generating method according to [1], wherein the point spread information includes an optical transfer function. In this case, the optical transfer function is a function in the frequency domain, so it is easy to apply to data processing in the frequency domain and does not require inverse frequency transformation. This improves the flexibility of data processing.

[0013] [5] The all-in-focus image generating method according to any one of [1] to [4], wherein the point spread information calculation step calculates the point spread information based on at least one image among the original images for each focus position. In this case, the point spread information can be calculated efficiently with a small amount of input data.

[0014] [6] The all-in-focus image generating method according to any one of [1] to [5], wherein the point spread information calculation step calculates the point spread information based on the original image for each focal position or a feature amount in the original image for each focal position. In this case, the options for input data for calculating the point spread information are expanded, thereby improving the degree of freedom in data processing.

[0015] [7] The all-in-focus image generating method according to any one of [1] to [6], wherein the point spread information is calculated based on optical parameters of an imaging device that captures the object in the point spread information calculation step. In this case, the point spread information can be calculated without being affected by image generation accuracy.

[0016] [8] An all-in-focus image generating device comprising: an original image acquiring unit that acquires original images for each focus position of an object obtained by capturing images of the object while shifting the focus position in a predetermined direction; a point spread information calculating unit that calculates point spread information for the original images for each focus position; a focus position determining unit that determines the focus position for each pixel based on the original images for each focus position; a focus position correcting unit that corrects the focus position for each pixel based on the point spread information; and an all-in-focus image generating unit that extracts pixels corresponding to the corrected focus position for each pixel from the original images for each focus position and integrates the extracted pixels to generate an all-in-focus image.

[0017] This omnifocus image generating device uses point spread information to suppress high-frequency components not present in the original image for each focus position, thereby reducing noise components. As a result, while retaining the focus position information for each pixel, the pixel values ​​of each pixel are smoothed to reduce noise components, allowing for accurate calculation of the focus position. Furthermore, by using point spread information for the original image as a correction parameter, there is no need to manually adjust the correction parameters to match the imaging target. An omnifocus image can be easily generated by extracting pixels corresponding to the corrected focus position from the original image for each focus position and integrating the extracted pixels.

[0018] [9] An all-in-focus image generation program that causes a computer to execute the following steps: an original image acquisition step that acquires original images for each focus position of an object obtained by capturing images of the object while shifting the focus position in a predetermined direction; a point spread information calculation step that calculates point spread information for the original images for each focus position; a focus position determination step that determines a focus position for each pixel based on the original images for each focus position; a focus position correction step that corrects the focus position for each pixel based on the point spread information; and an all-in-focus image generation step that extracts pixels corresponding to the corrected focus position for each pixel from the original images for each focus position and integrates the extracted pixels to generate an all-in-focus image.

[0019] This all-in-focus image generation program uses point spread information to suppress high-frequency components that are not present in the original image for each focus position, thereby reducing noise components. As a result, while retaining the focus position information for each pixel, the pixel values ​​of each pixel are smoothed to reduce noise components, allowing for accurate calculation of the focus position. Furthermore, by using point spread information for the original image as a correction parameter, there is no need to manually adjust the correction parameters to match the imaging target. An all-in-focus image can be easily generated by extracting pixels corresponding to the corrected focus position from the original image for each focus position and integrating the extracted pixels. [Effects of the Invention]

[0020] According to the present disclosure, an omnifocal image can be generated easily and accurately. [Brief explanation of the drawings]

[0021] [Figure 1] FIG. 1 is a block diagram illustrating an all-in-focus image generating device according to an embodiment. [Figure 2] 2A to 2C are diagrams showing examples of original images for each focal position acquired by an original image acquisition unit shown in FIG. 1. [Figure 3] 2 is a diagram showing an example of image data generated by the all-in-focus image device shown in FIG. 1; FIG. [Figure 4]1 is a flowchart illustrating an all-in-focus image generating method according to an embodiment. [Figure 5] FIG. 10 is a diagram illustrating an example of frequency decomposition. [Figure 6] 5 is a flowchart showing an example of a point spread information calculation step shown in FIG. 4. [Figure 7] 7A to 7C are diagrams showing an example of a method for calculating a point spread function corresponding to each step in FIG. 6. [Figure 8] Fig. 8(a) is a diagram showing another example of input data for calculating a point spread function, and Fig. 8(b) is a diagram showing yet another example of input data for calculating a point spread function. [Figure 9] FIG. 10 is a diagram illustrating an example of focus position correction. [Figure 10] FIG. 10 is a diagram for explaining a focus position correcting step according to a first modified example. [Figure 11] FIG. 10 is a diagram for explaining a step of excluding original images that do not include a focus position. [Figure 12] FIG. 10 is a diagram for explaining a step of selecting a specific region. [Figure 13] 10A and 10B are diagrams for explaining steps of converting an all-in-focus image into an image focused on a specific region. [Figure 14] FIG. 13 is a diagram showing an example of application of an image information determination model according to a sixth modified example. [Figure 15] FIG. 10 is a block diagram showing an example of an omnifocus image generating program and a recording medium therefor. [Figure 16] FIG. 1 is a diagram showing the results of calculating evaluation indexes in Example 1 and Comparative Example 1. [Figure 17] FIG. 1 is a diagram showing an evaluation sample. [Figure 18] FIG. 10 is a diagram showing the results of calculating evaluation indexes in Example 2, Example 3, Comparative Example 2, and Comparative Example 3. DETAILED DESCRIPTION OF THE INVENTION

[0022] Hereinafter, preferred embodiments of an omnifocus image generating method, an omnifocus image generating device, and an omnifocus image generating program according to an embodiment of the present disclosure will be described in detail with reference to the drawings.

[0023] [Omnifocus image generation device] FIG. 1 is a block diagram showing an all-in-focus image generating device according to an embodiment. The all-in-focus image generating device 1 generates an all-in-focus image in which each pixel is in focus from an original image (a plurality of original images) acquired for each focal position. The all-in-focus image generating device 1 is configured as a device that acquires original images for each focal position by capturing images of a cell sample placed on a glass slide, for example, and generates an all-in-focus image of the cell sample. The generated all-in-focus image is used, for example, for cytological diagnosis and pathological diagnosis.

[0024] The all-in-focus image generating device 1 includes an original image acquiring unit 11, a feature extracting unit 12, a focal position determining unit 13, a point spread information calculating unit 14, a focal position correcting unit 15, an all-in-focus image generating unit 16, and an all-in-focus image output unit 17. Physically, the all-in-focus image generating device 1 is a computer including a processor such as a CPU, and storage media such as RAM and ROM. The computer may be a smart device such as a smartphone or tablet terminal that is integrated with a display unit and an input unit. The computer may be configured as a microcomputer or a field-programmable gate array (FPGA). The computer may be configured externally via a network.

[0025] The original image acquisition unit 11 acquires original images 2 for each focal position of an object by capturing images of the object while shifting (moving) the focal position in a predetermined direction. The original image acquisition unit 11 is connected to, for example, an imaging device 10. The imaging device 10 is, for example, a microscope imaging device, and captures an enlarged image of an object placed on a stage. The original image acquisition unit 11 acquires original images for each focal position from the imaging device 10. Note that the original image acquisition unit 11 may include an imaging device, and the original image acquisition unit 11 may capture and acquire original images 2 for each focal position.

[0026] FIG. 2 is a diagram showing an example of an original image for each focal position acquired by the original image acquisition unit 11. The original image acquisition unit 11 acquires an original image 2 of a thick object. Here, for convenience of explanation, the thickness direction of the object is defined as the Z-axis direction. The direction perpendicular to the Z-axis direction is defined as the X-axis direction, and the direction perpendicular to the Z-axis direction and the X-axis direction is defined as the Y-axis direction. The imaging device 10 fixes the position (position in the X-axis direction and the Y-axis direction) of the same object during imaging other than the Z-axis direction, and images the object multiple times in succession while shifting the focal position in the Z-axis direction (a predetermined direction). In this way, the original image acquisition unit 11 acquires original images 2 for each focal position having a different focal position from each other. The original images 2 for each focal position are called Z-stack images because they are continuous in the Z-axis direction.

[0027] The feature extraction unit 12 extracts features in the original image 2 for each focal position. The original image 2 for each focal position acquired by the original image acquisition unit 11 is input to the feature extraction unit 12. As shown in FIG. 3, the feature extraction unit 12 generates a feature image 3 for each focal position in which the features are reflected. The feature images 3 for each focal position are continuous in the Z-axis direction, and the same number of feature images 3 are generated as the original images 2 for each focal position. The feature is, for example, a frequency component. Each original image 2 for each focal position contains a frequency component that indicates the strength of the edge at the focal position. The feature extraction unit 12 extracts, for example, high-frequency components from the frequency components contained in the original image 2. The higher the frequency of the frequency component, the stronger the edge strength and the closer it is to the focal position.

[0028] The focus position determination unit 13 determines the focus position for each pixel based on the feature extracted by the feature extraction unit 12. A feature image 3 for each focus position is input to the focus position determination unit 13. The focus position determination unit 13, for example, integrates the feature image 3 for each focus position in the Z-axis direction to generate a focus position image 4. In the example of FIG. 3, the focus position image 4 is a single image. The focus position image 4 is an image that indicates the focus position for each pixel. In other words, the focus position image 4 is an image in which each pixel is in focus.

[0029] The point spread information calculation unit 14 calculates point spread information P1. For example, the original image 2 for each focal position acquired by the original image acquisition unit 11 is input to the point spread information calculation unit 14. The point spread information P1 is a representation of the frequency components of the original image 2 for each focal position in the spatial domain or frequency domain, in other words, information on the frequency distribution in the original image 2 for each focal position.

[0030] The focus position correction unit 15 corrects the focus position for each pixel based on the point spread information P1. The focus position correction unit 15 receives the point spread information P1 calculated by the point spread information calculation unit 14 and the focus position image 4 generated by the focus position determination unit 13. The focus position correction unit 15 generates a focus position corrected image 5 by correcting the focus position image 4. The focus position correction unit 15 reduces noise components by using the point spread information P1 to suppress high-frequency components that do not exist in the original image 2 for each focus position. When the focus position correction unit 15 performs a convolution operation on the focus position image using the point spread information P1 generated from the original image 2 for each focus position, it can remove high-frequency components that do not exist in the original image 2 for each focus position. However, simply performing a convolution operation may result in excessive removal of high-frequency components. Therefore, by performing a convolution operation using edge-preserving smoothing (e.g., a bilateral filter) as described later in FIG. 9, it is possible to suppress high-frequency components that are not present in the original image 2 for each focus position while maintaining the high-frequency components of the focus position image 4. As a result, a focus-position-corrected image 5 is generated in which the pixel values ​​of each pixel are smoothed and noise components are reduced while retaining focus position information for each pixel. This improves the accuracy of the focus position of each pixel in the focus-position-corrected image 5 compared to the accuracy of the focus position of each pixel in the focus position image 4.

[0031] The all-focus image generation unit 16 generates an all-focus image 6. The all-focus image generation unit 16 receives the focus position corrected image 5 generated by the focus position correction unit 15 and the original image 2 for each focus position acquired by the original image acquisition unit 11. The all-focus image generation unit 16 extracts pixels corresponding to the corrected focus position for each pixel from the original image 2 for each focus position based on the focus position corrected image 5. The all-focus image generation unit 16 then combines the extracted pixels to generate an all-focus image 6. The all-focus image 6 is an image obtained by combining pixels in focus in the original image 2 for each focus position into a single image. As a result, the all-focus image 6 is an image in which each pixel is in focus with high precision.

[0032] The omnifocal image output unit 17 displays the omnifocal image 6 generated by the omnifocal image generation unit 16 on an external display (not shown), for example, or transfers the generated omnifocal image 6 to an external recording medium or device, for example.

[0033] [All-in-focus image generation method] 4 is a flowchart showing an all-in-focus image generating method (hereinafter referred to as method MT1) according to one embodiment. Method MT1 is made up of steps ST1 to ST7. Method MT1 starts when the original image acquisition unit 11 acquires an original image 2 for each focus position.

[0034] First, the original image acquisition unit 11 acquires an original image 2 for each focal position (original image acquisition step: step ST1). Subsequently, the feature extraction unit 12 extracts a feature from the original image 2 for each focal position (feature extraction step: step ST2). The feature extraction unit 12 extracts, for example, frequency components indicating the strength of the edge at the focal position included in each original image 2, in order from highest frequency band to lowest frequency band. In other words, the feature extraction unit 12 extracts high frequency components in order by repeatedly filtering each original image 2.

[0035] Fig. 5 is a diagram showing an example of frequency decomposition as an example of step ST2. As shown in Fig. 5, the feature extraction unit 12 generates a feature image 3 for each focus position for each frequency band by repeatedly performing frequency decomposition on each original image 2. In the example of Fig. 5, feature images 31 to 33 for each focus position are generated in descending order of frequency. The feature images 31 to 33 for each focus position have different focus positions among the multiple images in each frequency band.

[0036] The feature extraction unit 12 performs frequency decomposition on each original image 2 using, for example, a Laplacian pyramid. First, the feature extraction unit 12 compresses the resolution of each original image 2 by half, using each original image 2 as an original image. Next, the feature extraction unit 12 resizes the compressed image. Because the resolution of the resized image is the same as that of the compressed image, high-frequency components have been removed from each original image 2, resulting in an image that is blurred compared to each original image 2. Next, the feature extraction unit 12 subtracts the resized image from each original image 2 to generate a feature image 31 in which only the high-frequency components of each original image 2 have been extracted. By repeating this operation a predetermined number of times, a feature image 3 is generated for each frequency band and for each focus position.

[0037] Next, the focus position determination unit 13 determines the focus position for each pixel based on the feature extracted by the feature extraction unit 12 (focus position determination step: ST3). As shown in Fig. 5, the focus position determination unit 13, for example, integrates the feature image 3 for each focus position in the Z-axis direction to generate a focus position image 4. In the example of Fig. 5, focus position images 41 to 43 are generated in descending order of frequency.

[0038] The focus position determination unit 13 generates the focus position image 4 by, for example, a maximum value extraction method. An example will be described in which a focus position image 41 is generated from a feature amount image 31 for each focus position. First, at each pixel position of the focus position image 41, the focus position determination unit 13 extracts position information in the Z-axis direction to which the feature amount image 31 having the maximum pixel value belongs from any of the images constituting the feature amount image 31 for each focus position. The maximum pixel value is the pixel value of the in-focus pixel in the feature amount image 31 for each focus position. Next, the focus position determination unit 13 generates the focus position image 41 by integrating the position information in the Z-axis direction extracted for all pixel positions of the focus position image 41. The focus position image 41 is an image that represents the position in the Z-axis direction at which each pixel in the original image 2 for each focus position is in focus.

[0039] Next, the point spread information calculation unit 14 calculates point spread information P1 representing the distribution of frequency components in the spatial domain or frequency domain based on the focal position for the original image 2 for each focal position (point spread information calculation step: ST4). The point spread information P1 is, for example, a point spread function (PSF). For example, when the imaging device 10 includes a point light source, the point spread function is a function that quantitatively represents the spread of light from a spot diameter when light emitted from the point light source is projected onto the imaging device. The point spread function is a function in the spatial domain. It can also be said that the point spread function represents the distribution of frequency components for the original image 2 for each focal position in the spatial domain. In the point spread function, the more high-frequency components the original image 2 for each focal position has, the higher the luminance value becomes only near the center (closer to the point), and if the original image 2 for each focal position is composed only of low-frequency components, the luminance value tends to spread outward from the center.

[0040] FIG. 6 is a flowchart showing an example of a method for calculating a point spread function. FIG. 7 is a diagram showing an example of a method for calculating a point spread function corresponding to each step of FIG. 6. The method for calculating a point spread function is composed of steps ST41 to ST44, although step ST43 may be omitted. First, the point spread information calculation unit 14 acquires input data (step ST41). The input data is, for example, an original image 2 for each focal position. The point spread information calculation unit 14 may calculate point spread information based on at least one image of the original images 2 for each focal position. For example, the point spread information calculation unit 14 may acquire any one of the original images 2 for each focal position as input data. In this case, the point spread information calculation unit 14 may use, as input data, an image with the highest variance value among the original images 2 for each focal position, or an image with the lowest brightness in the Z direction.

[0041] In step ST41, the point spread information calculation unit 14 may use all of the original images 2 constituting the original image 2 for each focal position as input data, or may use a predetermined number of original images 2 as input data. In this case, the point spread information calculation unit 14 may calculate a point spread function for each original image 2 and then average the point spread functions for all of the original images 2 to calculate a single point spread function. Alternatively, the point spread information calculation unit 14 may divide each original image 2 into multiple blocks on a plane formed in the X and Y directions, and use each block as input data. Alternatively, the point spread information calculation unit 14 may perform frequency decomposition on each original image 2 using a Laplacian pyramid, and use each original image 2 for each frequency band as input data. In the example of FIG. 7, one original image 2 is used as input data.

[0042] Next, the point spread information calculation unit 14 obtains an amplitude spectrum P2 by frequency transforming the original image 2 (step ST42). The frequency transform may be a Fourier transform, a wavelet transform, or a transform using a Laplacian pyramid. In the example of FIG. 7, the amplitude spectrum P2 is obtained by a Fourier transform. The amplitude spectrum P2 is an image in which the frequency components change with increasing distance from the center.

[0043] Next, the point spread information calculation unit 14 performs noise removal on the amplitude spectrum P2 (step ST43). The point spread information calculation unit 14 removes noise components by, for example, performing low-pass filtering on the amplitude spectrum P2, and obtains a noise-removed amplitude spectrum P3.

[0044] Next, the point spread information calculation unit 14 performs an inverse frequency transform on the noise-removed amplitude spectrum P3 to obtain a point spread function P4 (step ST44). The inverse frequency transform may be an inverse Fourier transform. The amplitude spectrum P2 is an image in which the frequency components change with increasing distance from the focal position, with the focal position as the reference point. Therefore, by performing an inverse frequency transform on the noise-removed amplitude spectrum P3, the distribution of frequency components in the spatial domain is calculated. The calculation of the point spread function P4 completes the method for calculating the point spread function.

[0045] In step ST41, the input data is not limited to the original image 2 for each focal position. The point spread information calculation unit 14 may calculate point spread information based on the original image 2 for each focal position or the feature extracted by the feature extraction unit 12. For example, as shown in FIG. 8(a), the feature image 3 for each focal position calculated in step ST2 may be used as the input data. In this case, any one of the feature images 3 for each focal position may be used as the input data, or all of the original images 2 constituting the feature image 3 for each focal position may be used as the input data, or a predetermined number of original images 2 may be used as the input data. The method of selecting the input data may be the same as when the original image 2 for each focal position is used as the input data.

[0046] 8(b), the point spread information calculation unit 14 may calculate the point spread information P1 based on an optical parameter P6 of the imaging device 10 that captures an image of the object. The point spread information calculation unit 14 may store the optical parameter P6 in advance, or may acquire the optical parameter P6 from an image sensor included in the imaging device 10. The optical parameter P6 is, for example, the wavelength λ and numerical aperture NA of the measurement light. When the optical parameter P6 is the wavelength λ and numerical aperture NA of the measurement light, the point spread information calculation unit 14 calculates the point spread function P4, for example, as follows:

[0047] First, the point spread information calculation unit 14 calculates the diffraction limit d from the wavelength λ of the measurement light and the numerical aperture NA based on equation (1).

number

[0048] Next, the point spread information calculation unit 14 calculates a point spread function P4 based on a model equation of scalar-based diffraction occurring inside the microscope. The point spread function P4 is expressed by equation (2). In equation (2), the point spread function P4 is expressed as h(x, y, z).

number

[0049] Since the point spread function P4 at the focal position (Z=0) is calculated, equation (3) is derived by substituting 0 for Z in equation (2).

number

number

[0050] Next, the focus position correction unit 15 corrects the focus position for each pixel based on the point spread information P1 (focus position correction step: ST5). FIG. 9 is a diagram showing an example of focus position correction. In step ST5, a focus position corrected image 5 is generated in which the pixel values ​​of each pixel are smoothed to reduce noise components while maintaining focus position information for each pixel. In the example of FIG. 9, a focus position corrected image 51 is generated based on a point spread function P4 and a focus position image 41. The focus position corrected image 51 is generated based on a point spread function P42 obtained by compressing the resolution of the point spread function P4 by half and a focus position image 42. The focus position corrected image 53 is generated based on a point spread function P43 obtained by compressing the resolution of the point spread function P42 by half and a focus position image 43. In the example of FIG. 9, the focus position corrected images 51 to 53 are generated in descending order of frequency.

[0051] In step ST5, for example, focus position corrected images 51 to 53 are generated by a bilateral filter. An example will be described in which the focus position corrected image 51 is generated based on the focus position image 41. In focus position correction using a bilateral filter, for each pixel of the focus position image 41, values ​​of other pixels in the vicinity of each pixel are weighted and averaged. This weight is calculated based on the point spread function P4.

[0052] The focus position correction using the bilateral filter is expressed by equations (5) and (6).

number

number

[0053] w(x,y,m,n) represents the weight. In equation (6), Φ j (Mj+m,Nj+n) is the point spread function P4 expressed as the coefficients of a Gaussian filter in the spatial direction. Φ0(k×|G i,j (x,y)-G i,j (x+m, y+n)|, N0) expresses the point spread function P4 as the coefficients of a Gaussian filter in the Z-axis direction. In equation (6), the weight of pixels close to the pixel value of the center pixel of the kernel size is reduced, and the weight of pixels farther from the pixel value of the center pixel of the kernel size is increased.

[0054] Next, the omnifocus image generation unit 16 extracts pixels corresponding to the corrected focus positions of each pixel from the original image 2 for each focus position and integrates the extracted pixels (omnifocus image generation step: ST6). The omnifocus image generation unit 16 integrates the focus position corrected images 51 to 53 generated for each frequency band into a single image, thereby generating the omnifocus image 6.

[0055] Finally, the omni-focus image output unit 17 outputs the omni-focus image 6 (step ST7). With the output of the omni-focus image 6, the method MT1 ends.

[0056] [Action and effect] As described above, method MT1 uses point spread information P1 to suppress high-frequency components not present in original image 2 for each focal position, thereby reducing noise. As a result, while retaining information on the focal position for each pixel, the pixel values ​​of each pixel are smoothed to reduce noise, allowing for accurate calculation of the focal position. Furthermore, by using point spread information P1 for original image 2 as a correction parameter, it is no longer necessary to manually adjust the correction parameters to match the imaging target. By extracting pixels corresponding to the corrected focal positions from original image 2 for each focal position and integrating the extracted pixels, an all-in-focus image 6 can be easily generated.

[0057] The point spread information P1 may include a point spread function P4. In this case, the point spread function P4 is a function in the spatial domain. Since the focal position of each pixel is data in image space, it is easy to directly apply the point spread function P4. This allows the omnifocus image 6 to be generated efficiently.

[0058] The point spread information calculation step (step ST4) may include a step of frequency-transforming the original image 2 for each focal position to obtain an amplitude spectrum P2, and a step of inverse-frequency-transforming the amplitude spectrum P2 to calculate a point spread function P4. In the amplitude spectrum P2, the frequency components change with increasing distance from the focal position, with the focal position as the reference point. By inverse-frequency-transforming this amplitude spectrum P2, the distribution of frequency components in the spatial domain can be simply calculated.

[0059] In step ST4, the point spread information P1 may be calculated based on at least one image of the original images 2 for each focal position. In this case, the point spread information P1 can be calculated efficiently with a small number of input data.

[0060] In step ST4, the point spread information P1 may be calculated for each focal position based on the original image 2 or the feature image 3. In this case, the options for input data for calculating the point spread information P1 are expanded, and the degree of freedom in data processing can be improved.

[0061] In step ST4, the point spread information P1 may be calculated based on optical parameters P6 of the imaging device 10 that captures the object. In this case, the point spread information P1 can be calculated without being affected by the accuracy of image generation.

[0062] [Variations] Although the embodiments of the present disclosure have been described above, the present disclosure is not limited to the above-described embodiments.

[0063] [First Modification] The point spread information P1 is not limited to the point spread function P4, but may also be an optical transfer function (OTF). The optical transfer function is a function in the frequency domain. The optical transfer function can be calculated by frequency converting the point spread function P4. For example, in FIG. 7, the amplitude spectrum P3 after noise removal before inverse frequency converting the point spread function P4 corresponds to the optical transfer function. If step ST43 is omitted, the amplitude spectrum P2 may also correspond to the optical transfer function. In the following description, the amplitude spectrum P3 after noise removal will be described as the optical transfer function P5.

[0064] Even when the optical transfer function P5 is used, the point spread information calculation unit 14 may use any one of the original images 2 for each focal position as input data, or all of the original images 2 constituting the original images 2 for each focal position as input data, or a predetermined number of original images 2 as input data. When the point spread information P1 is the optical transfer function P5, step ST5A is executed instead of step ST5 in method MT1. The other steps are the same as those in method MT1.

[0065] 10 is a diagram for explaining step ST5A. In step ST5A, first, the focus position correction unit 15 generates a focus position frequency image 4A by frequency converting the focus position image 4. The focus position frequency image 4A is image data in the frequency domain, similar to the optical transfer function P5.

[0066] Next, the focus position correction unit 15 integrates the optical transfer function P5 and the focus position frequency image 4A to generate a corrected focus position frequency image 4B. The focus position correction unit 15, for example, calculates the Hadamard product of the optical transfer function P5 and the focus position frequency image 4A. In this case, when the pixel value of each pixel of the optical transfer function P5 and the pixel value of each pixel of the focus position frequency image 4A are expressed as a determinant, the focus position correction unit 15 multiplies each element of the determinant. At this time, the focus position correction unit 15 may assign a weight to each element of the determinant. For example, the focus position correction unit 15 may set the cutoff frequency to half the value of the maximum frequency of the frequency components included in the optical transfer function P5. The focus position correction unit 15 may then assign a weight of less than 1 to frequency components greater than the cutoff frequency.

[0067] Finally, the focus position correction unit 15 generates a focus position corrected image 5 by performing inverse frequency conversion on the corrected focus position frequency image 4B.

[0068] As described above, the point spread information P1 may include the optical transfer function P5. In this case, the optical transfer function P5 is a function in the frequency domain, and therefore is easily applicable to data processing in the frequency domain, and does not require inverse frequency transformation for application. This improves the degree of freedom in data processing.

[0069] [Second Modification] Step ST1 may include a step of excluding original images that do not include a focus position. FIG. 11 is a diagram for explaining the step of excluding original images that do not include a focus position. The original image acquisition unit 11 excludes original images 2A that do not include a focus position at all pixel positions from the original images 2 acquired for each focus position. In the example of FIG. 11, there are multiple original images 2A that do not include a focus position. The original image 2 in the top layer, the original image 2 in the layer immediately below the top layer, the original image 2 in the bottom layer, and the original image 2 in the layer immediately above the bottom layer are excluded as original images 2A that do not include a focus position. There may be only one original image 2A that does not include a focus position.

[0070] The original image acquisition unit 11 may calculate a variance value as an index for excluding an original image 2A that does not contain a focal position. For example, the original image acquisition unit 11 calculates the deviation of pixel values ​​for all pixels of each original image 2 and calculates the variance value of each original image 2 from the calculated deviation. Original images 2A that do not contain a focal position tend to have small variance values. If the variance value of each original image 2 is below a variance threshold, the original image acquisition unit 11 excludes the corresponding original image as an original image 2A that does not contain a focal position. Alternatively, the original image acquisition unit 11 may perform edge detection using first or second derivatives as an index for excluding an original image 2A that does not contain a focal position. The original image acquisition unit 11 may exclude an original image 2 in which no edge is detected as a result of edge detection as an original image 2A that does not contain a focal position.

[0071] By excluding the original image 2A that does not include the focal position, the number of images from which feature amounts are extracted in step ST2 can be reduced, and the speed at which the omnifocus image 6 is generated can be improved.

[0072] [Third Modification] Method MT may include a step of selecting a specific region between step ST1 and step ST2. FIG. 12 is a diagram for explaining the step of selecting a specific region. As shown in FIG. 12, the original image acquisition unit 11 selects a specific region R to be focused on in each original image of the acquired original images 2 for each focus position, based on, for example, user input from a user interface. The specific region R can be selected arbitrarily by, for example, the user. The outer edge of the specific region R may be curved or straight. The specific region R is common to all original images 2 for each focus position.

[0073] When the method MT includes a step of selecting a specific region, the steps from step ST2 onwards are performed only on the specific region R. In other words, they are performed only on the pixels included in the specific region R. In step ST2, the feature extraction unit 12 extracts features in the specific region R from the original image 2 for each focus position.

[0074] In step ST3, the focal position determination unit 13 determines the focal position for each pixel based on the feature amount in the specific region R. In step ST4, the point spread information calculation unit 14 calculates point spread information P1 representing the distribution of frequency components in the spatial domain or frequency domain based on the focal position for the specific region R. In step ST5, the focal position correction unit 15 corrects the focal position for each pixel in the specific region R based on the point spread information P1. In step ST6, the all-in-focus image generation unit 16 extracts pixels corresponding to the corrected focal positions for each pixel in the specific region R from the original image 2 for each focal position and integrates the extracted pixels. In the all-in-focus image 6 obtained by integration, all pixels included in the specific region R are in focus, while areas outside the specific region R are out of focus.

[0075] The step of selecting the specific region does not necessarily have to be performed by the original image acquisition unit 11. For example, the omnifocus-image generation device 1 may further include a selection unit for selecting the specific region R, and the selection unit may select the specific region R based on, for example, a user input from a user interface.

[0076] When the method includes the step of selecting a specific region, it is possible to reduce the amount of data of the image from which features are extracted, thereby improving the speed of generating the all-in-focus image 6. Note that the specific region R here is an area selected solely to reduce the amount of data, and the selected specific region R does not affect the accuracy of the all-in-focus image 6 generated by this method.

[0077] [Fourth Modification] The method MT may include, after step ST6, a step of converting the all-in-focus image 6 into an image focused on a specific region. FIG. 13 is a diagram illustrating the step of converting the all-in-focus image 6 into an image focused on a specific region. As shown in FIG. 13, the all-in-focus image generator 16 selects a specific region R to be focused on from the generated all-in-focus image 6, based on, for example, a user input from a user interface. The specific region R can be arbitrarily selected by, for example, the user. The outer edge of the specific region R may be curved or straight.

[0078] Next, the omnifocus image generation unit 16 converts the omnifocus image 6 into an image 6R that is in focus only in the selected specific region R. In the image 6R, all pixels included in the specific region R are in focus, while regions other than the specific region R are out of focus.

[0079] The all-in-focus image generation unit 16 may convert the all-in-focus image 6 into an image 6R by reducing the resolution of an area other than the specific area R in the all-in-focus image 6. The all-in-focus image generation unit 16 may convert the all-in-focus image 6 into an image 6R by cutting out an area other than the specific area R from any image of the original image 2 for each focus position or an image obtained by averaging pixel values ​​of the original image 2 for each focus position, and applying the cut-out area to the all-in-focus image 6.

[0080] The step of converting the all-in-focus image 6 into an image focused on a specific region does not necessarily have to be performed by the all-in-focus image generation unit 16. For example, the all-in-focus image generation device 1 may further include a conversion unit, which may convert the all-in-focus image 6 into an image focused on a specific region. By being able to convert the all-in-focus image 6 into an image 6R focused on only the specific region R selected by the user, the usability of the all-in-focus image 6 can be improved.

[0081] [Fifth Modification] When the feature extraction unit 12 performs frequency decomposition of each original image 2 using a Laplacian pyramid, the feature extraction unit 12 may calculate the image similarity in the feature image for each focus position for each frequency band, and if the image similarity satisfies a predetermined similarity condition, stop the subsequent frequency decomposition. For example, in Fig. 5, if the image similarity in the feature image 32 for each focus position satisfies a predetermined similarity condition, the feature extraction unit 12 may stop the subsequent frequency decomposition and not generate the feature image 33 for each focus position.

[0082] When the image similarity between the feature images for each focus position satisfies a predetermined similarity condition, it is considered that there is no significant difference in pixel values ​​between an image that includes pixels corresponding to the focus position and an image that does not include pixels corresponding to the focus position, and therefore the frequency band can be determined to be a frequency band unnecessary for determining the focus level. In other words, the fact that the image similarity satisfies the predetermined similarity condition serves as a judgment index that indicates that further frequency decomposition will not contribute to improving the accuracy of the omnifocus image.

[0083] An example of a method for calculating image similarity between feature images for each focus position is SSIM (Structural SIMilarity). The formula for calculating similarity using SSIM is shown in Equation (7), for example.

number

[0084] SSIM consists of the multiplication of the comparison terms of brightness (l(x,y)), contrast (c(x,y)), and structure (s(x,y)) of the edge image. The definition of SSIM is shown in Equation (8).

number

number

number

number

[0085] When α, β, and γ=1 in equation (8) and C3=C2 / 2 in equations (10) and (11), the SSIM is calculated by equation (12) from equations (8) to (11).

number

[0086] Another example of calculating the image similarity between edge images at each focus position that share a common frequency band is PSNR (Peak Signal to Noise Ratio). For example, when the sum of the differences between the PSNR of the Mth edge image and the PSNR of the (M+1)th edge image between the edge images at each focus position is below a predetermined difference threshold, the feature extraction unit 12 stops frequency decomposition for the frequency band below the difference threshold and subsequent frequency bands.

[0087] Another example of calculating the image similarity is, for example, a variance value. For example, when the sum of the differences between the variance values ​​of the Mth edge image and the (M+1)th edge image between edge images at each focus position is below a predetermined difference threshold, the feature extractor 12 stops frequency decomposition for the frequency band below the difference threshold and thereafter.

[0088] When the image similarity satisfies a predetermined similarity condition, it is considered that there is no significant difference in pixel values ​​between an image that includes a pixel corresponding to the focus position and an image that does not include a pixel corresponding to the focus position, and therefore the frequency band can be determined to be an unnecessary frequency band for determining the focus level. Therefore, by stopping frequency decomposition thereafter, the speed at which the all-in-focus image 6 is generated can be improved without sacrificing accuracy.

[0089] [Sixth Modification] The all-in-focus image generating methods according to the above-described embodiments and modifications may be applied to machine learning. For example, the all-in-focus image generating device 1 may further include a model generation unit, and the image information determination model may be generated in the model generation unit. FIG. 14 is a diagram illustrating an example of an application of the image information determination model according to the sixth modification. As shown in FIG. 14, input data to the image information determination model 100 may be an original image 2 for each focus position, a feature image 3 for each focus position, or a focus position image 4. In the example of FIG. 14, output data from the image information determination model 100 is an image 6R in which only a specific region R selected from the all-in-focus image 6 is in focus. In this case, the specific region R may be automatically selected by the image information determination model 100, rather than being selected by a user.

[0090] As another example of output data from the image information determination model 100, a new original image 2 or all-in-focus image 6 for each focus position may be output. Alternatively, when an image for cytological diagnosis is prepared as input data, image data reflecting, for example, detection of the position of a specific cell, estimation of the cell type, and staining of the cell according to the cell type and state may be output. Furthermore, the image information determination model 100 is not limited to outputting image data, and may, for example, determine whether or not to execute the method MT again.

[0091] In the above-described embodiment and each modified example, in step ST2 or step ST2A, the feature extraction unit 12 may extract features using a method other than frequency decomposition using a Laplacian pyramid. The feature extraction unit 12 may implement the Brenner gradient method, which evaluates the sharpness of an image. The feature extraction unit 12 may implement the Laplacian method or the modified Laplacian method, which are edge detection methods. The feature extraction unit 12 may implement filtering using a wavelet transform, a Gaussian difference filter, or a kernel, which are frequency decomposition methods. The feature extraction unit 12 may implement a convolutional neural network or a vision transformer as a deep learning model. The feature extraction unit 12 may implement an inner channel prior, which extracts high-frequency components of image saturation and high-frequency components of image brightness as features.

[0092] In the above-described embodiment and each modified example, in step ST3, the focus position determination unit 13 may determine the focus position for each pixel using a method other than the maximum value extraction method. The focus position determination unit 13 may, for example, perform principal component analysis. In this case, in the principal component analysis, for example, when the feature image 31 for each focus position shown in FIG. 5 is used as a variable, each variable may be multiplied by a weighting constant to form a composite variable, and the value at which the variance of the composite variable is maximized may be extracted as the principal component. In addition to the maximum value extraction method, the focus position determination unit 13 may perform weighted addition, average calculation, weighted average calculation, or waveform fitting.

[0093] In the above-described embodiment and each modification, in step ST5, the focus position correction unit 15 may correct the focus position for each pixel by a method other than the bilateral filter. The focus position correction unit 15 may also perform a mean-shift filter, a non-local means filter, a guided filter, a fast guided filter, a majority filter, or CBLM filtering.

[0094] FIG. 15 is a block diagram showing an example of an all-in-focus image generation program and its recording medium. In the example shown in FIG. 15, the all-in-focus image generation program 7 includes a main module 71, an original image acquisition module 72, a point spread information calculation module 73, a feature extraction module 74, a focus position determination module 75, a focus position correction module 76, an all-in-focus image generation module 77, and an all-in-focus image output module 78. Functions realized by a computer through execution of the all-in-focus image generation program 7 are similar to those of the all-in-focus image generation device 1 described above. The all-in-focus image generation program 7 is provided by a computer-readable recording medium 8, such as a CD-ROM, DVD, or ROM. The all-in-focus image generation program 7 may be provided by a semiconductor memory or as a computer data signal superimposed on a carrier wave via a network.

[0095] [Seventh Modification] In the omni-focus image generating step (step ST6), the omni-focus image generating unit 16 may, for example, integrate the focus position corrected images 51 to 53 generated for each frequency band to generate a single focus position corrected image. The focus positions of all pixels are accurately reflected in the single focus position corrected image. As a result, the omni-focus image generating unit 16 accurately extracts pixels corresponding to the focus positions of each pixel from the original image 2 for each focus position by referring to the single focus position corrected image.

[0096] [Eighth Modification] Extraction of features is not necessarily required when determining the focus position. For example, in method MT1, the feature extraction step (step ST2) may be omitted. In this case, since the feature image 3 is not generated, the focus position determination unit 13 does not need to generate the focus position image 4 in the focus position determination step (step ST3). Similarly, the focus position correction unit 15 does not need to generate the focus position correction image 5 in the focus position correction step (step ST5). The focus position determination unit 13 may generate the all-focus image 6 by, for example, combining pixels of images having the highest focus level at each pixel position (pixels indicating the focus position) from the original image 2 for each focus position. Then, the focus position correction unit 15 corrects the focus position for each pixel of the generated all-focus image 6 based on, for example, the point spread information P1 calculated in step ST4.

[0097] Two or more of the above-described first to eighth modifications may be combined with each other. For example, when the first modification and the fourth modification are combined, the focus-position corrected image 5 may be generated using the optical transfer function P5 as the point spread function P1, and the omnifocus image 6 generated from the focus-position corrected image 5 may be converted into an image focused on a specific region.

[0098] [First Example] Examples of the present disclosure will be described below. In Example 1, all-in-focus images were generated using the all-in-focus image generation methods of Example 1 and Comparative Example 1. Then, each of the generated all-in-focus images was evaluated using an evaluation index Qs to determine whether or not the focus was uniform across all pixels.

[0099] In Example 1, an all-in-focus image was generated according to the method MT shown in Fig. 4. In step ST2 in Example 1, frequency decomposition using a Laplacian pyramid was performed to extract features. In step ST3 in Example 1, a focus position image was generated using a maximum value extraction method. In step ST5 in Example 1, a focus position correction image was generated using a bilateral filter.

[0100] In Comparative Example 1, an all-in-focus image was generated without performing step ST4 in the method MT shown in Fig. 4. That is, point spread information was not calculated in Comparative Example 1. In Comparative Example 1, a plurality of all-in-focus images was generated by changing the parameter σ of the Gaussian filter in the bilateral filter, and the plurality of all-in-focus images was evaluated using the evaluation index Qs.

[0101] Evaluation index Q S In the structural similarity based metrics (SSM), an all-in-focus image is divided into multiple images by a window, and the image similarity between the divided images is evaluated. S is calculated by equation (13).

number

number

[0102] 16 shows the results of calculating the evaluation index Qs in Example 1 and Comparative Example 1. sσ indicates that the focus is more uniform across all pixels, and the quality of the omnifocus image is superior. Comparative Example 1 shows the change in evaluation index Qs when the parameter σ is changed. The lower the parameter σ, the more noise there is in the omnifocus image, but the blur is reduced. On the other hand, the higher the parameter σ, the less noise there is in the omnifocus image, but the blur is increased. In Comparative Example 1, it can be seen that the set value of the parameter σ is gradually increased from 0, and the evaluation index Qs becomes highest around 10. It can also be seen that the evaluation index Qs decreases as the set value of the parameter σ is gradually increased from 10. In contrast, in Example 1, the evaluation index Qs exhibits a value of approximately 80% to 90% of the maximum value of the evaluation index Qs in Comparative Example 1. In Example 1, by using a point spread function as a parameter of the bilateral filter, the evaluation index Qs is determined uniformly, regardless of the parameter σ. It can be confirmed that the parameters of the bilateral filter are calculated so as to calculate a high level of evaluation index Qs.

[0103] [Second Example] In the second example, all-in-focus images were generated using the all-in-focus image generation methods of Example 2, Example 3, Comparative Example 2, and Comparative Example 3. Then, each of the generated all-in-focus images was evaluated using the evaluation index Qs to determine whether or not the focus was uniform across all pixels.

[0104] In Example 2, an all-in-focus image was generated according to the method MT shown in Fig. 4. In step ST2 in Example 2, frequency decomposition using a Laplacian pyramid was performed to extract features. In step ST3 in Example 2, a focus position image was generated using a maximum value extraction method. In step ST5 in Example 2, a focus position correction image was generated using a bilateral filter.

[0105] In Example 3, an all-in-focus image was generated according to the method MT shown in Fig. 4. Example 3 is the same as Example 2, except that in step ST2, saturation and brightness components are extracted as features using the Inner Channel Prior instead of the Laplacian pyramid.

[0106] In Comparative Example 2, an all-in-focus image was generated without performing step ST4 in method MT shown in Fig. 4. In step ST2 in Comparative Example 2, frequency decomposition using a Laplacian pyramid was performed to extract features. In step ST3 in Comparative Example 2, a focus position image was generated using a maximum value extraction method. In step ST5 in Comparative Example 2, a focus position correction image was generated using a majority filter.

[0107] In Comparative Example 3, an all-in-focus image was generated without performing step ST4 in the method MT shown in Fig. 4. Comparative Example 3 is the same as Comparative Example 2 except that in step ST2, saturation and brightness components were extracted as features by the Inner Channel Prior instead of the Laplacian pyramid, and in step ST5, a focus position corrected image was generated by a high-speed guided filter instead of the majority filter.

[0108] The samples to be evaluated are multiple images equivalent to the original image 2 for each focal position in Fig. 2. As shown in Fig. 17, each of samples A to G is a multiple of images for cytological diagnosis. Each of samples A to G differs in the number of images and the size of each image.

[0109] FIG. 18 shows the results of the evaluation index Q s The results of calculating the above are shown in Table 1. It can be seen that the average values ​​of the evaluation index Qs in Examples 2 and 3 are improved compared to the average values ​​of the evaluation index Qs in Comparative Examples 2 and 3. This confirms that the quality of the omnifocus image is improved by using point spread information in focus position correction. [Explanation of symbols]

[0110] 1...all-in-focus image generation device, 10...imaging device, 11...original image acquisition unit, 13...focal position determination unit, 14...point spread information calculation unit, 15...focal position correction unit, 16...all-in-focus image generation unit, 2...original image, 6...all-in-focus image, 7...all-in-focus image generation program, MT1...all-in-focus image generation method, P1...point spread information, P2...amplitude spectrum, P4...point spread function, P5...optical transfer function, P6...optical parameters, ST1...original image acquisition step, ST3...focal position determination step, ST4...point spread information calculation step, ST5...focal position correction step, ST6...all-in-focus image generation step.

Claims

1. an original image acquisition step of acquiring original images for each focal position of the object by capturing images of the object while shifting the focal position in a predetermined direction; a point spread information calculation step of calculating point spread information regarding the original image for each focus position; a focus position determination step of determining a focus position for each pixel based on the original image for each focus position; a focus position correcting step of correcting a focus position for each pixel based on the point spread information; an all-in-focus image generation step of extracting pixels corresponding to the corrected focus positions for each pixel from the original image for each focus position and integrating the extracted pixels to generate an all-in-focus image.

2. The all-in-focus image generating method of claim 1 , wherein the point spread information includes a point spread function.

3. The point spread information calculation step includes: a step of frequency-converting the original image for each focal position to obtain an amplitude spectrum; The all-in-focus image generating method according to claim 2 , further comprising the step of: performing an inverse frequency transform on the amplitude spectrum to calculate the point spread function.

4. The all-in-focus image generating method of claim 1 , wherein the point spread information includes an optical transfer function.

5. 5. The all-in-focus image generating method according to claim 1, wherein the point spread information calculation step calculates the point spread information based on at least one image among the original images for each focus position.

6. 5. The all-in-focus image generating method according to claim 1, wherein the point spread information calculation step calculates the point spread information based on the original image for each focus position or a feature amount in the original image for each focus position.

7. 5. The omnifocal image generating method according to claim 1, wherein the point spread information calculating step calculates the point spread information based on optical parameters of an imaging device that images the object.

8. an original image acquisition unit that acquires original images of an object for each focal position by capturing images of the object while shifting the focal position in a predetermined direction; a point spread information calculation unit that calculates point spread information regarding the original image for each focus position; a focus position determination unit that determines a focus position for each pixel based on the original image for each focus position; a focus position correction unit that corrects a focus position for each pixel based on the point spread information; an omnifocal image generation unit that extracts pixels corresponding to the corrected focus positions of each pixel from the original image for each focus position and integrates the extracted pixels to generate an omnifocal image.

9. an original image acquisition step of acquiring original images for each focal position of the object by capturing images of the object while shifting the focal position in a predetermined direction; a point spread information calculation step of calculating point spread information regarding the original image for each focus position; a focus position determination step of determining a focus position for each pixel based on the original image for each focus position; a focus position correcting step of correcting a focus position for each pixel based on the point spread information; an all-in-focus image generation step of extracting pixels corresponding to the corrected focus positions for each pixel from the original image for each focus position and integrating the extracted pixels to generate an all-in-focus image.

Images