All-in-focus image generation method, all-in-focus image generation device, and all-in-focus image generation program

Abstract

A method MT1 comprises: an original image acquisition step (step ST1) in which an original image is acquired for each focus position regarding an object; a point spread information calculation step (step ST4) regarding the original image for each focus position; a focus position determination step (step ST3) in which, on the basis of the original image for each focus position, the focus position of each pixel is determined; a focus position correction step (step ST5) in which the focus position of each pixel is corrected on the basis of the point spread information; and an all-in-focus image generation step (step ST6) in which pixels corresponding to the respective corrected focus positions of the pixels are extracted, and an all-in-focus image is generated by combining the extracted pixels.

Description

All-in-focus image generating method, all-in-focus image generating device, and all-in-focus image generating program

[0001] The present disclosure relates to an omnifocus image generating method, an omnifocus image generating device, and an omnifocus image generating program.

[0002] A method is known in which the same object is photographed in the thickness direction (Z-axis direction) of the object while shifting the focal position, thereby obtaining multiple images (Z-stack images) at different focal positions, and generating an all-in-focus image in which each pixel is in focus from the Z-stack images (for example, Patent Document 1).

[0003] In the method of Patent Document 1, an all-in-focus image is generated by combining pixels (pixels indicating the focal position) of images with the highest degree of focus at each pixel position from acquired Z-stack images. Specifically, a focus degree calculation unit calculates the degree of focus at all pixel positions of two-dimensional images constituting the Z-stack image and sends the calculated values ​​to an all-in-focus image generation unit. The all-in-focus image generation unit selects a pixel value from a two-dimensional image in the Z direction that has the highest degree of focus (extracts the focal position) and generates the all-in-focus image by combining the selected pixel values ​​at all pixel positions.

[0004] International Publication No. WO 2018 / 042629

[0005] In the method of Patent Document 1, a focus level calculation unit uses a Laplacian pyramid that can represent high-frequency components such as edge information to calculate the focus level based on the high-frequency components, which may include noise components in addition to focus components.

[0006] In order to calculate the focus degree with high accuracy and generate an all-in-focus image with high accuracy, it is effective to separate the focus component from the noise component of the acquired Z-stack image or all-in-focus image, for example, as in the method of Patent Document 1. For example, it is expected that noise filtering will be performed on the Z-stack image or all-in-focus image, but manually adjusting the filter parameters depending on the imaging target is very cumbersome.

[0007] The present disclosure provides an all-in-focus image generating method, an all-in-focus image generating device, and an all-in-focus image generating program that can generate an all-in-focus image simply and accurately.

[0008] The gist of the present disclosure is as follows: [1] An all-in-focus image generating method including: an original image acquiring step of acquiring original images for each focus position of an object obtained by capturing images of the object while shifting the focus position in a predetermined direction, a point spread information calculating step of the original images for each focus position, a focus position determining step of determining a focus position for each pixel based on the original images for each focus position, a focus position correcting step of correcting the focus position for each pixel based on the point spread information, and an all-in-focus image generating step of extracting pixels corresponding to the corrected focus position for each pixel from the original images for each focus position and integrating the extracted pixels to generate an all-in-focus image.

[0009] This all-in-focus image generation method uses point spread information to suppress high-frequency components that are not present in the original image for each focus position, thereby reducing noise components. As a result, while retaining the focus position information for each pixel, the pixel values ​​of each pixel are smoothed to reduce noise components, allowing for accurate calculation of the focus position. Furthermore, by using point spread information for the original image as a correction parameter, there is no need to manually adjust the correction parameters to match the imaging target. An all-in-focus image can be easily generated by extracting pixels corresponding to the corrected focus position from the original image for each focus position and integrating the extracted pixels.

[0010] [2] The all-in-focus image generating method according to [1], wherein the point spread information includes a point spread function. In this case, the point spread function is a function in the spatial domain. Since the focal position of each pixel is data in image space, it is easy to directly apply the point spread function. This allows for efficient generation of an all-in-focus image.

[0011] [3] The all-in-focus image generating method according to [2], wherein the point spread information calculation step includes a step of frequency-transforming the original image for each focus position to obtain an amplitude spectrum, and a step of inverse-frequency-transforming the amplitude spectrum to calculate the point spread function. In the amplitude spectrum, frequency components change with increasing distance from the focus position. By performing an inverse frequency transform on this amplitude spectrum, it is possible to simply calculate the distribution of frequency components in a spatial domain based on the focus position.

[0012] [4] The all-in-focus image generating method according to [1], wherein the point spread information includes an optical transfer function. In this case, since the optical transfer function is a function in the frequency domain, it is easy to apply to data processing in the frequency domain and does not require inverse frequency transformation. This improves the flexibility of data processing.

[0013] [5] The all-in-focus image generating method according to any one of [1] to [4], wherein the point spread information calculation step calculates the point spread information based on at least one image among the original images for each focal position. In this case, the point spread information can be calculated efficiently with a small number of input data.

[0014] [6] The all-in-focus image generating method according to any one of [1] to [5], wherein the point spread information calculation step calculates the point spread information based on the original image for each focal position or a feature amount in the original image for each focal position. In this case, the options for input data for calculating the point spread information are expanded, thereby improving the degree of freedom in data processing.

[0015] [7] The all-in-focus image generating method according to any one of [1] to [6], wherein the point spread information is calculated based on optical parameters of an imaging device that captures the object in the point spread information calculation step. In this case, the point spread information can be calculated without being affected by image generation accuracy.

[0016] [8] An all-in-focus image generating device including: an original image acquiring unit that acquires original images for each focus position of an object obtained by capturing an image of the object while shifting the focus position in a predetermined direction; a point spread information calculating unit that calculates point spread information for the original images for each focus position; a focus position determining unit that determines the focus position for each pixel based on the original images for each focus position; a focus position correcting unit that corrects the focus position for each pixel based on the point spread information; and an all-in-focus image generating unit that extracts pixels corresponding to the corrected focus position for each pixel from the original images for each focus position and integrates the extracted pixels to generate an all-in-focus image.

[0017] This omnifocus image generating device uses point spread information to suppress high-frequency components not present in the original image for each focus position, thereby reducing noise components. As a result, while retaining the focus position information for each pixel, the pixel values ​​of each pixel are smoothed to reduce noise components, allowing for accurate calculation of the focus position. Furthermore, by using point spread information for the original image as a correction parameter, there is no need to manually adjust the correction parameters to match the imaging target. An omnifocus image can be easily generated by extracting pixels corresponding to the corrected focus position from the original image for each focus position and integrating the extracted pixels.

[0018] [9] An all-in-focus image generation program that causes a computer to execute an original image acquisition step of acquiring an original image for each focus position of an object obtained by imaging the object while shifting the focus position in a predetermined direction; a point spread information calculation step of calculating point spread information for the original image for each focus position; a focus position determination step of determining a focus position for each pixel based on the original image for each focus position; a focus position correction step of correcting the focus position for each pixel based on the point spread information; and an all-in-focus image generation step of extracting pixels corresponding to the corrected focus position for each pixel from the original image for each focus position and integrating the extracted pixels to generate an all-in-focus image.

[0019] This all-in-focus image generation program uses point spread information to suppress high-frequency components that are not present in the original image for each focus position, thereby reducing noise components. As a result, while retaining the focus position information for each pixel, the pixel values ​​of each pixel are smoothed to reduce noise components, allowing for accurate calculation of the focus position. Furthermore, by using point spread information for the original image as a correction parameter, there is no need to manually adjust the correction parameters to match the imaging target. An all-in-focus image can be easily generated by extracting pixels corresponding to the corrected focus position from the original image for each focus position and integrating the extracted pixels.

[0020] According to the present disclosure, an omnifocal image can be generated easily and accurately.

[0021] 8( a ) is a block diagram showing an all-in-focus image generating device according to an embodiment; FIG. 10 is a diagram showing an example of an original image for each focus position acquired by the original image acquisition unit shown in FIG. 1 ; FIG. 11 is a diagram showing an example of image data generated by the all-in-focus image device shown in FIG. 1 ; FIG. 12 is a flowchart showing an all-in-focus image generating method according to an embodiment; FIG. 13 is a diagram showing an example of frequency decomposition; FIG. 14 is a flowchart showing an example of a point spread information calculation step shown in FIG. 4 ; FIG. 15 is a diagram showing an example of a method for calculating a point spread function corresponding to each step of FIG. 6 ; FIG. 8( a ) is a diagram showing another example of input data for calculating a point spread function; FIG. 8( b ) is a diagram showing yet another example of input data for calculating a point spread function; FIG. 16 is a diagram showing an example of focus position correction; FIG. 17 is a diagram for explaining a focus position correction step according to a first modified example; FIG. 18 is a diagram for explaining a step of excluding an original image that does not include a focus position; FIG. 19 is a diagram for explaining a step of selecting a specific region; FIG. 20 is a diagram for explaining a step of converting an all-in-focus image into an image focused on a specific region; FIG. 21 is a diagram showing an example of application of an image information determination model according to a sixth modified example; FIG. 22 is a block diagram showing an example of an all-in-focus image generating program and a recording medium thereof; FIG. 23 is a diagram showing results of calculating evaluation indexes in Example 1 and Comparative Example 1; FIG. 24 is a diagram showing an evaluation target sample. FIG. 10 is a diagram showing the results of calculating evaluation indexes in Example 2, Example 3, Comparative Example 2, and Comparative Example 3.

[0022] Hereinafter, preferred embodiments of an omnifocus image generating method, an omnifocus image generating device, and an omnifocus image generating program according to an embodiment of the present disclosure will be described in detail with reference to the drawings.

[0023] [All-in-focus image generating device] Figure 1 is a block diagram showing an all-in-focus image generating device according to one embodiment. The all-in-focus image generating device 1 generates an all-in-focus image in which each pixel is in focus from an original image (a plurality of original images) acquired for each focal position. The all-in-focus image generating device 1 is configured as a device that acquires original images for each focal position by capturing images of a cell sample placed on a glass slide, for example, and generates an all-in-focus image of the cell sample. The generated all-in-focus image is used, for example, for cytological diagnosis and pathological diagnosis.

[0024] The all-in-focus image generating device 1 includes an original image acquisition unit 11, a feature extraction unit 12, a focal position determination unit 13, a point spread information calculation unit 14, a focal position correction unit 15, an all-in-focus image generating unit 16, and an all-in-focus image output unit 17. The all-in-focus image generating device 1 is physically a computer including a processor such as a CPU, and storage media such as RAM and ROM. The computer may be a smart device such as a smartphone or tablet terminal that is integrated with a display unit and an input unit. The computer may be configured as a microcomputer or a field-programmable gate array (FPGA). The computer may be configured externally via a network.

[0025] The original image acquisition unit 11 acquires original images 2 for each focal position of an object by capturing images of the object while shifting (moving) the focal position in a predetermined direction. The original image acquisition unit 11 is connected to, for example, an imaging device 10. The imaging device 10 is, for example, a microscope imaging device, and captures an enlarged image of an object placed on a stage. The original image acquisition unit 11 acquires original images for each focal position from the imaging device 10. Note that the original image acquisition unit 11 may include an imaging device, and the original image acquisition unit 11 may capture and acquire original images 2 for each focal position.

[0026] FIG. 2 is a diagram showing an example of an original image for each focal position acquired by the original image acquisition unit 11. The original image acquisition unit 11 acquires an original image 2 of a thick object. Here, for convenience of explanation, the thickness direction of the object is defined as the Z-axis direction. The direction perpendicular to the Z-axis direction is defined as the X-axis direction, and the direction perpendicular to the Z-axis direction and the X-axis direction is defined as the Y-axis direction. The imaging device 10 fixes the position of the same object at the time of imaging (positions in the X-axis and Y-axis directions) other than the Z-axis direction, and images the object multiple times in succession while shifting the focal position in the Z-axis direction (a predetermined direction). In this way, the original image acquisition unit 11 acquires original images 2 for each focal position with different focal positions. Because the original images 2 for each focal position are continuous in the Z-axis direction, they are referred to as Z-stack images.

[0027] The feature extraction unit 12 extracts feature amounts in the original image 2 for each focal position. The original image 2 for each focal position acquired by the original image acquisition unit 11 is input to the feature extraction unit 12. As shown in FIG. 3 , the feature extraction unit 12 generates a feature image 3 for each focal position in which the feature amounts are reflected. The feature images 3 for each focal position are continuous in the Z-axis direction, and the same number of feature images 3 are generated as the original images 2 for each focal position. The feature amounts are, for example, frequency components. Each original image 2 for each focal position contains frequency components that indicate the strength of the edge at the focal position. The feature extraction unit 12 extracts, for example, high-frequency components from the frequency components contained in the original image 2. The higher the frequency of the frequency component, the stronger the edge strength and the closer it is to the focal position.

[0028] The focus position determination unit 13 determines the focus position for each pixel based on the feature extracted by the feature extraction unit 12. A feature image 3 for each focus position is input to the focus position determination unit 13. The focus position determination unit 13, for example, integrates the feature image 3 for each focus position in the Z-axis direction to generate a focus position image 4. In the example of FIG. 3 , the focus position image 4 is a single image. The focus position image 4 is an image that indicates the focus position for each pixel. In other words, the focus position image 4 is an image in which each pixel is in focus.

[0029] The point spread information calculation unit 14 calculates point spread information P1. For example, the original image 2 for each focal position acquired by the original image acquisition unit 11 is input to the point spread information calculation unit 14. The point spread information P1 represents the frequency components of the original image 2 for each focal position in the spatial domain or frequency domain, in other words, it is information on the frequency distribution in the original image 2 for each focal position.

[0030] The focus position correction unit 15 corrects the focus position for each pixel based on the point spread information P1. The focus position correction unit 15 receives the point spread information P1 calculated by the point spread information calculation unit 14 and the focus position image 4 generated by the focus position determination unit 13. The focus position correction unit 15 generates a focus position corrected image 5 by correcting the focus position image 4. The focus position correction unit 15 reduces noise components by using the point spread information P1 to suppress high-frequency components that are not present in the original image 2 for each focus position. When the focus position correction unit 15 performs a convolution operation on the focus position image using the point spread information P1 generated from the original image 2 for each focus position, it can remove high-frequency components that are not present in the original image 2 for each focus position. However, simply performing a convolution operation may result in excessive removal of high-frequency components. Therefore, by performing a convolution operation using edge-preserving smoothing (e.g., a bilateral filter) as described later in FIG. 9 , it is possible to suppress high-frequency components that are not present in the original image 2 for each focus position while maintaining the high-frequency components of the focus position image 4. As a result, a focus position corrected image 5 is generated in which the pixel values ​​of each pixel are smoothed and noise components are reduced while retaining focus position information for each pixel. As a result, the accuracy of the focus position of each pixel in the focus position corrected image 5 is improved compared to the accuracy of the focus position of each pixel in the focus position image 4.

[0031] The omnifocus image generation unit 16 generates an omnifocus image 6. The omnifocus image generation unit 16 receives the focus position corrected image 5 generated by the focus position correction unit 15 and the original image 2 for each focus position acquired by the original image acquisition unit 11. The omnifocus image generation unit 16 extracts pixels corresponding to the corrected focus position for each pixel from the original image 2 for each focus position based on the focus position corrected image 5. The omnifocus image generation unit 16 then integrates the extracted pixels to generate an omnifocus image 6. The omnifocus image 6 is an image obtained by integrating pixels in focus in the original image 2 for each focus position into a single image. As a result, the omnifocus image 6 is an image in which each pixel is in focus with high precision.

[0032] The omnifocal image output unit 17 displays the omnifocal image 6 generated by the omnifocal image generation unit 16 on an external display (not shown), for example, or transfers the generated omnifocal image 6 to an external recording medium or device, for example.

[0033] 4 is a flowchart showing an all-in-focus image generating method (hereinafter referred to as method MT1) according to one embodiment. Method MT1 is composed of steps ST1 to ST7. Method MT1 starts when the original image acquisition unit 11 acquires an original image 2 for each focus position.

[0034] First, the original image acquisition unit 11 acquires an original image 2 for each focal position (original image acquisition step: step ST1). Next, the feature extraction unit 12 extracts features from the original image 2 for each focal position (feature extraction step: step ST2). The feature extraction unit 12 extracts, for example, frequency components indicating the strength of the edges at the focal positions included in each original image 2, in order from highest to lowest frequency band. In other words, the feature extraction unit 12 extracts high-frequency components in order by repeatedly filtering each original image 2.

[0035] Fig. 5 is a diagram showing an example of frequency decomposition as an example of step ST2. As shown in Fig. 5, the feature extractor 12 generates a feature image 3 for each focus position for each frequency band by repeatedly performing frequency decomposition on each original image 2. In the example of Fig. 5, feature images 31 to 33 for each focus position are generated in descending order of frequency. The feature images 31 to 33 for each focus position differ from one another among the multiple images in each frequency band.

[0036] The feature extraction unit 12 performs frequency decomposition on each original image 2 using, for example, a Laplacian pyramid. First, the feature extraction unit 12 compresses the resolution of each original image 2 by half, using each original image 2 as an original image. Next, the feature extraction unit 12 resizes the compressed image. Because the resolution of the resized image is the same as that of the compressed image, high-frequency components have been removed from each original image 2, resulting in an image that is blurred compared to each original image 2. Next, the feature extraction unit 12 subtracts the resized image from each original image 2 to generate a feature image 31 in which only the high-frequency components of each original image 2 have been extracted. By repeating this operation a predetermined number of times, a feature image 3 is generated for each frequency band and for each focus position.

[0037] Next, the focus position determination unit 13 determines the focus position for each pixel based on the feature extracted by the feature extraction unit 12 (focus position determination step: ST3). As shown in Fig. 5, the focus position determination unit 13, for example, integrates the feature image 3 for each focus position along the Z axis direction to generate a focus position image 4. In the example of Fig. 5, focus position images 41 to 43 are generated in descending order of frequency.

[0038] The focus position determination unit 13 generates the focus position image 4 using, for example, a maximum value extraction method. An example will be described in which a focus position image 41 is generated from a feature amount image 31 for each focus position. First, at each pixel position of the focus position image 41, the focus position determination unit 13 extracts position information in the Z-axis direction to which the feature amount image 31 having the maximum pixel value belongs from any of the images constituting the feature amount image 31 for each focus position. The maximum pixel value is the pixel value of the in-focus pixel in the feature amount image 31 for each focus position. Next, the focus position determination unit 13 generates the focus position image 41 by integrating the position information in the Z-axis direction extracted at all pixel positions of the focus position image 41. The focus position image 41 is an image that represents the position in the Z-axis direction at which each pixel in the original image 2 for each focus position is in focus.

[0039] Next, the point spread information calculation unit 14 calculates point spread information P1 representing the distribution of frequency components in the spatial domain or frequency domain based on the focal position for the original image 2 for each focal position (point spread information calculation step: ST4). The point spread information P1 is, for example, a point spread function (PSF). For example, when the imaging device 10 includes a point light source, the point spread function is a function that quantitatively represents the spread of light from the spot diameter when light emitted from the point light source is projected onto the imaging device. The point spread function is a function in the spatial domain. It can also be said that the point spread function represents the distribution of frequency components for the original image 2 for each focal position in the spatial domain. In the point spread function, the more high-frequency components the original image 2 for each focal position has, the higher the brightness value becomes only near the center (closer to the point), and if the original image 2 for each focal position is composed only of low-frequency components, the brightness value tends to spread out from the center.

[0040] FIG. 6 is a flowchart showing an example of a method for calculating a point spread function. FIG. 7 is a diagram showing an example of a method for calculating a point spread function corresponding to each step in FIG. 6. The method for calculating a point spread function includes steps ST41 to ST44, although step ST43 may be omitted. First, the point spread information calculation unit 14 acquires input data (step ST41). The input data is, for example, an original image 2 for each focal position. The point spread information calculation unit 14 may calculate point spread information based on at least one image from the original images 2 for each focal position. For example, the point spread information calculation unit 14 may acquire one of the original images 2 for each focal position as input data. In this case, the point spread information calculation unit 14 may use the image with the highest variance value from the original images 2 for each focal position as input data, or the image with the lowest brightness in the Z direction as input data.

[0041] In step ST41, the point spread information calculation unit 14 may use all of the original images 2 constituting the original image 2 for each focal position as input data, or may use a predetermined number of original images 2 as input data. In this case, the point spread information calculation unit 14 may calculate a point spread function for each original image 2 and then average the point spread functions for all of the original images 2 to calculate a single point spread function. Alternatively, the point spread information calculation unit 14 may divide each original image 2 into multiple blocks on a plane formed in the X direction and the Y direction, and use each block as input data. Alternatively, the point spread information calculation unit 14 may perform frequency decomposition on each original image 2 using a Laplacian pyramid, and use each original image 2 for each frequency band as input data. In the example of FIG. 7 , one original image 2 is used as input data.

[0042] Next, the point spread information calculation unit 14 obtains an amplitude spectrum P2 by frequency transforming the original image 2 (step ST42). The frequency transform may be a Fourier transform, a wavelet transform, or a Laplacian pyramid transform. In the example of Fig. 7, the amplitude spectrum P2 is obtained by a Fourier transform. The amplitude spectrum P2 is an image whose frequency components change with distance from the center.

[0043] Next, the point spread information calculation unit 14 performs noise removal on the amplitude spectrum P2 (step ST43). For example, the point spread information calculation unit 14 performs low-pass filtering on the amplitude spectrum P2 to remove noise components, and obtains a noise-removed amplitude spectrum P3.

[0044] Next, the point spread information calculation unit 14 performs an inverse frequency transform on the noise-removed amplitude spectrum P3 to obtain a point spread function P4 (step ST44). The inverse frequency transform may be an inverse Fourier transform. The amplitude spectrum P2 is an image in which the frequency components change with increasing distance from the focal position. Therefore, by performing an inverse frequency transform on the noise-removed amplitude spectrum P3, the distribution of frequency components in the spatial domain is calculated. The calculation of the point spread function P4 completes the method for calculating the point spread function.

[0045] In step ST41, the input data is not limited to the original image 2 for each focal position. The point spread information calculation unit 14 may calculate point spread information based on the original image 2 for each focal position or the feature extracted by the feature extraction unit 12. For example, as shown in FIG. 8A , the feature image 3 for each focal position calculated in step ST2 may be used as input data. In this case, any one of the feature images 3 for each focal position may be used as input data, or all of the original images 2 constituting the feature image 3 for each focal position may be used as input data, or a predetermined number of original images 2 may be used as input data. The method of selecting input data may be the same as when the original image 2 for each focal position is used as input data.

[0046] 8B, the point spread information calculation unit 14 may calculate the point spread information P1 based on optical parameters P6 of the imaging device 10 that captures an image of the object. The point spread information calculation unit 14 may store the optical parameters P6 in advance, or may acquire the optical parameters P6 from an image sensor included in the imaging device 10. The optical parameters P6 are, for example, the wavelength λ and numerical aperture NA of the measurement light. When the optical parameters P6 are the wavelength λ and numerical aperture NA of the measurement light, the point spread information calculation unit 14 calculates the point spread function P4, for example, as follows:

[0047] First, the point spread information calculation unit 14 calculates the diffraction limit d from the wavelength λ of the measurement light and the numerical aperture NA based on equation (1).

[0048] Next, the point spread information calculation unit 14 calculates a point spread function P4 based on a model equation of scalar-based diffraction occurring within the microscope. The point spread function P4 is expressed by equation (2). In equation (2), the point spread function P4 is expressed as h(x, y, z). In formula (2), J ０ represents the Bessel function of the first kind when the order is 0. C represents a normalization constant. d represents the diffraction grating calculated by equation (1). n i represents the refractive index of the medium between the objective lens and the sample. The model in equation (2) assumes that an index matching agent (oil, water, etc.) is placed between the objective lens and the sample. The index matching agent replaces the air between the objective lens and the sample, reducing the refractive index mismatch.

[0049] Since the point spread function P4 at the focal position (Z=0) is calculated, equation (3) is derived by substituting 0 for Z in equation (2). Using the constant k=2π / λ and equation (1), equation (4) is derived as an equation for calculating the point spread function P4.

[0050] Next, the focus position correction unit 15 corrects the focus position for each pixel based on the point spread information P1 (focus position correction step: ST5). FIG. 9 is a diagram showing an example of focus position correction. In step ST5, a focus position correction image 5 is generated in which the pixel values ​​of each pixel are smoothed to reduce noise components while maintaining the focus position information for each pixel. In the example of FIG. 9, a focus position correction image 51 is generated based on a point spread function P4 and a focus position image 41. A focus position correction image 51 is generated based on a point spread function P42 obtained by compressing the resolution of the point spread function P4 by half and a focus position image 42. A focus position correction image 53 is generated based on a point spread function P43 obtained by compressing the resolution of the point spread function P42 by half and a focus position image 43. In the example of FIG. 9, focus position correction images 51 to 53 are generated in descending order of frequency.

[0051] In step ST5, for example, focus position corrected images 51 to 53 are generated by a bilateral filter. An example will be described in which the focus position corrected image 51 is generated based on the focus position image 41. In focus position correction using a bilateral filter, for each pixel in the focus position image 41, the values ​​of other pixels in the vicinity of each pixel are weighted and averaged. This weight is calculated based on the point spread function P4.

[0052] The focus position correction using the bilateral filter is expressed by equations (5) and (6). In formula (5), G ｉ，ｊ (x, y) represents the image before correction, i.e., the focus position image 41. G2 ｉ，ｊ (x, y) represents the image after correction, that is, the focus position corrected image 51. Since the range of Σ is −Nj to Nj and −Mj to Mj, equation (5) represents a Gaussian filter using a kernel size of 2N_j rows × 2M_j columns.

[0053] w(x, y, m, n) represents the weight. ｊ (Mj+m, Nj+n) is the point spread function P4 expressed as the coefficients of a Gaussian filter in the spatial direction. ０ (k×|G ｉ，ｊ(x, y)-G ｉ，ｊ (x+m, y+n) |, N ０ ) expresses the point spread function P4 as a coefficient of a Gaussian filter in the Z-axis direction. In equation (6), the weight of pixels close to the pixel value of the center pixel of the kernel size is reduced, and the weight of pixels farther from the pixel value of the center pixel of the kernel size is increased.

[0054] Next, the omnifocus image generation unit 16 extracts pixels corresponding to the corrected focus positions of each pixel from the original image 2 for each focus position and integrates the extracted pixels (omnifocus image generation step: ST6). The omnifocus image generation unit 16 integrates the focus position corrected images 51 to 53 generated for each frequency band into a single image, thereby generating the omnifocus image 6.

[0055] Finally, the omnifocal image output unit 17 outputs the omnifocal image 6 (step ST7). With the output of the omnifocal image 6, the method MT1 ends.

[0056] [Actions and Effects] As described above, method MT1 uses point spread information P1 to suppress high-frequency components that are not present in the original image 2 for each focal position, thereby reducing noise components. As a result, while retaining information on the focal position for each pixel, the pixel values ​​of each pixel are smoothed to reduce noise components, allowing the focal position to be calculated with high accuracy. Furthermore, by using point spread information P1 for the original image 2 as correction parameters, it is no longer necessary to manually adjust the correction parameters to match the imaging target. Instead, pixels corresponding to the corrected focal positions are extracted from the original image 2 for each focal position, and the extracted pixels are integrated to easily generate the omnifocal image 6.

[0057] The point spread information P1 may include a point spread function P4. In this case, the point spread function P4 is a function in the spatial domain. Since the focal position of each pixel is data in image space, it is easy to directly apply the point spread function P4. This allows the omnifocus image 6 to be generated efficiently.

[0058] The point spread information calculation step (step ST4) may include a step of frequency-transforming the original image 2 for each focal position to obtain an amplitude spectrum P2, and a step of inverse-frequency-transforming the amplitude spectrum P2 to calculate a point spread function P4. In the amplitude spectrum P2, the frequency components change with increasing distance from the focal position, with the focal position as the reference point. By performing an inverse frequency transform on this amplitude spectrum P2, the distribution of frequency components in the spatial domain can be easily calculated.

[0059] In step ST4, the point spread information P1 may be calculated based on at least one image of the original images 2 for each focal position. In this case, the point spread information P1 can be calculated efficiently with a small number of input data.

[0060] In step ST4, the point spread information P1 may be calculated for each focal position based on the original image 2 or the feature image 3. In this case, the options for input data for calculating the point spread information P1 are expanded, and the degree of freedom in data processing can be improved.

[0061] In step ST4, the point spread information P1 may be calculated based on optical parameters P6 of the imaging device 10 that captures the object. In this case, the point spread information P1 can be calculated without being affected by the accuracy of image generation.

[0062] [Modifications] Although the embodiments of the present disclosure have been described above, the present disclosure is not limited to the above-described embodiments.

[0063] [First Modification] The point spread information P1 is not limited to the point spread function P4, but may also be an optical transfer function (OTF). The optical transfer function is a function in the frequency domain. The optical transfer function can be calculated by frequency converting the point spread function P4. For example, in FIG. 7, the amplitude spectrum P3 after noise removal before inverse frequency converting the point spread function P4 corresponds to the optical transfer function. If step ST43 is omitted, the amplitude spectrum P2 may also correspond to the optical transfer function. In the following description, the amplitude spectrum P3 after noise removal will be described as the optical transfer function P5.

[0064] Even when the optical transfer function P5 is used, the point spread information calculation unit 14 may use any one of the original images 2 for each focal position as input data, or all of the original images 2 constituting the original images 2 for each focal position as input data, or a predetermined number of original images 2 as input data. When the point spread information P1 is the optical transfer function P5, step ST5A is executed instead of step ST5 in method MT1. The other steps are the same as those in method MT1.

[0065] 10 is a diagram for explaining step ST5A. In step ST5A, the focus position correction unit 15 first generates a focus position frequency image 4A by frequency converting the focus position image 4. The focus position frequency image 4A is image data in the frequency domain, similar to the optical transfer function P5.

[0066] Next, the focus position correction unit 15 integrates the optical transfer function P5 and the focus position frequency image 4A to generate a corrected focus position frequency image 4B. The focus position correction unit 15, for example, calculates the Hadamard product of the optical transfer function P5 and the focus position frequency image 4A. In this case, when the pixel values ​​of each pixel of the optical transfer function P5 and the pixel values ​​of each pixel of the focus position frequency image 4A are expressed as a determinant, the focus position correction unit 15 multiplies each element of the determinant. At this time, the focus position correction unit 15 may assign a weight to each element of the determinant. For example, the focus position correction unit 15 may set the cutoff frequency to a value that is half the maximum frequency of the frequency components included in the optical transfer function P5. The focus position correction unit 15 may then assign a weight of less than 1 to frequency components with frequencies greater than the cutoff frequency.

[0067] Finally, the focus position correction unit 15 generates a focus position corrected image 5 by performing inverse frequency conversion on the corrected focus position frequency image 4B.

[0068] As described above, the point spread information P1 may include the optical transfer function P5. In this case, since the optical transfer function P5 is a function in the frequency domain, it is easy to apply to data processing in the frequency domain and does not require an inverse frequency transform for application. This allows for greater flexibility in data processing.

[0069] [Second Modification] Step ST1 may include a step of excluding original images that do not include a focal position. FIG. 11 is a diagram for explaining the step of excluding original images that do not include a focal position. The original image acquisition unit 11 excludes original images 2A that do not include a focal position at all pixel positions from the original images 2 acquired for each focal position. In the example of FIG. 11, there are multiple original images 2A that do not include a focal position. The original image 2 in the top layer, the original image 2 in the layer immediately below the top layer, the original image 2 in the bottom layer, and the original image 2 in the layer immediately above the bottom layer are excluded as original images 2A that do not include a focal position. There may be only one original image 2A that does not include a focal position.

[0070] The original image acquisition unit 11 may calculate a variance value as an index for excluding an original image 2A that does not include a focal position. For example, the original image acquisition unit 11 calculates the deviation of pixel values ​​for all pixels of each original image 2 and calculates the variance value of each original image 2 from the calculated deviation. Original images 2A that do not include a focal position tend to have small variance values. If the variance value of each original image 2 is below a variance threshold, the original image acquisition unit 11 excludes the corresponding original image as an original image 2A that does not include a focal position. Alternatively, the original image acquisition unit 11 may perform edge detection using first or second derivatives as an index for excluding an original image 2A that does not include a focal position. The original image acquisition unit 11 may exclude an original image 2 in which no edge is detected as a result of edge detection as an original image 2A that does not include a focal position.

[0071] By excluding the original image 2A that does not include the focal position, the number of images from which feature values ​​are extracted in step ST2 can be reduced, and the speed at which the omnifocus image 6 is generated can be improved.

[0072] [Third Modification] The method MT may include a step of selecting a specific region between steps ST1 and ST2. FIG. 12 is a diagram illustrating the step of selecting a specific region. As shown in FIG. 12, the original image acquisition unit 11 selects a specific region R on which the user wishes to focus in each of the acquired original images 2 for each focus position, based on, for example, user input from a user interface. The specific region R can be arbitrarily selected by the user, for example. The outer edge of the specific region R may be curved or straight. The specific region R is common to all of the original images 2 for each focus position.

[0073] When the method MT includes a step of selecting a specific region, the steps from step ST2 onwards are performed only on the specific region R. In other words, they are performed only on the pixels included in the specific region R. In step ST2, the feature extraction unit 12 extracts features in the specific region R from the original image 2 for each focus position.

[0074] In step ST3, the focal position determination unit 13 determines the focal position for each pixel based on the feature amount in the specific region R. In step ST4, the point spread information calculation unit 14 calculates point spread information P1 representing the distribution of frequency components in the spatial domain or frequency domain based on the focal position for the specific region R. In step ST5, the focal position correction unit 15 corrects the focal position for each pixel in the specific region R based on the point spread information P1. In step ST6, the omnifocus image generation unit 16 extracts pixels corresponding to the corrected focal positions for each pixel in the specific region R from the original image 2 for each focal position and integrates the extracted pixels. In the omnifocus image 6 obtained by integration, all pixels included in the specific region R are in focus, while areas outside the specific region R are out of focus.

[0075] The step of selecting the specific region does not necessarily have to be performed by the original image acquisition unit 11. For example, the omnifocus-image generation device 1 may further include a selection unit for selecting the specific region R, and the selection unit may select the specific region R based on, for example, a user input from a user interface.

[0076] When the method includes the step of selecting a specific region, it is possible to reduce the amount of data of the image from which features are extracted, thereby improving the speed at which the all-in-focus image 6 is generated. Note that the specific region R here is an area selected solely for the purpose of reducing the amount of data, and the selected specific region R does not affect the accuracy of the all-in-focus image 6 generated by this method.

[0077] [Fourth Modification] The method MT may include, after step ST6, a step of converting the all-in-focus image 6 into an image focused on a specific region. FIG. 13 is a diagram illustrating the step of converting the all-in-focus image 6 into an image focused on a specific region. As shown in FIG. 13, the all-in-focus image generator 16 selects a specific region R to be focused on from the generated all-in-focus image 6, based on, for example, a user input from a user interface. The specific region R can be arbitrarily selected by, for example, the user. The outer edge of the specific region R may be curved or straight.

[0078] Next, the omnifocus image generator 16 converts the omnifocus image 6 into an image 6R in which only the selected specific region R is in focus. In the image 6R, all pixels included in the specific region R are in focus, while regions other than the specific region R are out of focus.

[0079] The omnifocus image generation unit 16 may convert the omnifocus image 6 into an image 6R by reducing the resolution of an area other than the specific area R in the omnifocus image 6. The omnifocus image generation unit 16 may convert the omnifocus image 6 into an image 6R by cutting out an area other than the specific area R from any of the original images 2 for each focus position or an image obtained by averaging the pixel values ​​of the original images 2 for each focus position, and applying the cut-out area to the omnifocus image 6.

[0080] The step of converting the all-focus image 6 into an image focused on a specific region does not necessarily have to be performed by the all-focus image generation unit 16. For example, the all-focus image generation device 1 may further include a conversion unit that converts the all-focus image 6 into an image focused on a specific region. Being able to convert the all-focus image 6 into an image 6R focused on only the specific region R selected by the user can improve the usability of the all-focus image 6.

[0081] [Fifth Modification] When the feature extraction unit 12 performs frequency decomposition of each original image 2 using a Laplacian pyramid, the feature extraction unit 12 may calculate, for each frequency band, image similarity in the feature image for each focus position. Then, when the image similarity satisfies a predetermined similarity condition, the feature extraction unit 12 may stop further frequency decomposition. For example, in FIG. 5 , when the image similarity in the feature image 32 for each focus position satisfies the predetermined similarity condition, the feature extraction unit 12 may stop further frequency decomposition and not generate the feature image 33 for each focus position.

[0082] When the image similarity between the feature images for each focus position satisfies a predetermined similarity condition, it is considered that there is no significant difference in pixel values ​​between an image that includes pixels corresponding to the focus position and an image that does not include pixels corresponding to the focus position, and therefore the frequency band can be determined to be a frequency band unnecessary for determining the focus level. In other words, the fact that the image similarity satisfies the predetermined similarity condition serves as a judgment index that indicates that further frequency decomposition will not contribute to improving the accuracy of the omnifocus image.

[0083] An example of calculating the image similarity between feature images for each focus position is SSIM (Structural Similarity). The calculation formula for the similarity using SSIM is shown in Equation (7), for example. In Equation 7, the number of feature images for each focal position is L (L is an integer equal to or greater than 1). If the Mth edge image (M is an integer equal to or greater than 1 and equal to or less than L) is set as a variable x[M] and the (M+1)th edge image is set as a variable y[M+1], the SSIM between the Mth edge image and the (M+1)th edge image is calculated, and the calculated results are summed to calculate the similarity DS.

[0084] SSIM consists of multiplication of comparison terms of brightness (l(x,y)), contrast (c(x,y)), and structure (s(x,y)) of the edge image. The definition of SSIM is shown in Equation (8). In Equation 8, α, β, and γ are parameters for adjusting the weights of three comparison terms. The brightness (l(x,y)), contrast (c(x,y)), and structure (s(x,y)) are expressed by Equations (9) to (11), respectively. μ ｘ , μ ｙ is the average value of x[M], y[M+1], and σ ｘ , σ y is the standard deviation of x[M], y[M+1], and σ ｘ σ y is the covariance of x[M] and y[M+1]. １ ~C ３ is a constant provided to improve the instability of the output value when the denominator is close to 0 in the formulas (9) to (11).

[0085] If α, β, and γ are set to 1 in equation (8) and C3 is set to C2 / 2 in equations (10) and (11), SSIM is calculated by equation (12) from equations (8) to (11).

[0086] Another example of calculating the image similarity between edge images at each focus position that share a common frequency band is the peak signal-to-noise ratio (PSNR). For example, when the sum of the differences between the PSNRs of the Mth edge image and the (M+1)th edge image between the edge images at each focus position is below a predetermined difference threshold, the feature extraction unit 12 stops frequency decomposition for the frequency band below the difference threshold and subsequent frequency bands.

[0087] Another example of calculating the image similarity is, for example, a variance value. For example, when the sum of the differences between the variance values ​​of the Mth edge image and the (M+1)th edge image between edge images at each focus position is below a predetermined difference threshold, the feature extractor 12 stops frequency decomposition for the frequency band below the difference threshold and subsequent frequency bands.

[0088] When the image similarity satisfies a predetermined similarity condition, it is considered that there is no significant difference in pixel values ​​between an image that includes a pixel corresponding to the focus position and an image that does not include a pixel corresponding to the focus position, and therefore, the frequency band can be determined to be an unnecessary frequency band for determining the focus level. Therefore, by stopping frequency decomposition thereafter, the speed at which the omnifocus image 6 is generated can be improved without sacrificing accuracy.

[0089] [Sixth Modification] The all-in-focus image generation methods according to the above-described embodiments and modifications may be applied to machine learning. For example, the all-in-focus image generation device 1 may further include a model generation unit, and the image information determination model may be generated in the model generation unit. FIG. 14 is a diagram illustrating an example of an application of the image information determination model according to the sixth modification. As shown in FIG. 14 , input data to the image information determination model 100 may be an original image 2 for each focus position, a feature image 3 for each focus position, or a focus position image 4. In the example of FIG. 14 , output data from the image information determination model 100 is an image 6R in which only a specific region R selected from the all-in-focus image 6 is in focus. In this case, the specific region R may be automatically selected by the image information determination model 100, rather than being selected by a user.

[0090] As another example of output data from the image information determination model 100, a new original image 2 or all-focus image 6 for each focus position may be output. Alternatively, when an image for cytological diagnosis is prepared as input data, image data reflecting, for example, detection of the position of a specific cell, estimation of the cell type, and staining of the cell according to the cell type or state may be output. Furthermore, the image information determination model 100 is not limited to outputting image data, and may, for example, determine whether or not to execute the method MT again.

[0091] In the above-described embodiment and each modified example, in step ST2 or step ST2A, the feature extraction unit 12 may extract features using a method other than frequency decomposition using a Laplacian pyramid. The feature extraction unit 12 may implement the Brenner gradient method to evaluate image sharpness. The feature extraction unit 12 may implement the Laplacian method or the modified Laplacian method, which are edge detection methods. The feature extraction unit 12 may implement a wavelet transform, a Gaussian difference filter, or a kernel filtering process, which are frequency decomposition methods. The feature extraction unit 12 may implement a convolutional neural network or a vision transformer as a deep learning model. The feature extraction unit 12 may implement an inner channel prior that extracts high-frequency components of image saturation and high-frequency components of image brightness as features.

[0092] In the above-described embodiment and each modified example, in step ST3, the focus position determination unit 13 may determine the focus position for each pixel using a method other than the maximum value extraction method. The focus position determination unit 13 may, for example, perform principal component analysis. In this case, in the principal component analysis, for example, when the feature image 31 for each focus position shown in FIG. 5 is used as a variable, each variable may be multiplied by a weighting constant to obtain a composite variable, and the value at which the variance of the composite variable is maximized may be extracted as the principal component. In addition to the maximum value extraction method, the focus position determination unit 13 may also perform weighted addition, average calculation, weighted average calculation, or waveform fitting.

[0093] In the above-described embodiment and each modification, in step ST5, the focus position correction unit 15 may correct the focus position for each pixel by a method other than the bilateral filter. The focus position correction unit 15 may also perform a mean-shift filter, a non-local means filter, a guided filter, a fast guided filter, a majority filter, or CBLM filtering.

[0094] 15 is a block diagram showing an example of an all-in-focus image generation program and its recording medium. In the example shown in FIG. 15, the all-in-focus image generation program 7 includes a main module 71, an original image acquisition module 72, a point spread information calculation module 73, a feature extraction module 74, a focus position determination module 75, a focus position correction module 76, an all-in-focus image generation module 77, and an all-in-focus image output module 78. Functions realized by a computer through execution of the all-in-focus image generation program 7 are similar to the functions of the all-in-focus image generation device 1 described above. The all-in-focus image generation program 7 is provided by a computer-readable recording medium 8, such as a CD-ROM, DVD, or ROM. The all-in-focus image generation program 7 may be provided by a semiconductor memory or as a computer data signal superimposed on a carrier wave via a network.

[0095] [Seventh Modification] In the all-focus-image generating step (step ST6), the all-focus-image generating unit 16 may, for example, integrate the focus-position-corrected images 51 to 53 generated for each frequency band to generate a single focus-position-corrected image. The focus positions of all pixels are accurately reflected in the single focus-position-corrected image. As a result, the all-focus-image generating unit 16 accurately extracts pixels corresponding to the focus positions of each pixel from the original image 2 for each focus position by referring to the single focus-position-corrected image.

[0096] Eighth Modification: Extraction of features is not necessarily required when determining the focal position. For example, in method MT1, the feature extraction step (step ST2) may be omitted. In this case, since the feature image 3 is not generated, the focal position determination unit 13 does not need to generate the focal position image 4 in the focal position determination step (step ST3). Similarly, the focus position correction unit 15 does not need to generate the focus position correction image 5 in the focus position correction step (step ST5). The focal position determination unit 13 may generate the all-focus image 6 by, for example, combining pixels of images having the highest focus level at each pixel position (pixels indicating the focal position) from the original image 2 for each focal position. Then, the focus position correction unit 15 corrects the focal position for each pixel of the generated all-focus image 6 based on, for example, the point spread information P1 calculated in step ST4.

[0097] Two or more of the first to eighth modified examples described above may be combined with each other. For example, when the first modified example and the fourth modified example are combined, the focus-position corrected image 5 may be generated using the optical transfer function P5 as the point spread function P1, and the omnifocal image 6 generated from the focus-position corrected image 5 may be converted into an image focused on a specific region.

[0098] First Example An example of the present disclosure will be described below. In the first example, all-in-focus images were generated using the all-in-focus image generation methods of Example 1 and Comparative Example 1. Then, each of the generated all-in-focus images was evaluated using the evaluation index Qs to determine whether or not the focus was uniform across all pixels.

[0099] In Example 1, an all-in-focus image was generated according to the method MT shown in Fig. 4. In step ST2 in Example 1, frequency decomposition using a Laplacian pyramid was performed to extract features. In step ST3 in Example 1, a focus position image was generated using a maximum value extraction method. In step ST5 in Example 1, a focus position correction image was generated using a bilateral filter.

[0100] In Comparative Example 1, an all-in-focus image was generated without performing step ST4 in the method MT shown in Fig. 4. That is, point spread information was not calculated in Comparative Example 1. In Comparative Example 1, a plurality of all-in-focus images was generated by changing the parameter σ of the Gaussian filter in the bilateral filter, and the plurality of all-in-focus images was evaluated using the evaluation index Qs.

[0101] Evaluation index Q Ｓ In the structural similarity based metrics, an all-in-focus image is divided into multiple images by windows, and the image similarity between the divided images is evaluated. Ｓ is calculated by equation (13). Here, W and w represent a sliding window, Q(x, f|w) represents a local quality index between variables x and f within window w, and Q(y, f|w) represents a local quality index between variables y and f within window w. λ(w) represents a weighting coefficient, which determines the weighting of Q(x, f|w) and Q(y, f|w). λ(w) is calculated using Equation (14).

[0102] FIG. 16 shows the results of calculating the evaluation index Qs in Example 1 and Comparative Example 1. ｓThe larger the value, the more uniform the focus across all pixels, indicating superior quality of the omnifocus image. Comparative Example 1 shows the change in evaluation index Qs when the parameter σ is changed. The lower the parameter σ, the more noise there is in the omnifocus image, but the less blurring there is. On the other hand, the higher the parameter σ, the less noise there is in the omnifocus image, but the more blurring there is. In Comparative Example 1, it can be seen that the set value of the parameter σ is gradually increased from 0, and the evaluation index Qs becomes highest around 10. It can also be seen that the evaluation index Qs decreases as the set value of the parameter σ is gradually increased from 10. In contrast, in Example 1, the evaluation index Qs exhibits a value of approximately 80% to 90% of the maximum value of the evaluation index Qs in Comparative Example 1. In Example 1, by using a point spread function as a parameter of the bilateral filter, the evaluation index Qs is determined uniformly regardless of the parameter σ. In Example 1, it can be confirmed that the parameters of the bilateral filter are calculated so as to calculate a high level of evaluation index Qs.

[0103] Second Example In the second example, all-in-focus images were generated using the all-in-focus image generation methods of Example 2, Example 3, Comparative Example 2, and Comparative Example 3. Then, each of the generated all-in-focus images was evaluated using the evaluation index Qs to determine whether or not the focus was uniform across all pixels.

[0104] In Example 2, an all-in-focus image was generated according to the method MT shown in Fig. 4. In step ST2 in Example 2, frequency decomposition using a Laplacian pyramid was performed to extract features. In step ST3 in Example 2, a focus position image was generated using a maximum value extraction method. In step ST5 in Example 2, a focus position correction image was generated using a bilateral filter.

[0105] In Example 3, an all-in-focus image was generated according to the method MT shown in Fig. 4. Example 3 is the same as Example 2 except that in step ST2, saturation and brightness components are extracted as features using an inner channel prior instead of the Laplacian pyramid.

[0106] In Comparative Example 2, an all-in-focus image was generated without performing step ST4 in the method MT shown in Fig. 4. In step ST2 in Comparative Example 2, frequency decomposition using a Laplacian pyramid was performed to extract features. In step ST3 in Comparative Example 2, a focus position image was generated using a maximum value extraction method. In step ST5 in Comparative Example 2, a focus position correction image was generated using a majority filter.

[0107] In Comparative Example 3, an all-in-focus image was generated without performing step ST4 in the method MT shown in Fig. 4. Comparative Example 3 is the same as Comparative Example 2 except that in step ST2, saturation components and brightness components were extracted as feature amounts by an inner channel prior instead of a Laplacian pyramid, and in step ST5, a focus position corrected image was generated by a high-speed guided filter instead of a majority filter.

[0108] The samples to be evaluated are multiple images equivalent to the original image 2 for each focal position in Fig. 2. As shown in Fig. 17, each of samples A to G is a multiple of images for cytological diagnosis. Each of samples A to G differs in the number of images and the size of each image.

[0109] FIG. 18 shows the evaluation index Q ｓ The results of calculating the above are shown in Table 1. It can be seen that the average values ​​of the evaluation index Qs in Examples 2 and 3 are improved compared to the average values ​​of the evaluation index Qs in Comparative Examples 2 and 3. This confirms that the quality of the omnifocus image is improved by using point spread information in focus position correction.

[0110] 1...All-in-focus image generation device, 10...imaging device, 11...original image acquisition unit, 13...focal position determination unit, 14...point spread information calculation unit, 15...focal position correction unit, 16...all-in-focus image generation unit, 2...original image, 6...all-in-focus image, 7...all-in-focus image generation program, MT1...all-in-focus image generation method, P1...point spread information, P2...amplitude spectrum, P4...point spread function, P5...optical transfer function, P6...optical parameters, ST1...original image acquisition step, ST3...focal position determination step, ST4...point spread information calculation step, ST5...focal position correction step, ST6...all-in-focus image generation step.

Claims

1. An all-in-focus image generation method comprising: an original image acquisition step of acquiring original images for each focus position of an object obtained by capturing images of the object while shifting the focus position in a predetermined direction; a point spread information calculation step of calculating point spread information for the original images for each focus position; a focus position determination step of determining a focus position for each pixel based on the original images for each focus position; a focus position correction step of correcting the focus position for each pixel based on the point spread information; and an all-in-focus image generation step of extracting pixels corresponding to the corrected focus position for each pixel from the original images for each focus position and integrating the extracted pixels to generate an all-in-focus image.

2. The all-in-focus image generating method according to claim 1, wherein the point spread information includes a point spread function.

3. The all-in-focus image generating method according to claim 2, wherein the point spread information calculation step includes: a step of frequency-transforming the original image for each focus position to obtain an amplitude spectrum; and a step of inverse-frequency-transforming the amplitude spectrum to calculate the point spread function.

4. The all-in-focus image generating method according to claim 1, wherein the point spread information includes an optical transfer function.

5. The all-in-focus image generating method according to any one of claims 1 to 4, wherein the point spread information calculation step calculates the point spread information based on at least one image among the original images for each focus position.

6. The all-in-focus image generating method according to any one of claims 1 to 5, wherein the point spread information calculation step calculates the point spread information based on the original image for each focus position or a feature amount in the original image for each focus position.

7. The all-in-focus image generating method according to any one of claims 1 to 6, wherein the point spread information calculation step calculates the point spread information based on optical parameters of an imaging device that images the object.

8. An all-in-focus image generating device comprising: an original image acquiring unit that acquires original images for each focus position of an object obtained by capturing images of the object while shifting the focus position in a predetermined direction; a point spread information calculating unit that calculates point spread information for the original images for each focus position; a focus position determining unit that determines the focus position for each pixel based on the original images for each focus position; a focus position correcting unit that corrects the focus position for each pixel based on the point spread information; and an all-in-focus image generating unit that extracts pixels corresponding to the corrected focus position for each pixel from the original images for each focus position and integrates the extracted pixels to generate an all-in-focus image.

9. An all-in-focus image generation program that causes a computer to execute the following steps: an original image acquisition step of acquiring original images for each focus position of an object obtained by capturing images of the object while shifting the focus position in a predetermined direction; a point spread information calculation step of calculating point spread information for the original images for each focus position; a focus position determination step of determining a focus position for each pixel based on the original images for each focus position; a focus position correction step of correcting the focus position for each pixel based on the point spread information; and an all-in-focus image generation step of extracting pixels corresponding to the corrected focus position for each pixel from the original images for each focus position and integrating the extracted pixels to generate an all-in-focus image.

Images