All-in-focus image generation method, all-in-focus image generation device, and all-in-focus image generation program

Abstract

A method MT1 comprises: an original image acquisition step (step ST1) for capturing an image of an object while shifting a focal position in a predetermined direction to acquire original images of the object at respective focal positions; a frequency decomposition step (step ST2) for frequency-decomposing the original images at the respective focal positions to generate edge images in respective frequency bands; an all-in-focus edge image generation step (step ST3) for integrating the edge images in the respective frequency bands in the predetermined direction to generate all-in-focus edge images in the respective frequency bands; and an all-in-focus image generation step (step ST4) for integrating the all-in-focus edge images in the respective frequency bands to generate an all-in-focus image.

Description

All-in-focus image generating method, all-in-focus image generating device, and all-in-focus image generating program

[0001] The present disclosure relates to an omnifocus image generating method, an omnifocus image generating device, and an omnifocus image generating program.

[0002] A method is known in which the same object is photographed in the thickness direction (Z-axis direction) of the object while shifting the focal position, thereby obtaining multiple images (Z-stack images) at different focal positions, and generating an all-in-focus image in which each pixel is in focus from the Z-stack images (for example, Patent Document 1).

[0003] In the method of Patent Document 1, an all-in-focus image is generated by collecting pixels (pixels indicating the focal position) of images with the highest degree of focus at each pixel position from acquired Z-stack images. Specifically, a focus degree calculation unit calculates the degree of focus at all pixel positions of two-dimensional images constituting the Z-stack image and sends the calculated values ​​to an all-in-focus image generation unit. The all-in-focus image generation unit selects the pixel value of any two-dimensional image in the Z direction that has the highest degree of focus (extracts the focal position) and generates the all-in-focus image by combining the selected pixel values ​​at all pixel positions.

[0004] International Publication No. WO 2018 / 042629

[0005] However, the method of Patent Document 1 extracts the focal position at every pixel position. Therefore, for example, for a large object spanning multiple pixels, the entire object cannot be evaluated relatively, and the center of the object may be blurred in the omnifocus image. To address this issue, it is possible to expand the area in which the focal position is extracted (e.g., every multiple pixels). However, this reduces the calculation speed and makes it difficult to accurately calculate the focus level for objects smaller than the range of the extraction area, which may result in the small object being blurred in the omnifocus image. Therefore, a method is desired that can generate an omnifocus image with high accuracy, regardless of the size of the extraction area.

[0006] The present disclosure provides an omnifocus image generating method, an omnifocus image generating device, and an omnifocus image generating program that can generate an omnifocus image with high accuracy without extracting a region from an original image.

[0007] The gist of the present disclosure is as follows.

[0008] [1] An all-focus image generation method comprising: an original image acquisition step of capturing an image of an object while shifting the focus position in a predetermined direction and acquiring an original image for each focus position of the object; a frequency decomposition step of frequency-decomposing the original image for each focus position and generating an edge image for each frequency band; an all-focus edge image generation step of integrating the edge images for each frequency band in the predetermined direction and generating an all-focus edge image for each frequency band; and an all-focus image generation step of integrating the all-focus edge images for each frequency band to generate an all-focus image.

[0009] In this all-in-focus image generation method, original images for each focus position, which are shifted from one another, are acquired, and frequency components contained in each original image for each focus position are extracted for each frequency band to generate edge images for each frequency band. Because the frequency components contain information indicating the focus position in each original image, frequency decomposition of each original image allows the focus position to be extracted with high accuracy in the edge image for each frequency band. All-in-focus images generated for each frequency band by integrating the edge images for each frequency band reflect information indicating the different focus positions for each frequency band, and all pixels are in focus. This method allows all-in-focus images to be generated with high accuracy without extracting specific regions from the original images for each focus position.

[0010] [2] The all-in-focus image generating method according to [1], wherein the frequency decomposition step decomposes each of the original images for each focus position into a plurality of color component images, and generates edge images for each of the frequency bands corresponding to the plurality of color component images by frequency-resolving each of the plurality of color component images. In this case, by frequency-resolving each of the plurality of color component images and generating edge images for each frequency band corresponding to the plurality of color component images, the all-in-focus image can further reflect information indicating different focus positions for each color component image. This allows the all-in-focus image to be generated with even higher accuracy.

[0011] [3] The all-in-focus image generating method according to [1] or [2], wherein the frequency decomposition step excludes images that do not include the focus position from the original images for each focus position. In this case, by excluding images that do not include the focus position, the number of images to be decomposed in the frequency decomposition step can be reduced, and the speed at which the all-in-focus image is generated can be improved.

[0012] [4] The all-focus image generating method according to any one of [1] to [3] further includes a selection step of selecting a specific region in the original image for each focus position, wherein the frequency decomposition step performs frequency decomposition on the specific region in each of the original images for each focus position, and generates an edge image for each frequency band in the specific region for each frequency band. In this case, the amount of data of the image to be decomposed in the frequency decomposition step can be reduced, thereby improving the speed of generating the all-focus image. Note that the specific region here is a region selected solely for the purpose of reducing the amount of data, and the selected specific region does not affect the accuracy of the all-focus image generated by this method.

[0013] [5] The all-in-focus image generating method according to any one of [1] to [4], further comprising a conversion step of selecting a specific region in the all-in-focus image and converting the all-in-focus image into an image focused on the specific region. In this case, the all-in-focus image can be converted into an image focused on only the specific region selected by the user, thereby improving the usability of the all-in-focus image.

[0014] [6] The all-in-focus image generation method according to any one of [1] to [5], wherein the frequency decomposition step generates edge images for each frequency band in descending order of frequency, calculates image similarity between edge images for each frequency band at each focus position that share the same frequency band, and stops subsequent frequency decomposition if the image similarity satisfies a predetermined similarity condition. When the image similarity satisfies the predetermined similarity condition, it is considered that there is no longer a significant difference in pixel values ​​between an image that includes a pixel corresponding to the focus position and an image that does not include a pixel corresponding to the focus position. Therefore, by stopping subsequent frequency decomposition, the speed of generating the all-in-focus image can be improved without sacrificing accuracy.

[0015] [7] An all-focus image generating device comprising: an original image acquisition unit that images an object while shifting the focus position in a predetermined direction and acquires an original image for each focus position of the object; a frequency decomposition unit that frequency-decomposes the original image for each focus position and generates an edge image for each frequency band; an all-focus edge image generation unit that integrates the edge images for each frequency band in the predetermined direction and generates an all-focus edge image for each frequency band; and an all-focus image generation unit that integrates the all-focus edge images for each frequency band to generate an all-focus image.

[0016] The above-described all-in-focus image generating device acquires original images for each focus position, which are offset from one another, and generates edge images for each frequency band by extracting frequency components contained in each original image for each focus position. Because the frequency components contain information indicating the focus position in each original image, frequency decomposition of each original image allows for accurate extraction of the focus position in the edge image for each frequency band. The all-in-focus images generated for each frequency band by integrating the edge images for each frequency band reflect information indicating the different focus positions for each frequency band, and all pixels are in focus. This method allows for highly accurate generation of all-in-focus images without extracting specific regions from the original images for each focus position.

[0017] [8] The all-in-focus image generating device according to [7], wherein the frequency decomposition unit decomposes each of the original images for each focal position into a plurality of color component images and frequency-resolves each of the plurality of color component images to generate edge images for each of the frequency bands corresponding to the plurality of color component images. In this case, by frequency-resolving each of the plurality of color component images to generate edge images for each frequency band corresponding to the plurality of color component images, information indicating different focal positions for each color component image can be further reflected in the all-in-focus image. This allows the all-in-focus image to be generated with even higher accuracy.

[0018] [9] The all-in-focus image generating device according to [7] or [8], wherein the frequency decomposition unit excludes images that do not include the focus position from the original images for each focus position. In this case, by excluding images that do not include the focus position, the number of images to be decomposed in the frequency decomposition unit can be reduced, and the speed at which the all-in-focus image is generated can be improved.

[0019]

[10] The all-in-focus image generating device according to any one of [7] to [9] further includes a selection unit that selects a specific region in the original image for each focus position, and the frequency decomposition unit performs frequency decomposition on the specific region in each of the original images for each focus position to generate an edge image for each frequency band in the specific region. In this case, the amount of data of the image to be decomposed by the frequency decomposition unit can be reduced, thereby improving the speed of generating the all-in-focus image. Note that the specific region here is a region selected solely to reduce the amount of data, and the selected specific region does not affect the accuracy of the all-in-focus image generated by this method.

[0020]

[11] The all-in-focus image generating device according to any one of [7] to

[10] , further comprising a conversion unit that selects a specific region in the all-in-focus image and converts the all-in-focus image into an image focused on the specific region. In this case, the usability of the all-in-focus image can be improved by converting the all-in-focus image into an image focused on only the specific region selected by the user.

[0021]

[12] The all-in-focus image generating device according to any one of [7] to

[11] , wherein the frequency decomposition unit generates edge images for each frequency band in descending order of frequency, calculates image similarity between edge images for each focus position that share the same frequency band, and stops further frequency decomposition when the image similarity satisfies a predetermined similarity condition. When the image similarity satisfies the predetermined similarity condition, it is considered that there is no longer a significant difference in pixel values ​​between an image that includes a pixel corresponding to the focus position and an image that does not include a pixel corresponding to the focus position. Therefore, the frequency band can be determined to be an unnecessary frequency band for determining the focus level. Therefore, by stopping further frequency decomposition, the speed of generating the all-in-focus image can be improved without sacrificing accuracy.

[0022]

[13] An all-in-focus image generation program that causes a computer to execute an original image acquisition step of capturing an image of an object while shifting the focus position in a predetermined direction and acquiring an original image for each focus position of the object; a frequency decomposition step of frequency-decomposing the original image for each focus position and generating an edge image for each frequency band; an all-in-focus edge image generation step of integrating the edge images for each frequency band in the predetermined direction and generating an all-in-focus edge image for each frequency band; and an all-in-focus image generation step of integrating the all-in-focus edge images for each frequency band to generate an all-in-focus image.

[0023] The above-described all-in-focus image generation program acquires original images for each focus position, which are offset from one another, and generates edge images for each frequency band by extracting frequency components contained in each original image for each focus position. Because the frequency components contain information indicating the focus position in each original image, frequency decomposition of each original image allows for accurate extraction of the focus position in the edge image for each frequency band. The all-in-focus images generated for each frequency band by integrating the edge images for each frequency band reflect information indicating the different focus positions for each frequency band, and all pixels are in focus. This method allows for highly accurate generation of all-in-focus images without extracting specific regions from the original images for each focus position.

[0024] According to the present disclosure, an omnifocus image can be generated with high accuracy without extracting regions from the original image.

[0025] 1. A block diagram showing an all-in-focus image generating device according to an embodiment. FIG. 2 is a diagram showing an example of an original image for each focus position acquired by the original image acquisition unit shown in FIG. 1. FIG. 3 is a flowchart showing an all-in-focus image generating method according to an embodiment. FIG. 4 is a diagram for explaining the all-in-focus image generating method shown in FIG. 3. FIG. 5 is a flowchart showing an example of the frequency decomposition step shown in FIG. 5. FIG. 6 is a flowchart showing an example of a method for generating an all-in-focus edge image and an all-in-focus bottom layer image for each frequency band. FIG. 7 is a diagram for explaining the frequency decomposition step according to a first modified example. FIG. 8 is a flowchart showing an all-in-focus image generating method according to a second modified example. FIG. 9 is a diagram for explaining the selection step shown in FIG. 10. FIG. 11 is a flowchart showing an all-in-focus image generating method according to a third modified example. FIG. 12 is a diagram for explaining the conversion step shown in FIG. 11. FIG. 13 is a flowchart showing the frequency decomposition step according to a fourth modified example. FIG. 14 is a diagram for explaining the frequency decomposition step shown in FIG. 13. FIG. 15 is a diagram showing an example of application of an image information determination model according to a fifth modified example. FIG. 26 is a block diagram showing an example of an all-in-focus image generating program and a recording medium therefor. FIG. 27 is a diagram for explaining an all-in-focus image generating method using patches according to Comparative Example 1 and Comparative Example 2. FIG. 28 is a diagram showing an evaluation target sample. FIG. 29 is a diagram showing results of calculating a first evaluation index and a second evaluation index. FIG. 10 is a diagram showing the results of measuring the processing time required from acquiring an original image for each focus position to generating an all-in-focus image.

[0026] Hereinafter, preferred embodiments of an omnifocus image generating method, an omnifocus image generating device, and an omnifocus image generating program according to an embodiment of the present disclosure will be described in detail with reference to the drawings.

[0027] [All-in-focus image generating device] Figure 1 is a block diagram showing an all-in-focus image generating device according to one embodiment. The all-in-focus image generating device 1 generates an all-in-focus image in which each pixel is in focus from an original image (a plurality of original images) acquired for each focal position. The all-in-focus image generating device 1 is configured as a device that acquires original images for each focal position by capturing images of a cell sample placed on a glass slide, for example, and generates an all-in-focus image of the cell sample. The generated all-in-focus image is used, for example, for cytological diagnosis and pathological diagnosis.

[0028] The omnifocal image generating device 1 includes an original image acquisition unit 11, a frequency decomposition unit 12, an omnifocal edge image generation unit 13, an omnifocal image generation unit 14, and an omnifocal image output unit 15. The frequency decomposition unit 12, the omnifocal edge image generation unit 13, the omnifocal image generation unit 14, and the omnifocal image output unit 15 are physically computers including a processor such as a CPU, and storage media such as RAM and ROM. The computer may be a smart device such as a smartphone or tablet terminal that is integrated with a display unit and an input unit. The computer may be configured with a microcomputer or FPGA (Field-Programmable Gate Array).

[0029] The original image acquisition unit 11 acquires an original image for each focal position. The original image acquisition unit 11 includes an imaging device. The imaging device is, for example, a microscope imaging device, and enlarges and captures an image of an object placed on a stage. FIG. 2 is a diagram showing an example of an original image for each focal position acquired by the original image acquisition unit 11. The original image acquisition unit 11 acquires an original image 2 of a thick object. Here, for convenience of explanation, the thickness direction of the object is defined as the Z-axis direction. The direction perpendicular to the Z-axis direction is defined as the X-axis direction, and the direction perpendicular to the Z-axis direction and the X-axis direction is defined as the Y-axis direction. The original image acquisition unit 11 fixes the position (position in the X-axis and Y-axis directions) of the same object during imaging other than the Z-axis direction, and continuously images the object multiple times while shifting the focal position in the Z-axis direction (a predetermined direction). In this way, the original image acquisition unit 11 acquires original images 2 for each focal position with different focal positions. Because the original images 2 for each focal position are continuous in the Z-axis direction, they are referred to as Z-stack images.

[0030] The frequency decomposition unit 12 generates an edge image (multiple edge images) for each frequency band by frequency-resolving each of the original images 2 for each focal position. The original images 2 for each focal position are input to the frequency decomposition unit 12 from the original image acquisition unit 11. Each of the original images 2 for each focal position contains frequency components that indicate the strength of the edge at the focal position. The frequency decomposition unit 12 generates an edge image for each frequency band by extracting the frequency components contained in the original image 2 in order from the highest frequency band.

[0031] The omnifocal edge image generation unit 13 integrates the edge images for each frequency band in the Z-axis direction to generate an omnifocal edge image for each frequency band. The edge images for each frequency band are input to the omnifocal edge image generation unit 13 from the frequency decomposition unit 12. The omnifocal edge image generation unit 13 integrates pixels indicating focal positions from edge images for each focal position (plurality of edge images having a common frequency band) among the edge images for each frequency band in the Z-axis direction (a predetermined direction) to generate an omnifocal edge image for each frequency band.

[0032] The omnifocal image generation unit 14 generates a single omnifocal image by integrating the omnifocal edge images for each frequency band. The omnifocal edge images for each frequency band are input to the omnifocal image generation unit 14 from the omnifocal edge image generation unit 13. The omnifocal image generation unit 14 outputs the generated omnifocal image to the omnifocal image output unit 15.

[0033] For example, the omnifocal image output unit 15 displays the omnifocal image generated by the omnifocal image generation unit 14 on an external display (not shown). Alternatively, the omnifocal image output unit 15 transfers the generated omnifocal image to an external recording medium or device.

[0034] [All-in-focus image generating method] Next, an all-in-focus image generating method will be described with reference to Fig. 3 and Fig. 4. Fig. 3 is a flowchart showing an all-in-focus image generating method (hereinafter, method MT1) according to one embodiment. Fig. 4 is a diagram for explaining the all-in-focus image generating method.

[0035] The method MT1 starts with the original image acquisition unit 11 capturing the original image 2 for each focal position. First, the original image acquisition unit 11 acquires the original image 2 for each focal position (original image acquisition step: step ST1).

[0036] Next, the frequency decomposition unit 12 performs frequency decomposition on the original image 2 for each focal position, thereby generating an edge image for each frequency band (frequency decomposition step: step ST2). The frequency decomposition unit 12 extracts frequency components indicating the strength of the edge at the focal position included in each original image 2, in order from the highest frequency band. In other words, the frequency decomposition unit 12 extracts high frequency components in order by repeatedly filtering each original image 2. The edge image for each frequency band includes, in each frequency band, edge images for each focal position that share a common frequency band.

[0037] In step ST2, the frequency decomposition unit 12 decomposes the original image 2 for each focus position into a plurality of color component images and frequency-decomposes each of the plurality of color component images. The plurality of color component images may be composed of, for example, a red image, a green image, and a blue image. The frequency decomposition unit 12 generates an edge image for each frequency band for each of the plurality of color component images.

[0038] 4, step ST2 generates edge images 31 to 3(N−1) for each frequency band and a bottom layer image 3N (multiple bottom layer images) for each focus position, where N is an integer equal to or greater than 2.

[0039] 4 , the frequency decomposition unit 12 performs frequency decomposition on each original image 2 N times, thereby generating (N−1) edge images for each frequency band and one bottom-layer image for each original image 2. The (N−1) edge images for each frequency band correspond to the respective frequency bands. The (N−1) edge images for each frequency band are composed of, in descending order of frequency band, an edge image 31 generated by the first frequency decomposition, an edge image 32 generated by the second frequency decomposition, an edge image 33 generated by the third frequency decomposition, and an edge image 3(N−1) generated by the (N−1)th frequency decomposition.

[0040] Each original image 2 may include a plurality of in-focus points at the focus position. In this case, in step ST2, the frequency decomposition unit 12 extracts the plurality of in-focus points in descending order of the degree of in-focus, for example.

[0041] The edge images 31 to 3(N-1) for each frequency band include edge images for each focal position that are common to the frequency band. The edge images 31 for each focal position, the edge image 32 for each focal position, the edge image 33 for each focal position, the edge image 3(N-1) for each focal position, and the bottom-most image 3N for each focal position are generated in the same number as the original images 2 for each focal position. The edge images 31 to 3(N-1) for each frequency band and the bottom-most image 3N for each focal position have different focal positions among the multiple images in each frequency band.

[0042] As described above, the frequency decomposition unit 12 generates an edge image for each frequency band in each of the multiple color component images in the original image 2 for each focus position. As a result, each edge image 31 for each focus position includes a red image 31r, a green image 31g, and a blue image 31b. Each edge image 32 for each focus position includes a red image 32r, a green image 32g, and a blue image 32b. Each edge image 33 for each focus position includes a red image 33r, a green image 33g, and a blue image 33b. Each edge image 3(N-1) for each focus position includes a red image 3(N-1)r, a green image 3(N-1)g, and a blue image 3(N-1)b.

[0043] 4, the red image 3Nr, the green image 3Ng, and the blue image 3Nb are not shown in the bottom layer image 3N for each focal position. This is because no edge image is generated from the bottom layer image 3N for each focal position, and the bottom layer image 3N is distinguished from the edge images 31 to 3(N−1) for each frequency band, and the bottom layer image 3N for each focal position includes the red image 3Nr, the green image 3Ng, and the blue image 3Nb.

[0044] An example of step ST2 will now be described with reference to Figures 5 and 6. In the example of Figures 5 and 6, the frequency decomposition unit 12 performs frequency decomposition on each original image 2 using a Laplacian pyramid. First, the frequency decomposition unit 12 compresses the resolution of each original image 2, using each original image 2 as an original image, to half (step ST21).

[0045] Next, the frequency decomposition unit 12 resizes the compressed image 21 (step ST22). Because the resolution of the resized image 22 is the same as that of the image 21, the high-frequency components have been removed from each of the original images 2, resulting in an image that is blurred compared to each of the original images 2. Next, the frequency decomposition unit 12 subtracts the image 22 from each of the original images 2 to generate an edge image 31 in which only the high-frequency components of each of the original images 2 have been extracted (step ST23).

[0046] Next, the frequency decomposition unit 12 determines whether the number of edge images for each frequency band generated from each original image 2 has reached a predetermined number (step ST24). In the present embodiment, the predetermined number is (N-1). If the number of edge images for each frequency band has not reached the predetermined number (step ST24: NO), the frequency decomposition unit 12 uses image 21 as the original image and performs frequency decomposition again from step ST21. The frequency decomposition unit 12 repeats step ST21 N times, and steps ST22 and ST23 (N-1) times.

[0047] When the number of edge images for each frequency band reaches a predetermined number (step ST24: YES), the frequency decomposition unit 12 ends the frequency decomposition (step ST25). In this embodiment, the frequency decomposition unit 12 ends the frequency decomposition when (N-1) edge images for each frequency band have been generated.

[0048] 3 and 4 again, after the frequency decomposition unit 12 generates the edge images 31 to 3(N-1) for each frequency band and the bottom layer image 3N for each focal position, the omnifocal edge image generation unit 13 generates omnifocal edge images 41 to 4(N-1) for each frequency band (plurality of omnifocal edge images) and an omnifocal bottom layer image 4N from the edge images 31 to 3(N-1) for each frequency band and the bottom layer image 3N for each focal position (omnifocal edge image generation step: step ST3).

[0049] In step ST3, the omni-focus edge image generation unit 13 generates an omni-focus edge image for each frequency band in each of the multiple color component images. As a result, the omni-focus edge image 41 includes an omni-focus red image 41r, an omni-focus green image 41g, and an omni-focus blue image 41b. The omni-focus edge image 42 includes an omni-focus red image 42r, an omni-focus green image 42g, and an omni-focus blue image 42b. The omni-focus edge image 43 includes an omni-focus red image 43r, an omni-focus green image 43g, and an omni-focus blue image 43b. The omni-focus edge image 4(N-1) includes an omni-focus red image 4(N-1)r, an omni-focus green image 4(N-1)g, and an omni-focus blue image 4(N-1)b.

[0050] The omnifocal edge images 41 to 4(N-1) for each frequency band are composed of (N-1) images. The (N-1) omnifocal edge images for each frequency band correspond to each frequency band. Each omnifocal edge image 41 to 4(N-1) is formed by integrating pixels of the edge image in the corresponding frequency band. The (N-1) omnifocal edge images for each frequency band are composed of omnifocal edge image 41, omnifocal edge image 42, omnifocal edge image 43, and omnifocal edge image 4(N-1), in descending order of frequency band.

[0051] The all-focus bottom-layer image 4N is composed of one image and is formed by integrating pixels of the bottom-layer image 3N for each focus position.

[0052] 7 is a diagram showing an example of a method for generating omni-focus edge images 41 to 4(N-1) for each frequency band and omni-focus bottom layer image 4N. In this embodiment, a maximum value extraction method will be described. Step ST3 includes steps ST31 to ST33.

[0053] First, the omnifocal edge image generation unit 13 extracts a pixel showing the maximum pixel value at each pixel position of each of the omnifocal edge images 41 to 4(N-1) and the omnifocal bottom layer image 4N from the edge image for each focus position among the edge images 31 to 3(N-1) for each frequency band and the bottom layer image 3N for each focus position (step ST31). For example, when generating the omnifocal edge image 41, the omnifocal edge image generation unit 13 extracts a pixel with the highest pixel value from the edge image 31 for each focus position at each pixel position of the omnifocal edge image 41. The pixel with the highest pixel value is the in-focus pixel in the edge image 31 for each focus position.

[0054] Next, the omni-focus edge image generation unit 13 generates an omni-focus edge image by integrating the pixels extracted at all pixel positions of each of the omni-focus edge images 41 to 4(N-1) and the omni-focus bottom-layer image 4N (step ST32). Each of the omni-focus edge images 41 to 4(N-1) is an image in focus at all pixel positions. For example, the omni-focus edge image generation unit 13 integrates the pixel values ​​of the edge image 31 for each focus position extracted at all pixel positions of the omni-focus edge image 41 to generate an omni-focus edge image 51.

[0055] Next, the omnifocus image generation unit 14 generates one omnifocus image from each of the omnifocus edge images 41 to 4(N-1) and the omnifocus bottommost image 4N (omnifocus image generation step: step ST4). The omnifocus image generation unit 14 generates the omnifocus image 5, for example, by the following procedure. The omnifocus image generation unit 14 generates an image with double the resolution of the omnifocus bottommost image 4N. Then, the omnifocus image generation unit 14 adds the image with double the resolution of the omnifocus bottommost image 4N and the omnifocus edge image 4(N-1). The omnifocus image generation unit 14 repeats these procedures until the omnifocus edge image 41 is added, thereby generating the omnifocus image 5.

[0056] Finally, the all-in-focus image output unit 15 outputs the all-in-focus image 5 (ST5). With the output of the all-in-focus image 5, the method MT is completed.

[0057] As described above, method MT1 according to one aspect of the present disclosure involves acquiring original images 2 for each focus position, each of which is offset from the others, and extracting the frequency components contained in each original image 2 for each focus position for each frequency band to generate edge images 31-3(N-1) for each frequency band and a bottom-most image 3N for each focus position. Because the frequency components contain information indicating the focus position in each original image 2 (edge ​​strength at the focus position), frequency decomposition of each original image 2 allows for accurate extraction of the focus position in the edge images 31-3(N-1) for each frequency band and the bottom-most image 3N for each focus position. The all-focus image 5 generated for each frequency band by integrating the edge images 31-3(N-1) for each frequency band and the bottom-most image 3N for each focus position reflects information indicating the different focus positions for each frequency band, and all pixels are in focus. This method allows for highly accurate generation of all-focus images 5 without extracting specific regions from the original image 2.

[0058] In step ST2, each of the original images 2 for each focal position is decomposed into a plurality of color component images, and each of the plurality of color component images is frequency-decomposed to generate a plurality of edge images for each frequency band corresponding to the plurality of color component images. In this case, by frequency-decomposing each of the plurality of color component images to generate an edge image for each frequency band corresponding to the plurality of color component images, information indicating a different focal position for each color component image can be further reflected in the omnifocus image 5. This allows the omnifocus image 5 to be generated with even greater accuracy.

[0059] [Modifications] Although the embodiments of the present disclosure have been described above, the present disclosure is not limited to the above-described embodiments.

[0060] [First Modification] Step ST2 may include a step of excluding original images that do not include a focal position. FIG. 8 is a diagram for explaining the step of excluding original images that do not include a focal position. The frequency decomposition unit 12 excludes original images 2A that do not include a focal position at all pixel positions from the original images 2 for each acquired focal position. In the example of FIG. 8, there are multiple original images 2A that do not include a focal position. The original image 2 in the top layer, the original image 2 in the layer immediately below the top layer, the original image 2 in the bottom layer, and the original image 2 in the layer immediately above the bottom layer are excluded as original images 2A that do not include a focal position. There may be only one original image 2A that does not include a focal position.

[0061] The original image acquisition unit 11 may calculate a variance value as an index for the frequency decomposition unit 12 to exclude original images 2A that do not contain a focal position. The frequency decomposition unit 12, for example, calculates the deviation of pixel values ​​for all pixels of each original image 2 and calculates the variance value of each original image 2 from the calculated deviation. Original images 2A that do not contain a focal position tend to have small variance values. If the variance value of each original image 2 is below a variance threshold, the frequency decomposition unit 12 excludes the corresponding original image as an original image 2A that does not contain a focal position. Alternatively, the frequency decomposition unit 12 may perform edge detection using first or second derivatives as an index for excluding original images 2A that do not contain a focal position. The frequency decomposition unit 12 may exclude original images 2 in which no edges are detected as a result of edge detection as original images 2A that do not contain a focal position.

[0062] In step ST2, the frequency decomposition unit 12 may exclude original images 2A that do not include the focal position from the original images 2 for each focal position. In this case, by excluding the original images 2A that do not include the focal position, the number of images to be decomposed in step ST2 can be reduced, and the speed at which the omnifocus image 5 is generated can be improved.

[0063] [Second Modification] FIG. 9 is a flowchart showing an all-in-focus image generating method (hereinafter, method MT2) according to a second modification. Method MT2 differs from method MT1 in that it includes a step of selecting a specific region (step ST6) between steps ST1 and ST2. FIG. 10 is a diagram illustrating step ST6. As shown in FIG. 10, the original image acquisition unit 11 selects a specific region R, on which the user wants to focus, from each of the acquired original images 2 for each focus position, based on, for example, user input via a user interface. The specific region R can be arbitrarily selected by the user, for example. The outer edge of the specific region R may be curved or straight. The specific region R is common to all of the original images 2 for each focus position.

[0064] In method MT2, steps from step ST2 onwards are performed only on the specific region R. In other words, they are performed only on the pixels included in the specific region R. In step ST2, the specific region R in each of the original images 2 for each focal position is frequency decomposed to generate edge images 31 to 3(N-1) for each frequency band and a bottom-most image 3N for each focal position in the specific region R in each of the original images 2 for each focal position.

[0065] In step ST3, the omnifocal edge image generation unit 13 generates omnifocal edge images 41 to 4(N-1) for each frequency band and an omnifocal bottom layer image 4N in the specific region R (step ST3). In step ST4, the omnifocal image generation unit 14 generates an omnifocal image 5 that is focused only on the specific region R. In the omnifocal image 5 that is focused only on the specific region R, all pixels included in the specific region R are in focus, while regions other than the specific region R are out of focus.

[0066] In the method MT2, step ST6 does not necessarily have to be performed by the original image acquisition unit 11. For example, the omnifocus-image generation device 1 may further include a selection unit for selecting the specific region R, and the selection unit may select the specific region R based on, for example, a user input from a user interface.

[0067] Method MT2 may further include step ST6 (selection step) of selecting a specific region R in the original image 2 for each focus position. In step ST2, the specific region R in each of the original images 2 for each focus position may be frequency-decomposed to generate edge images 31 to 3(N-1) for each frequency band in the specific region R and a bottom-most image 3N for each focus position for each frequency band. In this case, the amount of data of the image to be decomposed in step ST2 can be reduced, thereby improving the speed at which the all-focus image 5 is generated. Note that the specific region R here is simply a region selected to reduce the amount of data, and the selected specific region R does not affect the accuracy of the all-focus image 5 generated by this method.

[0068] [Third Modification] FIG. 11 is a flowchart showing an all-in-focus image generating method according to a third modification (hereinafter, method MT3). Method MT3 differs from method MT1 in that it includes, after step ST4, a step (step ST7) of converting the all-in-focus image 5 into an image focused on a specific region. FIG. 12 is a diagram illustrating step ST7. In step ST7, as shown in FIG. 12, the all-in-focus image generating unit 14 selects a specific region R to be focused on from the generated all-in-focus image 5 based on, for example, user input via a user interface. The specific region R can be arbitrarily selected by, for example, the user. The outer edge of the specific region R may be curved or straight.

[0069] Next, in step ST7, the omnifocus image generation unit 14 converts the omnifocus image 5 into an image 5R in which only the selected specific region R is in focus. In the image 5R, all pixels included in the specific region R are in focus, while regions other than the specific region R are out of focus.

[0070] In step ST7, the omnifocus image generation unit 14 may convert the omnifocus image 5 into an image 5R by reducing the resolution of an area other than the specific area R in the omnifocus image 5. The omnifocus image generation unit 14 may convert the omnifocus image 5 into an image 5R by cutting out the area other than the specific area R from any of the original images 2 for each focus position or an image obtained by averaging the pixel values ​​of the original images 2 for each focus position, and applying the cut-out area to the omnifocus image 5.

[0071] In the method MT3, step ST7 does not necessarily have to be performed by the all-in-focus-image generation unit 14. For example, the all-in-focus-image generation device 1 may further include a conversion unit that converts the all-in-focus image 5 into an image focused on a specific region.

[0072] The method MT3 may further include step ST7 (conversion step) of selecting a specific region R in the all-focus image 5 and converting the all-focus image 5 into an image 5R focused on the specific region R. In this case, the usability of the all-focus image 5 can be improved by being able to convert the all-focus image 5 into an image 5R focused on only the specific region R selected by the user.

[0073] [Fourth Modification] Fig. 13 is a flowchart showing step ST2A according to the fourth modification. Step ST2A differs from step ST2 in that it includes step ST26 instead of step ST24. In step ST26, the frequency decomposition unit 12 determines whether the image similarity between edge images 31 to 3(N-1) for each frequency band, at each focus position, that share a common frequency band, satisfies a similarity threshold. In step ST2A, the frequency decomposition unit 12 calculates the image similarity between the edge images for each focus position, and if the image similarity satisfies a predetermined similarity condition, stops further frequency decomposition.

[0074] 14 , frequency decomposition is performed three times to generate an edge image 31 for each focal position, an edge image 32 for each focal position, and an edge image 33 for each focal position. The frequency decomposition unit 12 calculates the image similarity in the edge image 33 for each focal position, and if the image similarity satisfies a predetermined similarity condition (step ST26: YES), stops frequency decomposition on the edge image 33 for each focal position and thereafter.

[0075] When the image similarity between edge images for each focus position that share a common frequency band satisfies a predetermined similarity condition, it is considered that there is no significant difference in pixel values ​​between an image that includes pixels corresponding to the focus position and an image that does not include pixels corresponding to the focus position. Therefore, the frequency band can be determined to be an unnecessary frequency band for determining the focus level. In other words, the image similarity satisfying the predetermined similarity condition serves as a judgment index indicating that further frequency decomposition will not contribute to improving the accuracy of the omnifocus image.

[0076] An example of a method for calculating image similarity between edge images at each focus position that share a common frequency band is SSIM (Structural SIMilarity). The calculation formula for similarity using SSIM is shown in Equation 1, for example. In Equation 1, the number of edge images for each focal position is L (L is an integer equal to or greater than 1). If the Mth edge image (M is an integer equal to or greater than 1 and equal to or less than L) is set as a variable x[M] and the (M+1)th edge image is set as a variable y[M+1], the SSIM between the Mth edge image and the (M+1)th edge image is calculated, and the calculated results are summed to calculate the similarity DS.

[0077] SSIM consists of multiplication of comparison terms of brightness (l(x,y)), contrast (c(x,y)), and structure (s(x,y)) of the edge image. The definition of SSIM is shown in Equation 2. In Equation 2, α, β, and γ are parameters for adjusting the weights of three comparison terms. The brightness (l(x,y)), contrast (c(x,y)), and structure (s(x,y)) are expressed by Equations 3 to 5, respectively. μｘ , μ ｙ is the average value of x[M], y[M+1], and σ ｘ , σ y is the standard deviation of x[M], y[M+1], and σ ｘ σ y is the covariance of x[M] and y[M+1]. １ ~C ３ is a constant provided to improve the instability of the output value when the denominator is close to 0 in Equations 3 to 5.

[0078] When α, β, and γ=1 in Equation 2 and C3=C2 / 2 in Equations 4 and 5, SSIM is calculated by Equation 6 from Equations 2 to 5.

[0079] Another example of calculating the image similarity between edge images at each focus position that share a common frequency band is the peak signal-to-noise ratio (PSNR). For example, when the sum of the differences between the PSNRs of the Mth edge image and the (M+1)th edge image between the edge images at each focus position is below a predetermined difference threshold, the frequency decomposition unit 12 stops frequency decomposition for the frequency band below the difference threshold and subsequent frequency bands.

[0080] Another example of calculating the image similarity is, for example, a variance value. For example, when the sum of the differences between the variance values ​​of the Mth edge image and the (M+1)th edge image between edge images at each focus position is below a predetermined difference threshold, the frequency decomposition unit 12 stops frequency decomposition for the frequency band below the difference threshold and subsequent frequency bands.

[0081] In step ST2A, the image similarity between edge images for each focal position that share a common frequency band is calculated. If the image similarity satisfies a predetermined similarity condition, subsequent frequency decomposition may be stopped. If the image similarity satisfies the predetermined similarity condition, it is considered that there is no longer a significant difference in pixel values ​​between an image that includes a pixel corresponding to the focal position and an image that does not include a pixel corresponding to the focal position. Therefore, the frequency band can be determined to be unnecessary for determining the focus level. Therefore, by stopping subsequent frequency decomposition, the speed at which the omnifocus image 5 is generated can be improved without sacrificing accuracy.

[0082] [Fifth Modification] The all-in-focus image generation methods according to the above-described embodiments and modifications may be applied to machine learning as a fifth modification. For example, the all-in-focus image generation device 1 may further include a model generation unit, and the image information determination model may be generated in the model generation unit. FIG. 15 is a diagram illustrating an example of an application of the image information determination model according to the fifth modification. As shown in FIG. 15, the input data of the image information determination model 10 may be the original image 2 for each focus position, edge images 31 to 3(N-1) for each frequency band, and the bottom-most image 3N for each focus position, or the all-in-focus image 5. In the example of FIG. 15, the output data of the image information determination model 10 is an image 5R in which only a specific region R selected from the all-in-focus image 5 is in focus. In this case, the specific region R may be automatically selected by the image information determination model 10, rather than being selected by a user.

[0083] As another example of output data from the image information determination model 10, a new original image 2 or all-focus image 5 for each focus position may be output. Alternatively, when an image for cytological diagnosis is prepared as input data, image data reflecting, for example, the detection of the position of a specific cell, the estimation of the cell type, and the results of staining the cell according to the cell type and state may be output. Furthermore, the image information determination model 10 is not limited to outputting image data, and may, for example, determine whether or not to execute the method MT again.

[0084] In the above-described embodiment and each modification, in step ST2 or ST2A, the frequency decomposition unit 12 may perform frequency decomposition on each original image 2 using a method other than the Laplacian pyramid. Examples of other frequency decomposition methods include wavelet transform, discrete Fourier transform, fast Fourier transform, filtering using a Gaussian difference filter or kernel, and Hilbert transform.

[0085] In the above-described embodiment and each modified example, in step ST3, the omni-focus edge image generation unit 13 may generate the omni-focus edge images 41 to 4(N-1) for each frequency band and the omni-focus bottom layer image 4N using a method other than the maximum value extraction method. The omni-focus edge image generation unit 13 may, for example, perform principal component analysis. In this case, for example, in the principal component analysis, when the edge image 31 for each focus position shown in FIG. 4 is used as a variable, each variable may be multiplied by a weight constant to create a composite variable, and the value at which the variance of the composite variable is maximized may be extracted as a principal component. Furthermore, the omni-focus edge image 41 may be constructed using the extracted principal components.

[0086] In addition to the maximum value extraction method, the omnifocal edge image generation unit 13 may perform weighted addition, average calculation, outlier removal, and neighbor point calculation. Furthermore, the omnifocal edge image generation unit 13 may perform maximum value extraction by waveform fitting. The omnifocal edge image generation unit 13 may convert the edge images 31 to 3(N-1) for each frequency band and the bottom-most image 3N for each focus position into grayscale, and then perform maximum value extraction.

[0087] 16 is a block diagram showing an example of an omnifocus image generation program and its recording medium. In the example shown in FIG. 16, the omnifocus image generation program 6 includes a main module 61, an original image acquisition module 62, a frequency decomposition module 63, an omnifocus edge image generation module 64, an omnifocus image generation module 65, and an omnifocus image output module 66. Functions realized by a computer through execution of the omnifocus image generation program 6 are similar to the functions of the omnifocus image generation device 1 described above. The omnifocus image generation program 6 is provided by a computer-readable recording medium 7, such as a CD-ROM, DVD, or ROM. The omnifocus image generation program 6 may be provided by a semiconductor memory or as a computer data signal superimposed on a carrier wave via a network.

[0088] Examples of the present disclosure will be described below. In these examples, all-in-focus images were generated using the all-in-focus image generation methods of Example 1, Example 2, Comparative Example 1, and Comparative Example 2. Then, for each generated all-in-focus image, evaluation was performed using a first evaluation index and a second evaluation index to determine whether or not the focus was uniform across all pixels. Furthermore, in Example 1, Example 2, Comparative Example 1, and Comparative Example 2, the processing time required from acquiring an image for each focus position to generating an all-in-focus image was measured.

[0089] In Example 1, an all-focus image was generated according to method MT1 shown in Fig. 3. However, in step ST3, for generating each all-focus edge image, a pixel showing the maximum pixel value at each pixel position of each all-focus edge image was extracted from the edge image for each focus position, and the extracted pixels were integrated to generate the all-focus edge image. On the other hand, for generating the all-focus bottom layer image, a pixel showing the minimum pixel value at each pixel position of the all-focus bottom layer image was extracted from the bottom layer image for each focus position, and the extracted pixels were integrated to generate the all-focus bottom layer image.

[0090] In Example 2, in step ST3 of the method MT1 shown in FIG. 3, an all-in-focus image is generated by principal component analysis instead of maximum value extraction.

[0091] In Comparative Examples 1 and 2, an all-in-focus image generation method using a patch was used instead of Method MT1. A patch is, for example, a partial area of ​​an image composed of 16 pixels by 16 pixels. As shown in FIG. 17 , in Comparative Examples 1 and 2, a patch P is scanned along a plane defined by the X-axis and Y-axis for each original image. In Comparative Example 1, a focal position is extracted by first differentiation for each position of the patch P. In Comparative Example 2, a focal position is extracted by second differentiation for each position of the patch P. The focal position is extracted pixel by pixel within the patch P. In Comparative Examples 1 and 2, an all-in-focus image is generated by combining the pixels from which the focal positions have been extracted.

[0092] The first evaluation index is Q Ｓ (Structural similarity based metrics). Ｓ In the evaluation by Q, an all-in-focus image is divided into multiple images by a window, and the image similarity between the divided images is evaluated. Ｓ is calculated by Equation 7. Here, W and w represent a sliding window, Q(x, f|w) represents a local quality index between variables x and f within window w, and Q(y, f|w) represents a local quality index between variables y and f within window w. λ(w) represents a weighting coefficient, which determines the weighting of Q(x, f|w) and Q(y, f|w). λ(w) is calculated using Equation 8.

[0093] The second evaluation index is Q ＣＢ (Human perception based metrics). ＣＢ In the evaluation by Q, the all-in-focus image is divided into multiple images by windows. ＣＢ In the evaluation by [Equation 9], the similarity of contrast in each divided image is calculated, and finally the similarity of contrast in the omnifocus image is evaluated. First, the local contrast is calculated by Equation 9. where φ ｋ is a Gaussian distribution function and is calculated by Equation 10.

[0094] Next, the local contrast calculated by Equation 4 is adjusted by a hyperparameter that reflects the degree of recognition by human vision. The adjusted contrast C Ａ ' is calculated by Equation 11. where t, h, p, q, and Z are real scalar parameters that determine the nonlinear shape of the masking function, e.g., t=1, h=1, p=3, q=2, and Z=0.0001.

[0095] Next, the weighting coefficient λ Ａ (x, y) is calculated by Equation 12. Next, the contrast similarity Q ＡＦ (x, y) is calculated by Equation 13. Contrast similarity Q in separate images ＢＦ (x, y) is also calculated by Equation 8. Then, a global quality map Q is calculated based on the weighting coefficients calculated by Equations 12 and 13 and the contrast similarity. ＧＱＭ (x, y) is calculated by Equation 14. Finally, the contrast similarity in all in-focus images is denoted as a global quality map Q ＧＱＭ It is calculated by averaging (x, y) using Equation 15.

[0096] The samples to be evaluated are multiple images equivalent to the original image 2 for each focal position in Fig. 3. As shown in Fig. 18, each of samples A to G is a multiple of images for cytological diagnosis. Each of samples A to G differs in the number of images and the size of each image.

[0097] FIG. 19 shows the first evaluation index Q ｓ and the second evaluation index, Q ＣＢ The calculated results are shown below. ｓ and Q ＣＢ A larger value of indicates more uniform focus across the pixels and better quality of the all-in-focus image.

[0098] The first evaluation index is Q ｓ Regarding Q calculated in Example 2,ｓ is the Q calculated in Comparative Example 1 and Comparative Example 2 ｓ It can be seen that the second evaluation index, Q, is larger than the ＣＢ Regarding Q calculated in Example 1, ＣＢ is the Q calculated in Comparative Example 2 ＣＢ It can be seen that the values ​​are larger in all samples than those calculated in Example 1. ＣＢ is the Q calculated in Comparative Example 1 ＣＢ It can be seen that the Q calculated in Example 1 is larger in Samples A, B, and Samples D to F than in ＣＢ In sample C, the Q calculated in comparative example 1 ＣＢ The Q calculated in Example 1 is the same as ＣＢ In sample G, the Q calculated in comparative example 1 ＣＢ However, the difference is insignificant.

[0099] 20 shows the results of measuring the processing time required from acquiring an image for each focal position to generating an all-in-focus image in Example 1, Example 2, Comparative Example 1, and Comparative Example 2. It can be seen that the processing time measured in Example 1 is shorter for all samples than the processing time measured in Comparative Examples 1 and 2. In particular, it can be seen that the processing time measured in Example 1 is shortened to 1 / 4 to 1 / 6 of the processing time measured in Comparative Examples 1 and 2 for Samples E and G, which have a large number of images.

[0100] 1...All-focus image generation device, 11...Original image acquisition unit, 12...Frequency decomposition unit, 13...All-focus edge image generation unit, 14...All-focus image generation unit, 2...Original image, 2A...Image not including focus position, 31, 32, 33, 3(N-1)...Edge image, 31r, 31g, 31b...Color component image, 41, 42, 43, 4(N-1)...All-focus edge image, 5...All-focus image, 5R...Image focused on specific region, 6...All-focus image generation program, MT1, MT2, MT3...All-focus image generation method, R...Specific region, ST2, ST2A...Frequency decomposition step, ST3...All-focus edge image generation step, ST4...All-focus image generation step, ST6...Selection step, ST7...Conversion step.

Claims

1. An all-focus image generation method comprising: an original image acquisition step of capturing an image of an object while shifting the focus position in a predetermined direction and acquiring an original image for each focus position of the object; a frequency decomposition step of frequency decomposing the original image for each focus position and generating an edge image for each frequency band; an all-focus edge image generation step of integrating the edge images for each frequency band in the predetermined direction and generating an all-focus edge image for each frequency band; and an all-focus image generation step of integrating the all-focus edge images for each frequency band to generate an all-focus image.

2. An all-focus image generating method as described in claim 1, wherein in the frequency decomposition step, the original image for each focus position is decomposed into a plurality of color component images, and each of the plurality of color component images is frequency decomposed to generate edge images for each of the frequency bands corresponding to the plurality of color component images.

3. The all-focus image generating method according to claim 1 or 2, wherein in the frequency decomposition step, images that do not include the focus position from among the original images for each focus position are excluded from the frequency decomposition.

4. A method for generating an all-focus image as described in any one of claims 1 to 3, further comprising a selection step of selecting a specific area in the original image for each focus position, wherein in the frequency decomposition step, the specific area in each of the original images for each focus position is frequency decomposed to generate an edge image for each frequency band in the specific area.

5. An all-focus image generating method according to any one of claims 1 to 4, further comprising a conversion step of selecting a specific area in the all-focus image and converting the all-focus image into an image focused on the specific area.

6. An all-in-focus image generating method according to any one of claims 1 to 5, wherein in the frequency decomposition step, an edge image for each frequency band is generated in ascending order of frequency band, an image similarity between edge images for each focus position having a common frequency band is calculated for the edge images for each frequency band, and when the image similarity satisfies a predetermined similarity condition, subsequent frequency decomposition is stopped.

7. An all-focus image generating device comprising: an original image acquisition unit that images an object while shifting the focus position in a predetermined direction and acquires an original image for each focus position of the object; a frequency decomposition unit that frequency-decomposes the original image for each focus position and generates an edge image for each frequency band; an all-focus edge image generation unit that integrates the edge images for each frequency band in the predetermined direction and generates an all-focus edge image for each frequency band; and an all-focus image generation unit that integrates the all-focus edge images for each frequency band to generate an all-focus image.

8. An all-focus image generating device as described in claim 7, wherein the frequency decomposition unit decomposes the original image for each focus position into a plurality of color component images, and frequency-decomposes each of the plurality of color component images to generate edge images for each of the frequency bands corresponding to the plurality of color component images.

9. The all-focus image generating device according to claim 7 or 8, wherein the frequency decomposition section excludes images that do not include the focus position from the original images for each focus position as targets for frequency decomposition.

10. An all-focus image generating device as described in any one of claims 7 to 9, further comprising a selection unit that selects a specific area in the original image for each focus position, wherein the frequency decomposition unit frequency-decomposes the specific area in each of the original images for each focus position, and generates an edge image for each frequency band in the specific area.

11. An all-focus image generating device according to any one of claims 7 to 10, further comprising a conversion unit that selects a specific area in the all-focus image and converts the all-focus image into an image focused on the specific area.

12. An all-in-focus image generating device as described in any one of claims 7 to 11, wherein the frequency decomposition unit generates edge images for each frequency band in ascending order of frequency, calculates image similarity between edge images for each focus position that have a common frequency band in the edge images for each frequency band, and stops further frequency decomposition when the image similarity satisfies a predetermined similarity condition.

13. An all-focus image generation program that causes a computer to execute the following steps: an original image acquisition step of imaging an object while shifting the focus position in a predetermined direction and acquiring an original image for each focus position of the object; a frequency decomposition step of frequency-decomposing the original image for each focus position and generating an edge image for each frequency band; an all-focus edge image generation step of integrating the edge images for each frequency band in the predetermined direction and generating an all-focus edge image for each frequency band; and an all-focus image generation step of integrating the all-focus edge images for each frequency band to generate an all-focus image.

Images