Image processing method, image processing device, imaging device, and computer-readable program

Abstract

A smooth all-focus image and an appropriate height map that matches the all-focus image are generated. [Solution] In this image processing method, multiple sharpness indices are calculated for each pixel position from multiple captured images acquired while changing the focus position along the optical axis, and the multiple sharpness indices for each pixel position are compared to determine an image reference value to be referenced in determining the luminance value of that pixel position in the all-focus image (steps S11 to S13). The luminance value of that pixel position in the all-focus image is calculated based on the image reference values ​​for each pixel position and a group of pixel positions including that pixel position and its surrounding pixel positions (step S14). A composite height for that pixel position is obtained based on the image reference values ​​and / or sharpness indices for each pixel position and a group of pixel positions including that pixel position and its surrounding pixel positions, and a height map is generated (step S15).

Description

[Technical Field]

[0001] The present invention relates to a technique for generating an all-focus image and a height map for use in analyzing the all-focus image. [Background technology]

[0002] A conventional technique used in microscopes involves capturing an image of an object while varying the focal position along the optical axis, acquiring multiple captured images, and combining the in-focus portions of each captured image to generate an image in which the entire object is in focus (i.e., an all-focus image). For example, in Patent Document 1, multiple captured images are captured while varying the focal position, and the sharpness values ​​in the multiple captured images are compared for each pixel position (coordinate), thereby determining an image reference value, which is the number of the captured image to be referenced as the luminance value of the all-focus image. Then, the luminance value of each pixel position in the all-focus image is calculated by reflecting the luminance values ​​of the captured images indicated by the image reference values ​​of surrounding pixel positions on the luminance value of the captured image indicated by the image reference value of the pixel position. Furthermore, in Patent Document 2, for each pixel position, a predetermined number of sharpness values ​​with the highest sharpness values ​​in multiple captured images (images with varying focal positions) are extracted as correction sharpness values, and a predetermined number of image reference values, which are the numbers of the captured images corresponding to the predetermined number of correction sharpness values, are determined. Then, based on the predetermined number of image reference values ​​and the predetermined number of correction sharpness factors, a luminance value of the pixel position in the omnifocus image is calculated.

[0003] In Patent Document 3, edges are detected as features from multiple different input images captured with different focal lengths, and the intensity of the edges is used to estimate the depth, which is information indicating which of the multiple input images the target pixel is in focus in. This generates a depth map that makes it possible to identify the input image in focus for the pixel at each coordinate, making it possible to generate an image in focus on a subject located at a desired height. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2018-42006 [Patent Document 2] Japanese Patent Publication No. 2022-51094 [Patent Document 3] Japanese Patent Application Laid-Open No. 2015-111822 Summary of the Invention [Problem to be solved by the invention]

[0005] Incidentally, the image reference value at each pixel position in Patent Document 1 can be interpreted as the focal position (focus position) of the object at that pixel position, i.e., the height of the object. In this case, the array of image reference values ​​for all pixel positions becomes a height map showing the distribution of the object's height. When analyzing an object using an all-in-focus image, it is conceivable to extract an area to be analyzed in the all-in-focus image and obtain, as a feature, the average height of that area using a height map. However, in the all-in-focus image generated in Patent Document 1, the luminance value of each pixel position is affected by the surrounding pixel positions, so the height map may differ from the actual all-in-focus image.

[0006] The present invention has been made in consideration of the above-mentioned problems, and aims to generate a smooth omnifocus image and an appropriate height map that matches the omnifocus image. [Means for solving the problem]

[0007] A first aspect of the present invention is an image processing method for generating an all-in-focus image and a height map used in analyzing the all-in-focus image, comprising: a) preparing a plurality of captured images obtained by capturing an image of an object while changing the focus position along an optical axis; b) calculating a plurality of sharpness indices from the plurality of captured images for each pixel position; c) comparing the sharpness indices for each pixel position to determine, as an image reference value, the number of at least one captured image to be referenced in determining the luminance value of each pixel position of the all-in-focus image; d) calculating the luminance value of each pixel position of the all-in-focus image based on the image reference value at a pixel position group including each pixel position and pixel positions surrounding each pixel position, and generating the all-in-focus image; and e) obtaining a composite height for each pixel position based on the image reference value and / or sharpness at each pixel position and generating a height map.

[0008] A second aspect of the present invention is the image processing method of the first aspect, wherein the group of pixel positions in the step d) is the same as the group of pixel positions in the step e).

[0009] Aspect 3 of the present invention is the image processing method of aspect 1 (which may be aspect 1 or 2), wherein in step e), the composite height for each pixel position is calculated by reflecting the image reference values ​​of the surrounding pixel positions on the image reference value of each pixel position with a weighting that increases the influence the higher the sharpness.

[0010] A fourth aspect of the present invention is an image processing method according to the first aspect (which may be any one of the first to third aspects), in which in the step e), the composite height for each pixel position is calculated by reflecting the image reference values ​​of the surrounding pixel positions on the image reference value of each pixel position with a weighting that increases the influence the closer the distance.

[0011] A fifth aspect of the present invention is an image processing method according to any one of the first to fourth aspects, further comprising the steps of identifying an area to be analyzed in the all-in-focus image and calculating a feature amount for the height of the area to be analyzed using the height map.

[0012] A sixth aspect of the present invention is an image processing device that generates an all-focus image and a height map used in analyzing the all-focus image, and includes: an image storage unit that stores a plurality of captured images obtained by capturing an image of an object while changing the focus position along the optical axis; a sharpness calculation unit that calculates a plurality of sharpness indices from the plurality of captured images for each pixel position; an image reference value determination unit that compares the plurality of sharpness indices for each pixel position to determine, as an image reference value, the number of at least one captured image to be referenced in determining the luminance value of each pixel position of the all-focus image; an all-focus image generation unit that calculates the luminance value of each pixel position of the all-focus image based on the image reference value in a pixel position group including each pixel position and pixel positions surrounding each pixel position, and generates the all-focus image; and a height map generation unit that obtains a composite height for each pixel position based on the image reference value and / or sharpness in a pixel position group including each pixel position and pixel positions surrounding each pixel position, and generates a height map.

[0013] A seventh aspect of the present invention is the image processing device of the sixth aspect, wherein the group of pixel positions in the omni-focus image generating unit and the group of pixel positions in the height map generating unit are the same.

[0014] Aspect 8 of the present invention is an image processing device of aspect 6 (which may also be aspect 6 or 7), in which the height map generation unit obtains the composite height for each pixel position by reflecting the image reference values ​​of the surrounding pixel positions on the image reference value of each pixel position with a weighting such that the higher the sharpness, the greater the influence.

[0015] A ninth aspect of the present invention is an image processing device according to aspect 6 (which may be any one of aspects 6 to 8), in which the height map generation unit acquires the composite height for each pixel position by reflecting the image reference values ​​of the surrounding pixel positions on the image reference value of each pixel position with a weighting such that the closer the distance, the greater the influence.

[0016] A tenth aspect of the present invention is an image processing device according to the sixth aspect (which may be any one of the sixth to ninth aspects), further comprising a target area identification unit that identifies an area to be analyzed in the all-in-focus image, and a feature amount calculation unit that calculates a feature amount for the height of the area to be analyzed using the height map.

[0017] Aspect 11 of the present invention is an imaging device comprising an image processing device of any one of aspects 6 to 10, an imaging unit that images the object, an illumination unit that emits light toward the object, and a focal position changing mechanism that changes the focal position of the imaging unit along the optical axis.

[0018] A twelfth aspect of the present invention is a computer-readable program that causes a computer to generate an all-focus image and a height map used in analyzing the all-focus image. Execution of the program by a computer causes the computer to perform the following steps: a) preparing a plurality of captured images obtained by capturing an image of an object while changing the focus position along the optical axis; b) calculating a plurality of sharpness indices from the plurality of captured images for each pixel position; c) comparing the sharpness indices for each pixel position to determine, as an image reference value, the number of at least one captured image to be referenced in determining the luminance value of each pixel position of the all-focus image; d) calculating the luminance value of each pixel position of the all-focus image based on the image reference values ​​at a group of pixel positions including each pixel position and pixel positions surrounding each pixel position, thereby generating the all-focus image; and e) obtaining a composite height for each pixel position based on the image reference values ​​and / or sharpness at a group of pixel positions including each pixel position and pixel positions surrounding each pixel position, thereby generating a height map. [Effects of the Invention]

[0019] According to the present invention, it is possible to generate a smooth all-in-focus image and an appropriate height map that matches the all-in-focus image. [Brief explanation of the drawings]

[0020] [Figure 1] FIG. 1 is a diagram illustrating a configuration of an imaging device. [Figure 2] FIG. 1 is a perspective view showing an example of a well plate. [Figure 3] FIG. 1 illustrates the configuration of a computer. [Figure 4] FIG. 2 is a block diagram showing the functional configuration of a control unit. [Figure 5] FIG. 10 is a diagram showing a processing flow for generating an all-in-focus image and a height map. [Figure 6] FIG. 2 is a longitudinal cross-sectional view of a well. [Figure 7] FIG. [Figure 8] FIG. 10 is a diagram showing an arrangement of image reference values. [Figure 9] FIG. 10 is a diagram showing an array of reference sharpness indices. [Figure 10] FIG. 10 is a diagram illustrating how the positions of surrounding pixels affect the position of a pixel of interest. [Figure 11] FIG. 10 is a diagram showing an all-in-focus image. [Figure 12] FIG. 2 is a diagram showing an analysis target region. [Figure 13] FIG. 10 is a diagram for explaining a process of a comparative example. DETAILED DESCRIPTION OF THE INVENTION

[0021] FIG. 1 is a diagram showing the configuration of an imaging device 1 according to an embodiment of the present invention. FIG. 2 is a perspective view showing an example of a well plate 2 used in the imaging device 1. In FIGS. 1 and 2, three mutually orthogonal directions are indicated by arrows as the X direction, the Y direction, and the Z direction. In the example shown in FIGS. 1 and 2, the X direction and the Y direction are horizontal directions that are perpendicular to each other, and the Z direction is a vertical direction (i.e., an up-down direction). Depending on the object to be imaged by the imaging device 1, the Z direction may be a direction different from the vertical direction.

[0022] The imaging device 1 is a device that images a sample 9 held in a well plate 2. The sample 9 is, for example, a cell, a cell mass such as a spheroid or an organoid, or a biological sample such as bacteria. In the following description, cells, cell masses, bacteria, etc. are collectively referred to as "cells, etc."

[0023] The well plate 2 is a sample container having a substantially flat shape. The well plate 2 is made of a light-transmitting material (for example, a transparent resin). One main surface of the well plate 2 (the main surface on the (+Z) side in the example shown in FIGS. 1 and 2) is provided with a plurality of wells 21, which are recesses. The multiple wells 21 are regularly arranged, for example, along the X and Y directions. The shape of each well 21 in a plan view is, for example, substantially circular. The number, arrangement, shape, etc. of the wells 21 in the well plate 2 may be changed as appropriate.

[0024] Each well 21 of the well plate 2 holds a sample 9, which is the object to be imaged by the imaging device 1, together with a liquid or gel-like culture medium 90. The sample 9 is, for example, light-transmitting cells cultured under predetermined culture conditions in the culture medium 90. While FIG. 1 and FIG. 6 described below show the cells in each well 21 as a single mass, the cells may exist as multiple masses separated from one another. Note that the imaging device 1 may also be used to image the sample 9 held in a flat sample container called a dish, rather than in the well plate 2.

[0025] The imaging device 1 includes a holder 11, an illumination unit 12, an imaging unit 13, an elevation mechanism 14, an illumination unit moving mechanism 15, an imaging unit moving mechanism 16, and a control unit 5. The holder 11 is a holding unit that holds a well plate 2. The holder 11 abuts against the peripheral edge of the main surface (i.e., the lower surface) on the (-Z) side of the well plate 2 from below, and holds the well plate 2 in a substantially horizontal state.

[0026] The illumination unit 12 is disposed above the holder 11 and emits illumination light downward (i.e., toward the (-Z) side). The illumination light emitted from the illumination unit 12 is irradiated onto the well plate 2 held by the holder 11. As a result, the sample 9 in the well 21 is illuminated from above (i.e., toward the (+Z) side). The illumination unit 12 includes a light source and an illumination optical system (not shown). For example, a white LED (Light Emitting Diode) can be used as the light source.

[0027] The imaging unit 13 is disposed below the holder 11. The imaging unit 13 includes an imaging optical system 131 and an imaging element 132. The imaging optical system 131 includes a plurality of optical elements (not shown) including an objective lens. An optical axis J1 of the imaging optical system 131 extends substantially parallel to the Z direction (i.e., the up-down direction). The imaging element 132 is disposed below the imaging optical system 131. The imaging element 132 is an area image sensor having a two-dimensional light receiving surface. For example, a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor) can be used as the imaging element 132.

[0028] As will be described later, the imaging unit 13 is disposed vertically below one well 21 of the well plate 2. The illumination unit 12 is disposed vertically above the well 21 and faces the imaging unit 13 in the up-down direction, with the well 21 sandwiched therebetween. Illumination light emitted from the illumination unit 12 enters the well 21 from above, and the sample 9 in the well 21 is illuminated by the illumination light. Light travels downward from the bottom surface of the well 21 (i.e., the surface on the (-Z) side) and passes through the well plate 2, and enters the light-receiving surface of the imaging element 132 via the imaging optical system 131 of the imaging unit 13. The imaging element 132 captures an image of the sample 9 formed on the light-receiving surface by the imaging optical system 131. The image of the sample 9 captured by the imaging unit 13 (hereinafter referred to as the "captured image") is a transmitted image of the sample 9.

[0029] The lifting mechanism 14 moves the imaging unit 13 in the up and down direction. The lighting unit moving mechanism 15 moves the lighting unit 12 in the X and Y directions. The imaging unit moving mechanism 16 moves the imaging unit 13 and the lifting mechanism 14 in the X and Y directions. The lifting mechanism 14, the lighting unit moving mechanism 15, and the imaging unit moving mechanism 16 each include, for example, a ball screw and a motor. The lifting mechanism 14, the lighting unit moving mechanism 15, and the imaging unit moving mechanism 16 may use other mechanisms, such as a linear motor.

[0030] When generating an all-in-focus image (described later), the illumination unit 12 and the imaging unit 13 are disposed vertically above and below one well 21, respectively. The imaging unit 13 is then moved vertically (along the optical axis J1) by the lifting mechanism 14, and multiple images are acquired in which the imaging unit 13 is positioned vertically at different positions during imaging. In other words, multiple images of the sample 9 are acquired while the focal position is changed along the optical axis J1. After imaging of the sample 9 in the well 21 is completed, the illumination unit 12 and the imaging unit 13 are moved horizontally (i.e., in the X and Y directions) by the illumination unit moving mechanism 15 and the imaging unit moving mechanism 16, and positioned vertically above and below another well 21, respectively. Images of the sample 9 in the other well 21 are then acquired in substantially the same manner as described above. The imaging device 1 may be provided with a moving mechanism that moves the illumination unit 12 and the imaging unit 13 together. Furthermore, images of samples 9 in multiple wells 21 may be simultaneously acquired by the imaging unit 13.

[0031] The control unit 5 controls each component of the imaging device 1, such as the illumination unit 12, the imaging unit 13, the lifting mechanism 14, the illumination unit moving mechanism 15, and the imaging unit moving mechanism 16. The control unit 5 also stores a plurality of captured images acquired by the imaging unit 13, and performs image processing to generate an all-in-focus image or the like from the plurality of captured images.

[0032] 3 is a diagram showing the configuration of a computer that functions as the control unit 5. The computer has a configuration of a typical computer system including a CPU 51, a ROM 52, a RAM 53, a storage device 54, a display 55, an input unit 56, a reading device 57, a communication unit 58, a GPU 59, and a bus 50. The CPU 51 performs various types of arithmetic processing. The GPU 59 performs various types of arithmetic processing related to image processing and the like. The ROM 52 stores basic programs. The RAM 53 stores various types of information. The storage device 54 stores information. The display 55 is a display unit that displays various types of information such as images.

[0033] The input unit 56 includes a keyboard 56a and a mouse 56b that accept input from an operator. The reader 57 reads information from a computer-readable recording medium 571, such as an optical disk, a magnetic disk, a magneto-optical disk, or a memory card. The display 55, the keyboard 56a, the mouse 56b, and the reader 57 are connected to the bus 50 via an interface I / F. The communication unit 58 transmits and receives signals to and from other components of the imaging device 1. The bus 50 is a signal circuit that connects the CPU 51, the GPU 59, the ROM 52, the RAM 53, the storage device 54, the display 55, the input unit 56, the reader 57, and the communication unit 58.

[0034] In the imaging device 1, a program 572 is read in advance from a recording medium 571 via a reading device 57 and stored in the storage device 54. The program 572 may be computer-readable and may be stored in the storage device 54 via a network, for example. The CPU 51 and the GPU 59 execute arithmetic processing in accordance with the program 572 while using the RAM 53 and the storage device 54. The CPU 51 and the GPU 59 function as a calculation unit in the imaging device 1. Other components that function as a calculation unit may be employed in addition to the CPU 51 and the GPU 59.

[0035] FIG. 4 is a block diagram showing the functional configuration of the control unit 5, which is realized by the computer executing arithmetic processing and the like according to the program 572. The control unit 5 includes an image processing unit 500. The image processing unit 500 is an image processing device that generates an all-in-focus image and a height map corresponding to the all-in-focus image, as described below. The image processing unit 500 includes an image storage unit 501, a sharpness calculation unit 502, an image reference value determination unit 503, an all-in-focus image generation unit 504, a height map generation unit 505, a target region identification unit 506, and a feature calculation unit 507. Details of these functions will be described later. All or part of the functions of the image processing unit 500 may be realized by a dedicated electrical circuit, or each function may be realized by a separate program. The image processing unit 500 may also be realized by multiple computers.

[0036] Next, the process for generating the all-in-focus image and the height map will be described with reference to FIG. 5. As described above, the operation of the imaging device 1 in FIG. 1 is controlled by the control unit 5. First, the illumination unit moving mechanism 15 and the imaging unit moving mechanism 16 are driven, and the illumination unit 12 and the imaging unit 13 are positioned above and below one well 21, respectively. Next, the lifting mechanism 14 is driven, and the vertical position of the imaging unit 13 is adjusted. Then, an image of the sample 9 is acquired by the imaging unit 13. The captured image is sent to the image processing unit 500 (see FIG. 4) and stored in the image storage unit 501. In the imaging device 1, the vertical position of the imaging unit 13 is changed by the lifting mechanism 14, and the imaging unit 13 repeatedly captures the sample 9 (i.e., acquires the captured image).

[0037] FIG. 6 is a longitudinal cross-sectional view of one well 21. In the example of FIG. 6, the focal position on the optical axis J1 of the imaging unit 13 is set to four positions H0 to H3 indicated by black circles in FIG. 6, and images of the sample 9 are respectively taken. In this manner, multiple captured images are acquired by imaging the sample 9 while changing the focal position along the optical axis J1. The multiple captured images are stored and prepared in the image storage unit 501 (step S11). Typically, the four positions H0 to H3 are arranged at equal intervals in the vertical direction (i.e., the Z direction). In an actual imaging device 1, multiple captured images are acquired by sequentially positioning the focal positions at positions significantly more than four (e.g., several tens of positions) on the optical axis J1. For ease of explanation, the following description will be given using four captured images acquired when the focal positions are set to the four positions H0 to H3 in FIG. 6. In this specification, the direction along the optical axis J1 is the height direction.

[0038] FIG. 7 shows a plurality of captured images G0 to G3 acquired with the focal position positioned at a plurality of positions H0 to H3. In each of the plurality of captured images G0 to G3, pixels are arranged in the row and column directions. In the plurality of captured images G0 to G3, pixels having the same row and column positions (hereinafter referred to as "pixel positions") indicate positions that overlap with each other in the direction of the optical axis J1 within the well 21. The plurality of captured images G0 to G3 are assigned consecutive numbers, and the four captured images G0 to G3 in FIG. 7 are assigned the numbers "0," "1," "2," and "3," respectively. Note that an unwanted object is captured in the center of the captured image G3.

[0039] When the plurality of captured images G0 to G3 are prepared, the sharpness calculation unit 502 calculates the sharpness of each pixel position in each of the plurality of captured images G0 to G3 (step S12). Sharpness is an index that indicates the clarity of an image at the pixel position and its vicinity, and here, the higher the clarity, the greater the sharpness. The sharpness is, for example, edge strength or brightness variation, and is typically calculated based on the brightness change of pixels in an area of ​​a predetermined size centered on the pixel position. The calculation of the sharpness may use the brightness variance of pixels surrounding the pixel position, the maximum brightness value, the minimum brightness value, the brightness value of the pixel position, etc.

[0040] Once multiple sharpness indices are calculated for each pixel position from the multiple captured images G0 to G3, the image reference value determination unit 503 compares the multiple sharpness indices at each pixel position. Then, the number of the captured image with the maximum sharpness among the multiple sharpness indices is determined as the image reference value for that pixel position (step S13). The image reference value is the number of the captured image to be referenced when determining the luminance value of that pixel position in the omnifocus image. For example, if, for a certain pixel position, the sharpness of the captured image G1 is the maximum among the multiple sharpness indices in the multiple captured images G0 to G3, the image reference value for that pixel position is "1."

[0041] FIG. 8 is a diagram showing the arrangement of image reference values ​​at a plurality of pixel positions. For ease of explanation, it is assumed here that the number of pixels constituting each of the captured images G0 to G3 is 5 x 5, or 25 pixels. Focusing on the central pixel position, in the example of FIG. 7, the sharpness of the captured image G3 is high, and as shown in FIG. 8, the image reference value at the central pixel position is "3." At pixel positions eight pixels away from the central pixel position, the sharpness of the captured image G1 is high, and the image reference value is "1." In FIG. 8, the larger the image reference value, the darker the color at the pixel position.

[0042] The image reference value determination unit 503 also determines the sharpness corresponding to the image reference value at each pixel position (hereinafter referred to as "reference sharpness"). FIG. 9 is a diagram showing an arrangement of reference sharpness at a plurality of pixel positions. The reference sharpness at each pixel position means the sharpness at that pixel position in the captured image indicated by the image reference value at that pixel position. For example, in the arrangement of image reference values ​​in FIG. 8, at a pixel position where the image reference value is "1", the sharpness at that pixel position in the captured image G1 becomes the reference sharpness. In FIG. 9, the larger the reference sharpness, the darker the color of the pixel position.

[0043] Next, the omni-focus image generation unit 504 calculates the luminance value of each pixel position of the omni-focus image. At this time, when calculating the luminance value of each pixel position, the image reference values ​​of pixel positions surrounding the pixel position are taken into consideration (referenced). Specifically, the omni-focus image generation unit 504 calculates the luminance value of each pixel position of the omni-focus image using, for example, Equation 1.

[0044]

number

[0045] In Equation 1, the row and column directions are represented by the x and y directions, respectively, and V(xn,yn) is the luminance value of a pixel position (hereinafter referred to as the "target pixel position") at any coordinate (xn,yn) in the omni-focus image. k and l are the distances in the x and y directions between the pixel position referenced with respect to the target pixel position and the target pixel position. fx and fy are the maximum values ​​of the distances in the x and y directions, and indicate the range of pixel positions referenced with respect to the target pixel position. I(A(xn+k,yn+l),xn,yn) is the luminance value of the target pixel position in the captured image indicated by the image reference value A(xn+k,yn+l) at the pixel position with coordinate (xn+k,yn+l). S(xn+k,yn+l) is the reference sharpness at the pixel position with coordinate (xn+k,yn+l). Here, the reference sharpness is assumed to be normalized to a value between 0 and 1. σ d is the distance weighting factor, and σ s is a weighting coefficient for the reference sharpness. In Equation 1, the weighting amount for the distance and the weighting amount for the reference sharpness are expressed as Gaussian coefficients.

[0046] In Equation 1, the luminance value of the pixel position of interest in the omni-focus image is calculated by reflecting the luminance value of the pixel position of interest in the captured image indicated by the image reference values ​​of each pixel position surrounding the pixel position of interest on the luminance value of the pixel position of interest in the captured image indicated by the image reference value of the pixel position of interest. In this way, the luminance value of the pixel position of interest in the omni-focus image is calculated using multiple luminance values ​​derived from the image reference values ​​of the pixel position of interest and a group of pixel positions including the pixel positions surrounding the pixel position of interest. In Figure 10, when the central pixel position in the array of image reference values ​​in Figure 8 is set as the pixel position of interest, multiple arrows A1 indicate that the image reference values ​​of the surrounding pixel positions affect the luminance value of the pixel position of interest.

[0047] In Equation 1, the luminance value of the captured image indicated by the image reference value of the surrounding pixel positions (luminance value of the pixel position of interest) is assigned a weight according to the distance between the surrounding pixel positions and the pixel position of interest, and a weight according to the reference sharpness of the surrounding pixel positions. Specifically, the closer a pixel position is to the pixel position of interest, the greater the weight (the greater the influence on the luminance value of the omni-focus image). Furthermore, the higher the reference sharpness of the surrounding pixel positions, the greater the weight. The omni-focus image generation unit 504 calculates the luminance values ​​of all pixel positions using Equation 1. This generates the omni-focus image G10 shown in FIG. 11 (step S14). In the omni-focus image G10, the luminance value can be smoothly changed in the area where the image reference value switches. The omni-focus image generation unit 504 may reflect the luminance values ​​of the surrounding pixel positions in the captured image indicated by the image reference value of each pixel position around the pixel position of interest in the luminance value of the pixel position of interest in the captured image indicated by the image reference value of the pixel position of interest.

[0048] Next, the height map generator 505 calculates the composite height for each pixel position. Here, the image reference value for each pixel position indicates the number of the captured image that is most in focus for that pixel position, and can therefore be regarded as the height of the sample 9 at that pixel position. Meanwhile, the all-in-focus image generator 504 calculates the luminance value of each pixel position in the all-in-focus image G10 using not only the image reference value for that pixel position but also the image reference values ​​for the pixel positions surrounding that pixel position (see FIG. 10). Therefore, in calculating the composite height, the image reference value for each pixel position reflects the image reference values ​​for the pixel positions surrounding that pixel position, similar to the calculation of the luminance value of the all-in-focus image G10. Specifically, the composite height for each pixel position is calculated using Equation 2.

[0049]

number

[0050] In Equation 2, as in Equation 1, the row and column directions are represented by the x and y directions, respectively, and D(xn,yn) is the composite height for a pixel position (i.e., a pixel position of interest) at any coordinate (xn,yn) in the omni-focus image G10. k and l are the distances in the x and y directions between the pixel position referenced for the pixel position of interest (i.e., a pixel position to which the image reference value is applied) and the pixel position of interest. fx and fy are the maximum values ​​of the distances in the x and y directions, and indicate the range of pixel positions referenced for the pixel position of interest (which can also be considered as the smoothing range). A(xn+k,yn+l) is the image reference value for the pixel position of coordinate (xn+k,yn+l). S(xn+k,yn+l) is the reference sharpness at the pixel position of coordinate (xn+k,yn+l). Here, the reference sharpness is assumed to be normalized to a value between 0 and 1. σ d is the distance weighting factor, and σ s is a weighting coefficient for the reference sharpness. In Equation 2, the weighting amount for the distance and the weighting amount for the reference sharpness are expressed by Gaussian coefficients.

[0051] In Equation 2, the composite height of the pixel of interest is calculated by reflecting the image reference values ​​of the pixel of interest's surrounding pixel positions on the image reference value of the pixel of interest. In other words, the composite height of the pixel of interest is calculated using the image reference values ​​of the pixel of interest and a group of pixel positions including the pixel of interest and its surrounding pixel positions. At this time, the image reference values ​​of the surrounding pixel positions are assigned weights according to the distance between the pixel of interest and the surrounding pixel positions and weights according to the reference sharpness of the surrounding pixel positions. In particular, the closer a pixel position is to the pixel of interest, the greater the influence (weight) of its image reference value on the composite height. Furthermore, the higher the reference sharpness of the surrounding pixel positions, the greater the influence of its image reference value on the composite height.

[0052] The height map generator 505 calculates the composite heights of all pixel positions using Equation 2. As a result, a height map indicating the composite heights of all pixel positions is generated (step S15). As with the luminance value of the omni-focus image G10, the composite height of the pixel position of interest is calculated from the image reference values ​​of the pixel position group including the pixel position of interest and pixel positions surrounding the pixel position of interest, so an appropriate height map tailored to the omni-focus image G10 is obtained. As will be described later, the height map is used for analyzing the omni-focus image G10. The composite height may be expressed as a height from a predetermined reference position in the well 21 based on the distance between the focal positions when the multiple captured images G0 to G3 were acquired in step S11.

[0053] Next, the target region identifying unit 506 identifies and extracts each analysis target region in the all-in-focus image G10 (step S16). FIG. 12 is a diagram showing the analysis target region R1. The analysis target region R1 is a region that represents a portion of the sample 9 in the all-in-focus image G10. When multiple portions of the sample 9 are dispersed in the all-in-focus image G10, the analysis target regions R1 are extracted individually. The analysis target region R1 is identified using known techniques such as binarization using a predetermined threshold or segmentation using deep learning.

[0054] Once the analysis target regions R1 are extracted, the feature amount calculation unit 507 calculates the feature amount for the height of each analysis target region R1 using the height map (step S17). For example, the average value, mode, or variation (standard deviation, etc.) of the height in the analysis target region R1 is calculated as the feature amount. The feature amount for each analysis target region R1 is used to understand the state of the sample 9, etc. This completes the process for generating the all-in-focus image and height map in the imaging device 1.

[0055] Here, a comparative example of the image processing for generating an all-in-focus image and a height map will be described. The processing of the comparative example is similar to the processing of FIG. 5, except that the array of image reference values ​​in FIG. 8 is used as the height map. In the processing of the comparative example, for example, when multiple captured images 911 to 913 shown on the far left of FIG. 13 are prepared, a height map 921 (i.e., an array of image reference values) shown in the top row, second from the left, and an array of reference sharpness 922 shown in the bottom row are acquired, and an all-in-focus image 93, third from the left, is generated. At this time, when focusing on the central pixel position, the luminance value of the central pixel position is calculated taking into account the image reference values ​​of the surrounding pixel positions, etc. However, since the image reference values ​​of the surrounding pixel positions are the same as the image reference value of the central pixel position (see the height map 921), the influence of the surrounding pixel positions is not significant. Therefore, regarding the relationship between the central pixel position and the surrounding pixel positions, there is no significant discrepancy (difference) between the luminance value of the all-in-focus image 93 and the height indicated by the height map 921. As a result, it is possible to calculate an appropriate feature amount (here, average height) for the analysis target region 94 shown fourth from the left using the height map 921.

[0056] On the other hand, in the process of the comparative example, when multiple captured images G0 to G3 shown in FIG. 7 are prepared, for example, the influence of the surrounding pixel positions is significant when calculating the luminance value of the central pixel position in the all-in-focus image because the image reference values ​​of the surrounding pixel positions are significantly different from the image reference value of the central pixel position (see FIG. 8). That is, regarding the relationship between the central pixel position and the surrounding pixel positions, a large discrepancy occurs between the luminance value of the all-in-focus image and the height indicated by the height map (here, the array of image reference values ​​in FIG. 8). As a result, it is not possible to calculate appropriate feature amounts for the analysis target area that are tailored to the all-in-focus image.

[0057] In contrast, the image processing method of Fig. 5 includes a step of calculating a luminance value of a pixel position of an all-focus image based on image reference values ​​at each pixel position and a group of pixel positions including the pixel position and its surrounding pixel positions to generate the all-focus image (step S14), and a step of obtaining a composite height for the pixel position based on image reference values ​​at each pixel position and a group of pixel positions including the pixel position and its surrounding pixel positions to generate a height map (step S15). In this way, by taking into account the image reference values ​​of the pixel positions surrounding each pixel position in both the generation of the all-focus image and the generation of the height map, a large discrepancy between the luminance value of the all-focus image and the height (composite height) indicated by the height map regarding the relationship between each pixel position and its surrounding pixel positions can be prevented or suppressed. As a result, a smooth all-focus image and an appropriate height map that matches the all-focus image can be generated.

[0058] Preferably, the image processing method further includes a step of identifying an analysis target area in the all-in-focus image (step S16), and a step of calculating a feature amount for the height of the analysis target area using a height map (step S17). This allows for calculation of an appropriate feature amount suited to the all-in-focus image (i.e., a height feature amount that has an accurate relationship with the all-in-focus image), and allows for appropriate analysis of the analysis target area.

[0059] Preferably, the pixel positions in step S14 and the pixel positions in step S15 are the same. This allows a more appropriate height map to be obtained that is aligned with the omni-focus image. Note that, depending on the accuracy required for the height map, the pixel positions in step S14 and the pixel positions in step S15 may be partially different.

[0060] Preferably, in step S15, the composite height for each pixel position is calculated by reflecting the image reference values ​​of surrounding pixel positions on the image reference value of each pixel position with weighting such that the higher the sharpness, the greater the influence. In this way, by reflecting the image reference values ​​of surrounding pixel positions on the image reference value of each pixel position while taking sharpness into consideration, a preferable height map can be obtained. Also, in generating the omnifocus image, when calculating the luminance value of each pixel position, the image reference values ​​of surrounding pixel positions are referenced with weighting such that the higher the sharpness, the greater the influence. This makes it possible to further reduce the discrepancy between the luminance value of the omnifocus image and the composite height indicated by the height map.

[0061] Preferably, in step S15, the composite height for each pixel position is calculated by reflecting the image reference values ​​of surrounding pixel positions on the image reference value of the pixel position with a weighting that increases the influence the closer the pixel is. In this way, a preferable height map can be obtained by reflecting the image reference values ​​of surrounding pixel positions on the image reference value of each pixel position while taking distance into consideration. Also, in generating the omnifocus image, when calculating the luminance value of each pixel position, the image reference values ​​of surrounding pixel positions are referenced with a weighting that increases the influence the closer the pixel is. This makes it possible to further reduce the discrepancy between the luminance value of the omnifocus image and the composite height indicated by the height map. Note that weighting for sharpness and / or distance may be omitted in calculating the composite height in step S15.

[0062] In the above processing example, only one image reference value is determined for each pixel position in step S13, but as in JP 2022-51094 A (the above-mentioned Patent Document 2), M image reference values ​​may be determined corresponding to the top M sharpness levels (where M is an integer greater than or equal to 2 and less than the number of captured images). In the following description, at each pixel position, the number of the captured image with the greatest sharpness level will be referred to as the "first image reference value," the number of the captured image with the second greatest sharpness level will be referred to as the "second image reference value," ... and the number of the captured image with the Mth greatest sharpness level will be referred to as the "Mth image reference value."

[0063] The image reference value determination unit 503 also determines sharpness (i.e., reference sharpness) corresponding to the first through Mth image reference values ​​for each pixel position. The reference sharpness of the first image reference value is the highest sharpness, the reference sharpness of the second image reference value is the second highest sharpness, and the reference sharpness of the Mth image reference value is the Mth highest sharpness. The omnifocus image generation unit 504 calculates the luminance value of each pixel position of the omnifocus image using, for example, Equation 3, and generates the omnifocus image (step S14).

[0064]

number

[0065] In Equation 3, I(A(m, xn+k, yn+l), xn, yn) is the luminance value of the pixel position of interest with coordinates (xn, yn) in the captured image indicated by the mth image reference value A(m, xn+k, yn+l) at the pixel position with coordinates (xn+k, yn+l). S(m, xn+k, yn+l) is the reference sharpness of the mth image reference value at the pixel position with coordinates (xn+k, yn+l). M is the number of image reference values ​​for each pixel position. The rest is the same as Equation 1. In Equation 3, the luminance value of the pixel position of interest in the omnifocus image is calculated using multiple luminance values ​​derived from the first to Mth image reference values ​​of the pixel position group including the pixel position of interest and pixel positions surrounding the pixel position of interest.

[0066] In the height map generating unit 505, for example, the composite height for each pixel position is calculated using Equation 4, and a height map is generated (step S15).

[0067]

number

[0068] In Equation 4, A(m, xn+k, yn+l) is the mth image reference value at the pixel position of coordinates (xn+k, yn+l). S(m, xn+k, yn+l) is the reference sharpness of the mth image reference value at the pixel position of coordinates (xn+k, yn+l). M is the number of image reference values ​​for each pixel position. The rest is the same as Equation 2. In Equation 4, the composite height of the pixel position of interest is calculated using the first through Mth image reference values ​​of the pixel position group including the pixel position of interest and pixel positions surrounding the pixel position of interest. The processing of steps S16 and S17 is the same as the processing example above.

[0069] As described above, in step S13, by comparing multiple sharpness indices for each pixel position, the number of at least one captured image to be referenced in determining the luminance value of the pixel position in the omnifocus image can be determined as an image reference value. This makes it possible to generate a smooth omnifocus image and an appropriate height map that matches the omnifocus image using the image reference value.

[0070] Next, another example of processing in step S15 will be described. In this example, the height map generating unit 505 calculates an evaluation value for sharpness for each pixel position in each of the multiple captured images. Specifically, first, for a pixel position (i.e., a pixel position of interest) at any coordinate (xn, yn) in each captured image, weighted sharpness Sp is calculated for each of surrounding pixel positions located within a predetermined set range using Equation 5.

[0071]

number

[0072] In Equation 5, k and l are the distances in the x and y directions between the pixel position (surrounding pixel positions) referred to with respect to the pixel position of interest and the pixel position of interest. S(xn+k,yn+l) is the sharpness of the pixel position of interest in the captured image. σ d is a distance weighting coefficient. Next, the sum of the weighted sharpness Sp of pixel positions (including the pixel position of interest) within the above-mentioned set range relative to the pixel position of interest is calculated as an evaluation value. In this way, an evaluation value for each pixel position is calculated for each of the multiple captured images. Then, the number of the captured image with the largest evaluation value among the evaluation values ​​of the multiple captured images is acquired as the composite height of that pixel position.

[0073] As described above, in the other processing example of step S15, a composite height for each pixel position is obtained using the sharpness at each pixel position and at a pixel position group including pixel positions surrounding the pixel position. Also, in the first processing example of step S15, a composite height for each pixel position is obtained using the image reference value at each pixel position or the image reference value and the sharpness at each pixel position. In this way, by using the image reference value and / or the sharpness at each pixel position and at a pixel position group including pixel positions surrounding the pixel position, it is possible to obtain an appropriate composite height for the pixel position that matches the omni-focus image.

[0074] In the above method using Equation 2, for example, if there are two elements at a certain pixel position that are different in height along the optical axis J1 and have high sharpness, the composite height at the pixel position may be near the center of the two elements, which may not be desirable. In such cases, multiple height maps may be obtained by modifying the other processing example of step S15.

[0075] Specifically, of the evaluation values ​​of the multiple captured images, the number of the captured image with the largest evaluation value is acquired as the "first composite height," the number of the captured image with the second largest evaluation value is acquired as the "second composite height," and the number of the captured image with the Nth highest sharpness (where N is an integer greater than or equal to 2 and less than the number of captured images) is acquired as the "Nth composite height." This generates a first height map indicating the first composite height of all pixel positions, a second height map indicating the second composite height, and an Nth height map indicating the Nth composite height. By referring to the first to Nth height maps, it is possible to grasp the distribution of focus positions in the direction of the optical axis J1 for each pixel position.

[0076] The image processing method, image processing device, and imaging device 1 described above can be modified in various ways.

[0077] The above-mentioned formulas 1 to 5 used in the all-in-focus image generating section 504 and the height map generating section 505 are merely examples and may be changed as appropriate.

[0078] 1, the focal position changing mechanism that changes the focal position of the imaging unit 13 along the optical axis J1 is realized by an elevation mechanism 14 that moves the imaging unit 13, but the focal position changing mechanism may also be realized by a mechanism that moves some of the lenses in the imaging unit 13 along the optical axis J1. Also, a mechanism that moves the sample 9 along the optical axis J1 may be provided as the focal position changing mechanism.

[0079] The height map may be used for purposes other than calculating the feature amount of the analysis target region.

[0080] In the above embodiment, a transmission image of the sample 9 is acquired as the captured image, but a fluorescent image or the like of the sample 9 may also be acquired as the captured image. Furthermore, the object may be something other than a cell or the like.

[0081] The image processing unit 500 (image processing device) may be used independently of the imaging device 1.

[0082] The configurations in the above-described embodiment and each modification may be combined as appropriate as long as they are not mutually contradictory. [Explanation of symbols]

[0083] 1. Imaging device 5. Control section 9 Samples 12 Lighting Department 13 Imaging unit 14 Lifting mechanism 500 Image Processing Unit 501 Image storage unit 502 Sharpness calculation unit 503 Image reference value determination unit 504 All-focus image generation unit 505 Height Map Generator 506 Target Area Identification Unit 507 Feature Calculation Unit 572 Programs G0~G3 captured images G10 all focus image J1 optical axis R1 Analysis Area Steps S11~S17

Claims

1. 1. An image processing method for generating an all-in-focus image and a height map for use in analyzing the all-in-focus image, comprising: a) preparing a plurality of captured images obtained by capturing an image of an object while changing a focal position along an optical axis; b) calculating a plurality of sharpness indices from the plurality of captured images for each pixel position; c) determining, as an image reference value, the number of at least one captured image to be referenced in determining the luminance value of each pixel position of the omnifocus image by comparing the plurality of sharpness indices for each pixel position; d) calculating a luminance value of each pixel position of the all-in-focus image based on image reference values ​​at each pixel position and a group of pixel positions including pixel positions surrounding each pixel position, and generating the all-in-focus image; e) obtaining a composite height for each pixel position based on image reference values ​​and / or sharpness at each pixel position and a group of pixel positions including pixel positions surrounding each pixel position, and generating a height map; An image processing method comprising:

2. 2. The image processing method according to claim 1, The image processing method, wherein the group of pixel positions in step d) is the same as the group of pixel positions in step e).

3. 2. The image processing method according to claim 1, In the step e), the composite height for each pixel position is calculated by reflecting the image reference values ​​of the surrounding pixel positions on the image reference value of each pixel position with a weighting such that the higher the sharpness, the greater the influence.

4. 2. The image processing method according to claim 1, In the step e), the composite height for each pixel position is calculated by reflecting the image reference values ​​of the surrounding pixel positions on the image reference value of each pixel position with a weighting such that the closer the pixel position, the greater the influence.

5. 5. An image processing method according to claim 1, further comprising: identifying an analysis target region in the all-in-focus image; calculating a feature amount for the height of the analysis target region using the height map; The image processing method further comprises:

6. An image processing device that generates an all-in-focus image and a height map used in analyzing the all-in-focus image, an image storage unit that stores a plurality of captured images obtained by capturing images of an object while changing a focal position along an optical axis; a sharpness calculation unit that calculates a plurality of sharpness indices from the plurality of captured images for each pixel position; an image reference value determination unit that determines, as an image reference value, a number of at least one captured image to be referenced in determining a luminance value of each pixel position of the omnifocus image by comparing the plurality of sharpness indices for each pixel position; an all-in-focus image generating unit that calculates a luminance value of each pixel position of the all-in-focus image based on image reference values ​​at each pixel position and a pixel position group including pixel positions surrounding each pixel position, and generates the all-in-focus image; a height map generator that obtains a composite height for each pixel position based on an image reference value and / or a sharpness at each pixel position and a group of pixel positions including pixel positions surrounding each pixel position, and generates a height map; An image processing device comprising:

7. 7. The image processing device according to claim 6, an image processing device in which the pixel positions in the omni-focus image generation unit and the pixel positions in the height map generation unit are the same;

8. 7. The image processing device according to claim 6, An image processing device in which the height map generation unit obtains the composite height for each pixel position by reflecting the image reference values ​​of the surrounding pixel positions on the image reference value of each pixel position with a weighting such that the higher the sharpness, the greater the influence.

9. 7. The image processing device according to claim 6, An image processing device in which the height map generation unit acquires the composite height for each pixel position by reflecting the image reference values ​​of the surrounding pixel positions on the image reference value of each pixel position with a weighting such that the closer the distance, the greater the influence.

10. 7. The image processing device according to claim 6, a target region specifying unit that specifies an analysis target region in the omnifocus image; a feature amount calculation unit that calculates a feature amount regarding the height of the analysis target region using the height map; The image processing device further comprises:

11. An imaging device, An image processing device according to any one of claims 6 to 10; an imaging unit that images the object; an illumination unit that emits light toward the object; a focal position changing mechanism that changes the focal position of the imaging unit along an optical axis; An imaging device comprising:

12. A computer-readable program that causes a computer to generate an all-in-focus image and a height map used in analyzing the all-in-focus image, wherein execution of the program by a computer causes the computer to: a) preparing a plurality of captured images obtained by capturing an image of an object while changing a focal position along an optical axis; b) calculating a plurality of sharpness indices from the plurality of captured images for each pixel position; c) determining, as an image reference value, the number of at least one captured image to be referenced in determining the luminance value of each pixel position of the omnifocus image by comparing the plurality of sharpness indices for each pixel position; d) calculating a luminance value of each pixel position of the all-in-focus image based on image reference values ​​at each pixel position and a group of pixel positions including pixel positions surrounding each pixel position, and generating the all-in-focus image; e) obtaining a composite height for each pixel position based on image reference values ​​and / or sharpness at each pixel position and a group of pixel positions including pixel positions surrounding each pixel position, and generating a height map; A computer-readable program that causes a

Images