Image processing method, image processing device, imaging device, and computer-readable program

Abstract

The area to be analyzed is divided appropriately based on the height distribution, etc. [Solution] The image processing device includes a storage unit (501) that stores a target image showing an object viewed from a predetermined direction and a height map that indicates the height in the predetermined direction of parts of the object corresponding to each pixel position in the target image, a target region identification unit (503) that identifies an analysis target region in the target image, a region division unit (504) that divides the analysis target region into multiple divided regions according to the height indicated by the height map, an unsuitable region identification unit (505) that identifies unsuitable regions among the multiple divided regions that do not satisfy predetermined judgment conditions, and a region integration unit (506) that calculates a judgment score when an unsuitable region is integrated with other divided regions that contact the unsuitable region, and integrates the unsuitable region with the other divided regions based on the judgment score to obtain a new divided region. This allows the analysis target region to be appropriately divided based on height distribution, etc.

Description

[Technical Field]

[0001] The present invention relates to image processing. [Background technology]

[0002] A conventional technique used in microscopes involves capturing an image of an object while varying the focal position along the optical axis, acquiring multiple captured images, and combining the in-focus portions of each captured image to generate an image in which the entire object is in focus (i.e., an all-focus image). For example, in Patent Document 1, multiple captured images are captured while varying the focal position, and the sharpness values ​​in the multiple captured images are compared for each pixel position (coordinate), thereby determining an image reference value, which is the number of the captured image to be referenced as the luminance value of the all-focus image. Then, the luminance value of each pixel position in the all-focus image is calculated by reflecting the luminance values ​​of the captured images indicated by the image reference values ​​of surrounding pixel positions on the luminance value of the captured image indicated by the image reference value of the pixel position. Furthermore, in Patent Document 2, for each pixel position, a predetermined number of sharpness values ​​with the highest sharpness values ​​in multiple captured images (images with varying focal positions) are extracted as correction sharpness values, and a predetermined number of image reference values, which are the numbers of the captured images corresponding to the predetermined number of correction sharpness values, are determined. Then, based on the predetermined number of image reference values ​​and the predetermined number of correction sharpness factors, a luminance value of the pixel position in the omnifocus image is calculated. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2018-42006 [Patent Document 2] Japanese Patent Publication No. 2022-51094 Summary of the Invention [Problem to be solved by the invention]

[0004] When analyzing a cell cluster using an all-in-focus image in which the object in the all-in-focus image is, for example, a cell cluster, the cell cluster region is extracted from the all-in-focus image. On the other hand, when multiple cell clusters located at different positions along the optical axis overlap along the optical axis (when viewed along the optical axis), an all-in-focus image is generated in which the multiple cell clusters appear to be fused together. In the all-in-focus image, a single region where the multiple cell clusters overlap is extracted as the region to be analyzed. Watershed is a well-known method for dividing such regions. Watershed allows the division line to be obtained by setting a seed region that serves as the basis for division. However, if it is difficult to determine the seed region from the shape of the target region, division becomes difficult. Furthermore, the shape of the division region cannot be controlled. Therefore, a new method capable of appropriately dividing the region to be analyzed is needed.

[0005] The present invention has been made in view of the above-mentioned problems, and has as its object to appropriately divide an area to be analyzed. [Means for solving the problem]

[0006] A first aspect of the present invention is an image processing method comprising the steps of: a) preparing a target image showing an object viewed from a predetermined direction and a height map showing the height in the predetermined direction of a part of the object corresponding to each pixel position in the target image; b) identifying an area to be analyzed in the target image; c) dividing the area to be analyzed into a plurality of divided areas according to the heights shown in the height map; d) identifying an unsuitable area among the plurality of divided areas that does not satisfy a predetermined judgment condition; and e) calculating a judgment score when the unsuitable area is integrated with other divided areas that contact the unsuitable area, and integrating the unsuitable area with the other divided areas based on the judgment score to obtain a new divided area.

[0007] A second aspect of the present invention is the image processing method of the first aspect, wherein the a) step comprises: a1) preparing a plurality of captured images obtained by capturing images of the object while changing the focal position along an optical axis parallel to the predetermined direction; and a2) generating an all-in-focus image, which is the object image, and the height map based on the plurality of captured images.

[0008] Aspect 3 of the present invention is an image processing method of aspect 1 (which may be aspect 1 or 2), in which step c) comprises the steps of generating a histogram of heights in the area to be analyzed based on the height map, and obtaining a set of pixel positions corresponding to each peak in the histogram as a divided area.

[0009] Aspect 4 of the present invention is an image processing method of aspect 1 (which may be any one of aspects 1 to 3), further comprising a step of identifying the new divided area obtained by step e) as a new non-conforming area if the new divided area does not satisfy a predetermined judgment condition.

[0010] Aspect 5 of the present invention is an image processing method according to any one of aspects 1 to 4, wherein the height indicated by the height map is used to identify the unsuitable area in step d) and / or to calculate the judgment score in step e).

[0011] A sixth aspect of the present invention is an image processing device comprising: a memory unit that stores a target image showing an object viewed from a predetermined direction and a height map that indicates the height in the predetermined direction of a part of the object corresponding to each pixel position of the target image; a target area identification unit that identifies an area to be analyzed in the target image; an area division unit that divides the area to be analyzed into a plurality of divided areas according to the height indicated by the height map; an unsuitable area identification unit that identifies an unsuitable area among the plurality of divided areas that does not satisfy a predetermined judgment condition; and an area integration unit that calculates a judgment score when the unsuitable area is integrated with other divided areas that contact the unsuitable area, and integrates the unsuitable area with the other divided areas based on the judgment score to obtain a new divided area.

[0012] A seventh aspect of the present invention is an image processing device of the sixth aspect, further comprising an image generation unit that generates an all-in-focus image, which is the target image, and the height map based on a plurality of captured images obtained by capturing images of the target object while changing the focal position along an optical axis parallel to the predetermined direction.

[0013] Aspect 8 of the present invention is an image processing device of aspect 6 (which may also be aspect 6 or 7), in which the area division unit generates a histogram of heights in the area to be analyzed based on the height map, and obtains a set of pixel positions corresponding to each peak in the histogram as a divided area.

[0014] A ninth aspect of the present invention is an image processing device of aspect 6 (which may be any one of aspects 6 to 8), in which the incompatible area identification unit identifies the new divided area acquired by the area integration unit as a new incompatible area when the new divided area does not satisfy a predetermined judgment condition.

[0015] Aspect 10 of the present invention is an image processing device of aspect 6 (which may be any one of aspects 6 to 9), in which the height indicated by the height map is used in identifying the unsuitable area by the unsuitable area identification unit and / or in calculating the judgment score by the area integration unit.

[0016] Aspect 11 of the present invention is an imaging device comprising an image processing device of any one of aspects 6 to 10, an imaging unit that images the object, an illumination unit that emits light toward the object, and a focal position changing mechanism that changes the focal position of the imaging unit along the optical axis.

[0017] A twelfth aspect of the present invention is a computer-readable program that causes a computer to perform image processing, and execution of the program by a computer causes the computer to perform the following steps: a) preparing a target image showing an object viewed from a predetermined direction and a height map showing the height in the predetermined direction of a part of the object corresponding to each pixel position in the target image; b) identifying an area to be analyzed in the target image; c) dividing the area to be analyzed into multiple divided areas according to the height shown in the height map; d) identifying an unsuitable area among the multiple divided areas that does not satisfy predetermined judgment conditions; and e) calculating a judgment score when the unsuitable area is integrated with other divided areas that contact the unsuitable area, and integrating the unsuitable area with the other divided areas based on the judgment score to obtain a new divided area. [Effects of the Invention]

[0018] According to the present invention, the analysis target area can be appropriately divided based on the height distribution or the like. [Brief explanation of the drawings]

[0019] [Figure 1] FIG. 1 is a diagram illustrating a configuration of an imaging device. [Figure 2] FIG. 1 is a perspective view showing an example of a well plate. [Figure 3] FIG. 1 illustrates the configuration of a computer. [Figure 4] FIG. 2 is a block diagram showing the functional configuration of a control unit. [Figure 5] FIG. 10 is a diagram showing a processing flow relating to generation of an omnifocus image and division of an analysis target region. [Figure 6] FIG. 2 is a longitudinal cross-sectional view of a well. [Figure 7] FIG. 10 is a diagram for explaining generation of an all-in-focus image. [Figure 8] FIG. 10 is a diagram for explaining generation of an all-in-focus image. [Figure 9] FIG. 10 is a diagram showing a height map. [Figure 10] FIG. 10 is a diagram showing a histogram of heights. [Figure 11] FIG. 2 is a diagram showing a plurality of divided regions. [Figure 12] FIG. 1 illustrates conforming and non-conforming regions. [Figure 13A] FIG. 10 is a diagram illustrating an integrated region. [Figure 13B] FIG. 10 is a diagram illustrating an integrated region. [Figure 14] FIG. 10 is a diagram showing a new divided region. DETAILED DESCRIPTION OF THE INVENTION

[0020] FIG. 1 is a diagram showing the configuration of an imaging device 1 according to an embodiment of the present invention. FIG. 2 is a perspective view showing an example of a well plate 2 used in the imaging device 1. In FIGS. 1 and 2, three mutually orthogonal directions are indicated by arrows as the X direction, the Y direction, and the Z direction. In the example shown in FIGS. 1 and 2, the X direction and the Y direction are horizontal directions that are perpendicular to each other, and the Z direction is a vertical direction (i.e., an up-down direction). Depending on the object to be imaged by the imaging device 1, the Z direction may be a direction different from the vertical direction.

[0021] The imaging device 1 is a device that images a sample 9 held in a well plate 2. The sample 9 is, for example, a cell, a cell mass such as a spheroid or an organoid, or a biological sample such as bacteria. In the following description, cells, cell masses, bacteria, etc. are collectively referred to as "cells, etc."

[0022] The well plate 2 is a sample container having a substantially flat shape. The well plate 2 is made of a light-transmitting material (for example, a transparent resin). One main surface of the well plate 2 (the main surface on the (+Z) side in the example shown in FIGS. 1 and 2) is provided with a plurality of wells 21, which are recesses. The multiple wells 21 are regularly arranged, for example, along the X and Y directions. The shape of each well 21 in a plan view is, for example, substantially circular. The number, arrangement, shape, etc. of the wells 21 in the well plate 2 may be changed as appropriate.

[0023] Each well 21 of the well plate 2 holds a sample 9, which is the object to be imaged by the imaging device 1, together with a liquid or gel-like culture medium 90. The sample 9 is, for example, light-transmitting cells cultured under predetermined culture conditions in the culture medium 90. While FIG. 1 and FIG. 6 described below show the cells in each well 21 as a single mass, the cells may exist as multiple masses separated from one another. Note that the imaging device 1 may also be used to image the sample 9 held in a flat sample container called a dish, rather than in the well plate 2.

[0024] The imaging device 1 includes a holder 11, an illumination unit 12, an imaging unit 13, an elevation mechanism 14, an illumination unit moving mechanism 15, an imaging unit moving mechanism 16, and a control unit 5. The holder 11 is a holding unit that holds a well plate 2. The holder 11 abuts against the peripheral edge of the main surface (i.e., the lower surface) on the (-Z) side of the well plate 2 from below, and holds the well plate 2 in a substantially horizontal state.

[0025] The illumination unit 12 is disposed above the holder 11 and emits illumination light downward (i.e., toward the (-Z) side). The illumination light emitted from the illumination unit 12 is irradiated onto the well plate 2 held by the holder 11. As a result, the sample 9 in the well 21 is illuminated from above (i.e., toward the (+Z) side). The illumination unit 12 includes a light source and an illumination optical system (not shown). For example, a white LED (Light Emitting Diode) can be used as the light source.

[0026] The imaging unit 13 is disposed below the holder 11. The imaging unit 13 includes an imaging optical system 131 and an imaging element 132. The imaging optical system 131 includes a plurality of optical elements (not shown) including an objective lens. An optical axis J1 of the imaging optical system 131 extends substantially parallel to the Z direction (i.e., the up-down direction). The imaging element 132 is disposed below the imaging optical system 131. The imaging element 132 is an area image sensor having a two-dimensional light receiving surface. For example, a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor) can be used as the imaging element 132.

[0027] As will be described later, the imaging unit 13 is disposed vertically below one well 21 of the well plate 2. The illumination unit 12 is disposed vertically above the well 21 and faces the imaging unit 13 in the up-down direction, with the well 21 sandwiched therebetween. Illumination light emitted from the illumination unit 12 enters the well 21 from above, and the sample 9 in the well 21 is illuminated by the illumination light. Light travels downward from the bottom surface of the well 21 (i.e., the surface on the (-Z) side) and passes through the well plate 2, and enters the light-receiving surface of the imaging element 132 via the imaging optical system 131 of the imaging unit 13. The imaging element 132 captures an image of the sample 9 formed on the light-receiving surface by the imaging optical system 131. The image of the sample 9 captured by the imaging unit 13 (hereinafter referred to as the "captured image") is a transmitted image of the sample 9.

[0028] The lifting mechanism 14 moves the imaging unit 13 in the up and down direction. The lighting unit moving mechanism 15 moves the lighting unit 12 in the X and Y directions. The imaging unit moving mechanism 16 moves the imaging unit 13 and the lifting mechanism 14 in the X and Y directions. The lifting mechanism 14, the lighting unit moving mechanism 15, and the imaging unit moving mechanism 16 each include, for example, a ball screw and a motor. The lifting mechanism 14, the lighting unit moving mechanism 15, and the imaging unit moving mechanism 16 may use other mechanisms, such as a linear motor.

[0029] When generating an all-in-focus image (described later), the illumination unit 12 and the imaging unit 13 are disposed vertically above and below one well 21, respectively. The imaging unit 13 is then moved vertically (along the optical axis J1) by the lifting mechanism 14, and multiple images are acquired in which the imaging unit 13 is positioned vertically at different positions during imaging. In other words, multiple images of the sample 9 are acquired while the focal position is changed along the optical axis J1. After imaging of the sample 9 in the well 21 is completed, the illumination unit 12 and the imaging unit 13 are moved horizontally (i.e., in the X and Y directions) by the illumination unit moving mechanism 15 and the imaging unit moving mechanism 16, and positioned vertically above and below another well 21, respectively. Images of the sample 9 in the other well 21 are then acquired in substantially the same manner as described above. The imaging device 1 may be provided with a moving mechanism that moves the illumination unit 12 and the imaging unit 13 together. Furthermore, images of samples 9 in multiple wells 21 may be simultaneously acquired by the imaging unit 13.

[0030] The control unit 5 controls each component of the imaging device 1, such as the illumination unit 12, the imaging unit 13, the lifting mechanism 14, the illumination unit moving mechanism 15, and the imaging unit moving mechanism 16. The control unit 5 also stores a plurality of captured images acquired by the imaging unit 13, and performs image processing to generate an all-in-focus image or the like from the plurality of captured images.

[0031] 3 is a diagram showing the configuration of a computer that functions as the control unit 5. The computer has a configuration of a typical computer system including a CPU 51, a ROM 52, a RAM 53, a storage device 54, a display 55, an input unit 56, a reading device 57, a communication unit 58, a GPU 59, and a bus 50. The CPU 51 performs various types of arithmetic processing. The GPU 59 performs various types of arithmetic processing related to image processing and the like. The ROM 52 stores basic programs. The RAM 53 stores various types of information. The storage device 54 stores information. The display 55 is a display unit that displays various types of information such as images.

[0032] The input unit 56 includes a keyboard 56a and a mouse 56b that accept input from an operator. The reader 57 reads information from a computer-readable recording medium 571, such as an optical disk, a magnetic disk, a magneto-optical disk, or a memory card. The display 55, the keyboard 56a, the mouse 56b, and the reader 57 are connected to the bus 50 via an interface I / F. The communication unit 58 transmits and receives signals to and from other components of the imaging device 1. The bus 50 is a signal circuit that connects the CPU 51, the GPU 59, the ROM 52, the RAM 53, the storage device 54, the display 55, the input unit 56, the reader 57, and the communication unit 58.

[0033] In the imaging device 1, a program 572 is read in advance from a recording medium 571 via a reading device 57 and stored in the storage device 54. The program 572 may be computer-readable and may be stored in the storage device 54 via a network, for example. The CPU 51 and the GPU 59 execute arithmetic processing in accordance with the program 572 while using the RAM 53 and the storage device 54. The CPU 51 and the GPU 59 function as a calculation unit in the imaging device 1. Other components that function as a calculation unit may be employed in addition to the CPU 51 and the GPU 59.

[0034] FIG. 4 is a block diagram showing the functional configuration of the control unit 5, which is realized by the computer executing arithmetic processing and the like according to the program 572. The control unit 5 has an image processing unit 500. The image processing unit 500 is an image processing device that generates an all-in-focus image and divides an analysis target region in the all-in-focus image, as described below. The image processing unit 500 includes a storage unit 501, an image generation unit 502, a target region identification unit 503, a region division unit 504, an unsuitable region identification unit 505, a region integration unit 506, and a repeat control unit 507. Details of these functions will be described later. All or part of the functions of the image processing unit 500 may be realized by a dedicated electrical circuit, or each function may be realized by a separate program. The image processing unit 500 may also be realized by multiple computers.

[0035] Next, the processing for generating an all-in-focus image and dividing the analysis target region in the all-in-focus image will be described with reference to FIG. 5. As described above, the operation of the imaging device 1 in FIG. 1 is controlled by the control unit 5. First, the illumination unit moving mechanism 15 and the imaging unit moving mechanism 16 are driven, and the illumination unit 12 and the imaging unit 13 are positioned above and below one well 21, respectively. Next, the lifting mechanism 14 is driven, and the vertical position of the imaging unit 13 is adjusted. Then, an image of the sample 9 is acquired by the imaging unit 13. The captured image is sent to the image processing unit 500 (see FIG. 4) and stored in the storage unit 501. In the imaging device 1, the vertical position of the imaging unit 13 is changed by the lifting mechanism 14, and the imaging unit 13 repeatedly captures an image of the sample 9 (i.e., acquires a captured image).

[0036] FIG. 6 is a longitudinal cross-sectional view of one well 21. In the example of FIG. 6, the focal position on the optical axis J1 of the imaging unit 13 is set to a plurality of positions H0 to H5 indicated by black circles in FIG. 6, and images of the sample 9 are respectively taken. In this manner, a plurality of captured images are acquired by imaging the sample 9 while changing the focal position along the optical axis J1. The plurality of captured images are stored and prepared in the storage unit 501 (step S11). Typically, the plurality of positions H0 to H5 are arranged at equal intervals in the vertical direction (i.e., the Z direction). In an actual imaging device 1, a plurality of captured images are acquired by sequentially setting the focal position at a number of positions (e.g., several tens of positions) on the optical axis J1 that are significantly more than those in the example of FIG. 6 (e.g., several tens of positions). In this specification, the height direction refers to the direction along the optical axis J1 and approaching the imaging unit 13 (the (-Z) direction in FIG. 6).

[0037] Next, the image generation unit 502 generates an all-in-focus image and a height map (step S12). FIG. 7 is a diagram for explaining the generation of an all-in-focus image by the image generation unit 502. The leftmost part of FIG. 7 shows multiple captured images G10 to G12. The second top row from the left shows an array M11 of image reference values, which will be described later, and the bottom row shows an array M12 of reference sharpness, which will be described later. The third row from the left shows the all-in-focus image G1, and the rightmost row shows an analysis target region R1, which will be described later. Here, a process for generating the all-in-focus image G1 and a height map using three captured images G10 to G12 acquired with focus positions set at three different positions will be described.

[0038] In each of the plurality of captured images G10 to G12, pixels are arranged in the row and column directions. In the plurality of captured images G10 to G12, pixels having the same row and column positions (hereinafter referred to as "pixel positions") indicate positions that overlap with each other in the direction of the optical axis J1 within the well 21. Consecutive numbers are assigned to the plurality of captured images G10 to G12, and the three captured images G10 to G12 in Figure 7 are assigned the numbers "0," "1," and "2," respectively.

[0039] Once multiple captured images G10-G12 are prepared, the sharpness of each pixel position is calculated for each of the multiple captured images G10-G12. Sharpness is an index that indicates the clarity of the image at the pixel position and its vicinity, and here, the higher the clarity, the greater the sharpness. The sharpness is, for example, edge strength or brightness variation, and is typically calculated based on the brightness change of pixels in an area of ​​a predetermined size centered on the pixel position. The calculation of sharpness may use the brightness variance of pixels surrounding the pixel position, the maximum brightness value, the minimum brightness value, the brightness value of the pixel position, etc.

[0040] When multiple sharpness indices are calculated for each pixel position from the multiple captured images G10 to G12, the image generation unit 502 compares the multiple sharpness indices at that pixel position. Then, the number of the captured image with the maximum sharpness among the multiple sharpness indices is determined as the image reference value for that pixel position. The image reference value is the number of the captured image to be referenced when determining the luminance value of that pixel position in the omnifocus image. For example, if, for a certain pixel position, the sharpness of the captured image G11 is the maximum among the multiple sharpness indices in the multiple captured images G10 to G12, the image reference value for that pixel position is "1."

[0041] As mentioned above, the second top row from the left in Fig. 7 shows the array M11 of image reference values ​​at multiple pixel positions. For ease of explanation, it is assumed here that each of the captured images G10 to G12 comprises 5 x 5 pixels, or 25 pixels. Focusing on the central and its eight neighboring pixel positions, in the example of Fig. 7, the sharpness of captured image G11 is high, and in the array M11 of image reference values, the image reference values ​​at these pixel positions are "1".

[0042] In addition, the sharpness corresponding to the image reference value at each pixel position (hereinafter referred to as "reference sharpness") is also identified. As described above, the second bottom row from the left in FIG. 7 shows an array M12 of reference sharpness at multiple pixel positions. The reference sharpness at each pixel position is the sharpness at that pixel position in the captured image indicated by the image reference value at that pixel position. For example, in the array M11 of image reference values, at a pixel position where the image reference value is "1", the sharpness at that pixel position in the captured image G11 becomes the reference sharpness. In the array M12 of reference sharpness in FIG. 7, the larger the reference sharpness, the darker the color of the pixel position.

[0043] Next, the image generation unit 502 calculates the luminance value of each pixel position of the omnifocus image. When calculating the luminance value of each pixel position, image reference values ​​of pixel positions surrounding the pixel position are taken into consideration (referenced). For example, the luminance value of each pixel position of the omnifocus image is calculated using Equation 1.

[0044]

number

[0045] In Equation 1, the row and column directions are represented by the x and y directions, respectively, and V(xn,yn) is the luminance value of a pixel position (hereinafter referred to as the "target pixel position") at any coordinate (xn,yn) in the omni-focus image. k and l are the distances in the x and y directions between the pixel position referenced with respect to the target pixel position and the target pixel position. fx and fy are the maximum values ​​of the distances in the x and y directions, and indicate the range of pixel positions referenced with respect to the target pixel position. I(A(xn+k,yn+l),xn,yn) is the luminance value of the target pixel position in the captured image indicated by the image reference value A(xn+k,yn+l) at the pixel position with coordinate (xn+k,yn+l). S(xn+k,yn+l) is the reference sharpness at the pixel position with coordinate (xn+k,yn+l). Here, the reference sharpness is assumed to be normalized to a value between 0 and 1. σ d is the distance weighting factor, and σ s is a weighting coefficient for the reference sharpness. In Equation 1, the weighting amount for the distance and the weighting amount for the reference sharpness are expressed as Gaussian coefficients.

[0046] In Equation 1, the luminance value of the pixel position of interest in the omni-focus image is calculated by reflecting the luminance value of the pixel position of interest in the captured image indicated by the image reference values ​​of each pixel position surrounding the pixel position of interest on the luminance value of the pixel position of interest in the captured image indicated by the image reference value of the pixel position of interest. In this way, the luminance value of the pixel position of interest in the omni-focus image is calculated using multiple luminance values ​​derived from the image reference values ​​of the pixel position of interest and the pixel positions surrounding the pixel position of interest.

[0047] In Equation 1, the luminance value of the captured image indicated by the image reference value of the surrounding pixel positions (luminance value of the pixel position of interest) is assigned a weight according to the distance between the surrounding pixel positions and the pixel position of interest and a weight according to the reference sharpness of the surrounding pixel positions. Specifically, the closer a pixel position is to the pixel position of interest, the greater the weight (the greater the influence on the luminance value of the omni-focus image). Furthermore, the higher the reference sharpness of the surrounding pixel positions, the greater the weight. The image generation unit 502 calculates the luminance values ​​of all pixel positions using Equation 1. This generates the omni-focus image G1 shown third from the left in FIG. 7. In the omni-focus image G1, the luminance value can be smoothly changed in the area where the image reference value switches. The image generation unit 502 may reflect the luminance values ​​of the surrounding pixel positions in the captured image indicated by the image reference value of each pixel position around the pixel position of interest in the luminance value of the pixel position of interest in the captured image indicated by the image reference value of the pixel position of interest.

[0048] Next, a height map is obtained that indicates the height in the direction of the optical axis J1 of the portion of the sample 9 that corresponds to each pixel position in the all-in-focus image. Here, the image reference value at each pixel position indicates the number of the captured image that is most in focus for that pixel position, and can therefore be regarded as the height of the sample 9 at that pixel position. Therefore, in one example, the array M11 of image reference values ​​in Figure 7 serves as the height map.

[0049] On the other hand, the luminance value of each pixel position in the omni-focus image G1 is calculated using not only the image reference value of the pixel position but also the image reference values ​​of the pixel positions surrounding the pixel position. Therefore, from the viewpoint of generating a height map that matches the omni-focus image G1, it is preferable to obtain a height (hereinafter referred to as a "synthetic height") that reflects the image reference values ​​of the pixel positions surrounding the pixel position for the image reference value of the pixel position. Specifically, the synthetic height for each pixel position is calculated using Equation 2.

[0050]

number

[0051] In Equation 2, as in Equation 1, the row and column directions are represented by the x and y directions, respectively, and D(xn,yn) is the composite height for a pixel position (i.e., a pixel position of interest) at any coordinate (xn,yn) in the omnifocus image G1. k and l are the distances in the x and y directions between the pixel position referenced for the pixel position of interest (i.e., a pixel position to which the image reference value is applied) and the pixel position of interest. fx and fy are the maximum values ​​of the distances in the x and y directions, and indicate the range of pixel positions referenced for the pixel position of interest (which can also be considered as the smoothing range). A(xn+k,yn+l) is the image reference value for the pixel position of coordinate (xn+k,yn+l). S(xn+k,yn+l) is the reference sharpness at the pixel position of coordinate (xn+k,yn+l). Here, the reference sharpness is assumed to be normalized to a value between 0 and 1. σ d is the distance weighting factor, and σ s is the weighting factor of the reference sharpness.

[0052] In Equation 2, the composite height of the pixel of interest is calculated by reflecting the image reference values ​​of the pixel of interest's surrounding pixel positions on the image reference value of the pixel of interest. In other words, the composite height of the pixel of interest is calculated using the image reference values ​​of the pixel of interest and a group of pixel positions including the pixel of interest and its surrounding pixel positions. At this time, the image reference values ​​of the surrounding pixel positions are assigned weights according to the distance between the pixel of interest and the surrounding pixel positions and weights according to the reference sharpness of the surrounding pixel positions. In particular, the closer a pixel position is to the pixel of interest, the greater the influence (weight) of its image reference value on the composite height. Furthermore, the higher the reference sharpness of the surrounding pixel positions, the greater the influence of its image reference value on the composite height.

[0053] The image generation unit 502 calculates the composite heights of all pixel positions using Equation 2. This generates a height map indicating the composite heights of all pixel positions. As with the luminance value of the all-in-focus image G1, the composite height of the pixel position of interest is calculated from the image reference values ​​of the pixel position group including the pixel position of interest and pixel positions surrounding the pixel position of interest, thereby obtaining an appropriate height map that matches the all-in-focus image G1. The composite height may be expressed as a height from a predetermined reference position in the well 21 based on the distance between the focal positions when the multiple captured images G10 to G12 are acquired in step S11. The all-in-focus image G1 and the height map are stored in the storage unit 501 and prepared for the following processing.

[0054] Next, the target region identifying unit 503 identifies and extracts each analysis target region R1 in the all-in-focus image G1 (step S13), as shown on the right side of FIG. 7. The analysis target region R1 is an area that represents a portion of the sample 9 in the all-in-focus image G1. If multiple portions of the sample 9 are dispersed in the all-in-focus image G1, the analysis target regions R1 are extracted individually. The analysis target regions R1 are identified using known techniques, such as binarization using a predetermined threshold or segmentation using deep learning.

[0055] In the sample 9 shown by the solid line in FIG. 6, cells and the like exist as a single mass in the well 21, so an appropriate analysis target region R1 with high circularity is extracted, as shown on the far right side of FIG. 7. The circularity can be calculated, for example, by (4π × (area) ÷ (perimeter) 2 ) On the other hand, as shown by the two-dot chain line in FIG. 6, when cells or the like exist as multiple clumps and the multiple clumps overlap in the direction of the optical axis J1, multiple captured images G20 to G22 shown on the leftmost side of FIG. 8 are acquired. In this case, the image generating unit 502 generates the omnifocus image G2 shown second from the left in FIG. 8. Note that in the omnifocus image G2 in FIG. 8, "0" is written in the center of the in-focus area (area with high sharpness) in the captured image G20, and "2" is written in the center of the in-focus area in the captured image G22.

[0056] In processing the all-in-focus image G2 by the target region identification unit 503, an analysis target region R2 shown on the far right in FIG. 8 is extracted. As mentioned above, the analysis target region R2 is made up of multiple overlapping regions, and it is not desirable to treat it as a single region. While it is possible to adopt a known region division method such as watershed, dividing the region can be difficult depending on the shape of the analysis target region R2. In the following steps S14 to S19, a method for appropriately dividing the analysis target region R2 in FIG. 8 will be described. If multiple analysis target regions are extracted, steps S14 to S19 are performed for each analysis target region.

[0057] FIG. 9 is a diagram showing a height map M2. The height map M2 is generated together with the omni-focus image G2 of FIG. 8 in the processing of step S12 by the image generation unit 502. In the height map M2 of FIG. 9, the greater the composite height, the darker the color of the pixel position. In addition, a set of pixel positions in the height map M2 included in the analysis target region R2 of FIG. 8 is surrounded by a thick line. In the following description, the set of pixel positions in the height map M2 will also be referred to as the "analysis target region R2." The same applies to the divided regions D1 to D3 and the integrated regions T1 and T2 described below.

[0058] The region dividing unit 504 generates a histogram of composite heights in the analysis target region R2 based on the height map M2 (step S14). FIG. 10 is a diagram showing a histogram of heights in the analysis target region R2. In FIG. 10, the horizontal axis represents composite height, and the vertical axis represents frequency. The histogram in FIG. 10 has multiple (three) peaks P1 to P3. Typically, the multiple peaks P1 to P3 exist discretely. Each peak P1 to P3 includes a local peak in the histogram and a bar around the peak (a portion where the frequency is greater than 0). The region dividing unit 504 acquires a set of pixel positions corresponding to each peak P1 to P3 as a divided region (step S15). In other words, a set of pixel positions whose composite height falls within the range of each peak P1 to P3 is acquired as one divided region. If the set of pixel positions is a plurality of regions separated from each other, each region may be treated as a divided region. The process of step S15 can also be considered as a process of acquiring divided regions by performing binarization processing for each range of the peaks P1 to P3 in the height map M2.

[0059] FIG. 11 is a diagram showing a plurality of divided regions D1 to D3 in the analysis target region R2. In FIG. 11, the plurality of divided regions D1 to D3 are marked with different parallel diagonal lines. As described above, in the processing of steps S14 and S15 by the region dividing unit 504, the analysis target region R2 is divided into a plurality of divided regions D1 to D3 according to the composite height indicated by the height map M2. In the following description, the divided regions D1 to D3 corresponding to the three peaks P1 to P3 in FIG. 10 are also referred to as the "first divided region D1," the "second divided region D2," and the "third divided region D3," respectively. Note that if only one peak is identified in the histogram and multiple divided regions are not obtained (for example, the example of FIG. 7), the following processing is not performed.

[0060] Next, the unsuitable region identifying unit 505 determines whether each of the divided regions D1 to D3 is a suitable region that satisfies predetermined judgment conditions (step S16). The judgment conditions for a suitable region can use general feature quantities or combinations thereof. For example, if each divided region D1 to D3 satisfies both the first and second judgment conditions, the divided region is determined to be a suitable region. If the divided region D1 to D3 does not satisfy either the first or second judgment conditions, the divided region is determined to be an unsuitable region. The first judgment condition is that, when a rotational ellipsoid approximation (an ellipse approximation in which the orientation of the major axis is changeable) is performed on each divided region D1 to D3, the length of the minor axis is equal to or greater than a first threshold. The second judgment condition is that the circularity of each divided region D1 to D3 is equal to or greater than a second threshold. A divided region that satisfies both the first and second judgment conditions (i.e., a suitable region) is a region that is relatively large in size and nearly circular.

[0061] As shown in FIG. 12, in the analysis target region R2, the divided regions D1 and D3 marked with "OK" inside are suitable regions, and the divided region D2 marked with "NG" inside is unsuitable. Step S16 by the unsuitable region identifying unit 505 can be said to be a process of identifying unsuitable regions among the multiple divided regions D1 to D3 that do not satisfy a predetermined judgment condition. If no unsuitable region is identified in step S16, the subsequent processes are not performed. The above judgment conditions may also use other feature quantities related to the size of each divided region D1 to D3, such as the area, diameter, or perimeter. Furthermore, the variation (standard deviation) of the brightness values ​​within each divided region D1 to D3 in the omni-focus image G2 or the variation of the composite height within each divided region D1 to D3 in the height map M2 may also be used. In this case, the smaller the variation, the more likely it is that a region is judged to be suitable.

[0062] The region integration unit 506 calculates a judgment score when each unsuitable region is integrated with each of the other divided regions that contact the unsuitable region. In the example of FIG. 12, the divided region D2, which is an unsuitable region, contacts both the divided region D1 and the divided region D3. Therefore, a judgment score is calculated for the integrated region T1 obtained by integrating the divided region D2 and the divided region D1 as shown in FIG. 13A, and a judgment score is calculated for the integrated region T2 obtained by integrating the divided region D2 and the divided region D3 as shown in FIG. 13B. A general feature value or a combination thereof can be used as the judgment score. In one example, circularity is used as the judgment score.

[0063] In the examples of FIGS. 13A and 13B, the judgment score for integrated region T1 is greater than the judgment score for integrated region T2, so the integrated region T1 in FIG. 13A is selected as the new segmented region. By integrating the unsuitable region with other segmented regions based on the judgment score, a new segmented region D4 is obtained as shown in FIG. 14 (step S17). The judgment score may incorporate the range (max-min) and variation (standard deviation) of the composite height within each integrated region T1, T2 in the height map M2. In this case, the smaller the range and variation, the larger the judgment score, and an integrated region with a uniform composite height is more likely to be selected as the new segmented region. Furthermore, the range and variation of the luminance values ​​within each integrated region T1, T2 in the omnifocal image G2 may be incorporated into the judgment score. In this case, an integrated region that is less visually unnatural is more likely to be selected as the new segmented region.

[0064] When a new divided region D4 is acquired, the unsuitable region identifying unit 505 determines whether the new divided region D4 is a suitable region that satisfies a predetermined determination condition (step S18). The determination condition in step S18 may be the same as or different from the determination condition in step S16. In this processing example, only the first determination condition is used. That is, when a rotational ellipsoid approximation is performed on the new divided region D4, if the length of the minor axis is equal to or greater than a predetermined threshold, the divided region D4 is determined to be a suitable region, and if the length of the minor axis is less than the threshold, the divided region D4 is determined to be an unsuitable region. In this case, even if the circularity is low, a divided region D4 of an appropriate size is more likely to be determined to be a suitable region.

[0065] In the example of FIG. 14, the new divided area D4 is determined to be a suitable area, and the analysis target area R2 is divided into a suitable divided area D3 and a suitable divided area D4. As described below, the unsuitable divided area is repeatedly integrated with other divided areas in step S17. However, by selecting an appropriately sized divided area D4 as the suitable area, the generation of an excessively large divided area is prevented. Step S18, performed by the unsuitable area identification unit 505, can be considered a process of identifying the new divided area as a new unsuitable area if the new divided area does not satisfy a predetermined determination condition. The determination condition in step S18 may also include various feature quantities, similar to the determination condition in step S16.

[0066] The repeat control unit 507 checks whether a predetermined termination condition is satisfied. The termination condition is that all divided regions are suitable regions or that no other divided regions contact the unsuitable regions (i.e., the analysis target region R2 is one divided region). In the example of FIG. 14, all divided regions D3 and D4 included in the analysis target region R2 are suitable regions, and the termination condition is satisfied, so image processing by the image processing unit 500 terminates (step S19). The divided regions D3 and D4 correspond to, for example, multiple clumps of cells or the like indicated by the two-dot chain lines in FIG. 6, and are analyzed individually. If the analysis target region R2 includes an unsuitable region and there is another divided region contacting the unsuitable region, the above steps S17 and S18 are repeated (step S19). When all divided regions are suitable regions or there are no other divided regions contacting the unsuitable region, this image processing terminates (step S19).

[0067] As described above, the image processing device (image processing unit 500 in the above example) includes a storage unit 501 that stores the all-in-focus image G2 and the height map M2, a target region identification unit 503 that identifies an analysis target region R2 in the all-in-focus image G2, a region division unit 504 that divides the analysis target region R2 into multiple divided regions D1-D3 according to the height indicated by the height map M2, an unsuitable region identification unit 505 that identifies unsuitable regions among the multiple divided regions D1-D3 that do not satisfy a predetermined judgment condition, and a region integration unit 506 that calculates a judgment score for integrating an unsuitable region with other divided regions that contact the unsuitable region and integrates the unsuitable region with the other divided regions based on the judgment score to obtain a new divided region D4. This allows the analysis target region R2 to be appropriately divided based on the height distribution, etc., even when multiple clusters of cells or the like are widely distributed in the height direction in the sample 9 and the regions of multiple analysis targets (clusters of cells or the like) overlap in the analysis target region R2, as in the all-in-focus image G2 of FIG. 8 . Furthermore, by appropriately determining evaluation indices such as a judgment score and judgment conditions, it is possible to extract a divided region having a desired shape as an analysis region, thereby enabling the sample 9 to be appropriately analyzed.

[0068] A preferred image processing device further includes an image generating unit 502 that generates an all-in-focus image G2 and a height map M2 based on a plurality of captured images acquired by capturing an image of an object (the sample 9 in the above example) while changing the focal position along the optical axis J1. This makes it possible to prepare an appropriate all-in-focus image G2 and height map M2.

[0069] Preferably, the region dividing unit 504 generates a height histogram for the analysis region R2 based on the height map M2, and obtains a set of pixel positions corresponding to each of the peaks P1 to P3 in the histogram as divided regions D1 to D3. This makes it possible to accurately obtain the divided regions D1 to D3 based on the heights indicated by the height map M2.

[0070] Preferably, when a new divided area acquired by the area integration unit 506 does not satisfy a predetermined judgment condition, the unsuitable area identification unit 505 identifies the new divided area as a new unsuitable area. This prevents the new divided area from being left as an unsuitable area by performing further integration processing when the new divided area becomes an unsuitable area. As a result, the analysis target area R2 can be more appropriately divided. The unsuitable area identification unit 505 may refer to the judgment score and classify the new divided area as a suitable area or an unsuitable area based on the judgment score.

[0071] Preferably, the height indicated by the height map M2 is used in the identification of unsuitable regions by the unsuitable region identification unit 505 and / or in the calculation of the judgment score by the region integration unit 506. This makes it possible to identify divided regions that do not have a uniform height as unsuitable regions and / or to appropriately obtain new divided regions that have a substantially uniform height.

[0072] In the above processing example by the image generation unit 502, only one image reference value is determined for each pixel position in step S12. However, as in JP 2022-51094 A (the above-mentioned Patent Document 2), M image reference values ​​(first through Mth image reference values) corresponding to the top M sharpness values ​​(where M is an integer greater than or equal to 2 and less than the number of captured images) may be determined. In this case, a formula similar to Equation 1 is used to calculate the luminance value of each pixel position in the omnifocal image using multiple luminance values ​​derived from the first through Mth image reference values ​​of a pixel position group including each pixel position and pixel positions surrounding the pixel position. Furthermore, a formula similar to Equation 2 is used to calculate the composite height of each pixel position in the height map using the first through Mth image reference values ​​of a pixel position group including each pixel position and pixel positions surrounding the pixel position.

[0073] As described above, the image generation unit 502 compares multiple sharpness values ​​for each pixel position, and determines the number of at least one captured image to be referenced in determining the luminance value of that pixel position in the omnifocus image as the image reference value. This makes it possible to generate a smooth omnifocus image and an appropriate height map that matches the omnifocus image using the image reference value. Note that the above equations 1 and 2 used in the image generation unit 502 are merely examples and may be modified as appropriate.

[0074] The image processing device, image processing method, and imaging device 1 described above can be modified in various ways.

[0075] The divided regions may be acquired without using a histogram in the region dividing unit 504. For example, the height ranges of the first divided region, the second divided region, and the third divided region may be determined in advance, and regions within each range in the height map may be acquired as divided regions.

[0076] 1, the focal position changing mechanism that changes the focal position of the imaging unit 13 along the optical axis J1 is realized by an elevation mechanism 14 that moves the imaging unit 13, but the focal position changing mechanism may also be realized by a mechanism that moves some of the lenses in the imaging unit 13 along the optical axis J1. Also, a mechanism that moves the sample 9 along the optical axis J1 may be provided as the focal position changing mechanism.

[0077] In the above embodiment, an all-in-focus image is used as the target image for processing by the image processing device. However, the target image is not limited to an all-in-focus image and may be, for example, a single captured image. The target image may be any image showing an object viewed from a predetermined direction. Furthermore, a length measuring device such as a laser length measuring device may be used to obtain the height in a predetermined direction of a portion of the object corresponding to each pixel position in the target image, and a height map indicating the height may be obtained. When an all-in-focus image is used, the predetermined direction is a direction parallel to the optical axis J1 of the imaging unit 13.

[0078] In the above embodiment, a transmission image of the sample 9 is acquired as the captured image, but a fluorescent image or the like of the sample 9 may also be acquired as the captured image. Furthermore, the object may be something other than a cell or the like.

[0079] The image processing unit 500 (image processing device) may be used independently of the imaging device 1.

[0080] The configurations in the above-described embodiment and each modification may be combined as appropriate as long as they are not mutually contradictory. [Explanation of symbols]

[0081] 1. Imaging device 5. Control section 9 Samples 12 Lighting Department 13 Imaging unit 14 Lifting mechanism 500 Image Processing Unit 501 Storage section 502 Image generation unit 503 Target Area Identification Unit 504 Area division part 505 Nonconformity area identification section 506 Domain Integration Department 572 Programs D1~D3 divided area D4 New division area G1,G2 Full focus image G10~G12, G20~G22 Captured images J1 optical axis M2,M11 height maps P1~P3 Mountain area R1,R2 Analysis area S11~S19 steps

Claims

1. 1. An image processing method, comprising: a) preparing a target image showing an object viewed from a predetermined direction and a height map showing heights in the predetermined direction of portions of the object corresponding to each pixel position in the target image; b) identifying an analysis target region in the target image; c) dividing the analysis target area into a plurality of divided areas according to the height indicated by the height map; d) identifying a non-conforming area that does not satisfy a predetermined judgment condition from among the plurality of divided areas; e) calculating a judgment score when the unsuitable region is integrated with each of the other divided regions that contact the unsuitable region, and integrating the unsuitable region with the other divided regions based on the judgment score to obtain a new divided region; An image processing method comprising:

2. 2. The image processing method according to claim 1, The step a) a1) preparing a plurality of captured images obtained by capturing images of the object while changing a focal position along an optical axis parallel to the predetermined direction; a2) generating an all-in-focus image as the target image and the height map based on the plurality of captured images; An image processing method comprising:

3. 2. The image processing method according to claim 1, The step c) is generating a height histogram for the analyzed region based on the height map; obtaining a set of pixel positions corresponding to each peak in the histogram as a divided region; An image processing method comprising:

4. 2. The image processing method according to claim 1, The image processing method further comprises a step of identifying the new divided area acquired in the step e) as a new unsuitable area when the new divided area does not satisfy a predetermined determination condition.

5. 5. An image processing method according to claim 1, further comprising: An image processing method, wherein the height indicated by the height map is used in identifying the unconformity region in the step d) and / or in calculating the judgment score in the step e).

6. An image processing device, a storage unit that stores a target image showing an object viewed from a predetermined direction and a height map that shows heights in the predetermined direction of portions of the object corresponding to each pixel position of the target image; a target region specifying unit that specifies an analysis target region in the target image; a region dividing unit that divides the analysis target region into a plurality of divided regions according to the heights indicated by the height map; an unsuitable region identifying unit that identifies an unsuitable region that does not satisfy a predetermined determination condition among the plurality of divided regions; a region integration unit that calculates a judgment score when the unsuitable region is integrated with other divided regions that contact the unsuitable region, and integrates the unsuitable region with the other divided regions based on the judgment score to obtain a new divided region; An image processing device comprising:

7. 7. The image processing device according to claim 6, The image processing device further includes an image generation unit that generates an all-in-focus image, which is the target image, and the height map based on a plurality of captured images obtained by capturing images of the target object while changing the focal position along an optical axis parallel to the predetermined direction.

8. 7. The image processing device according to claim 6, The image processing device wherein the region dividing unit generates a histogram of heights in the region to be analyzed based on the height map, and obtains a set of pixel positions corresponding to each peak in the histogram as a divided region.

9. 7. The image processing device according to claim 6, The image processing device wherein, when the new divided area acquired by the area integration unit does not satisfy a predetermined determination condition, the unsuitable area identification unit identifies the new divided area as a new unsuitable area.

10. 7. The image processing device according to claim 6, The image processing device wherein the height indicated by the height map is used in the identification of the unsuitable region by the unsuitable region identification unit and / or in the calculation of the judgment score by the region integration unit.

11. An imaging device, An image processing device according to any one of claims 6 to 10; an imaging unit that images the object; an illumination unit that emits light toward the object; a focal position changing mechanism that changes the focal position of the imaging unit along an optical axis; An imaging device comprising:

12. A computer-readable program that causes a computer to perform image processing, wherein execution of the program by a computer causes the computer to: a) preparing a target image showing an object viewed from a predetermined direction and a height map showing heights in the predetermined direction of portions of the object corresponding to each pixel position in the target image; b) identifying an analysis target region in the target image; c) dividing the analysis target area into a plurality of divided areas according to the height indicated by the height map; d) identifying a non-conforming area that does not satisfy a predetermined judgment condition from among the plurality of divided areas; e) calculating a judgment score when the unsuitable region is integrated with each of the other divided regions that contact the unsuitable region, and integrating the unsuitable region with the other divided regions based on the judgment score to obtain a new divided region; A computer-readable program that causes a

Images