Image processing device, image processing method, and image processing program
The image processing device converts bright field images into pseudo phase contrast images using a trained model and noise removal techniques, effectively addressing the issue of high background noise in microscopic images by separating foreground and background, thereby enhancing image quality and noise distinction.
Patent Information
- Application Number
- PCT/JP2024/016259
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-02
- Filing Date
- 2024-04-25
- Publication Date
- 2024-11-07
AI Technical Summary
Microscopic images obtained by different observation methods, such as bright field and phase contrast, often result in high-contrast pseudo images that enhance cellular structures but also increase background noise visibility, making it difficult to distinguish between cell structures and noise.
An image processing device and method that converts bright field images into pseudo phase contrast images using a trained model, followed by mask image generation and noise removal, where a continuous mask image is used to separate foreground and background, allowing for noise reduction while maintaining the appearance of biological samples.
The solution effectively reduces background noise in pseudo images, improving the visibility of cellular structures without altering the appearance of the biological samples, thus enhancing image quality and noise distinction.
Smart Images

Figure JP2024016259_07112024_PF_FP_ABST
Abstract
Description
Image processing device, image processing method, and image processing program
[0001] The present invention relates to an image processing device, an image processing method, and an image processing program.
[0002] Patent Document 1 describes a method for generating a trained model for image conversion, an image processing method, an image conversion device, and a program. [Prior art documents] [Patent documents] [Patent document 1] International Publication No. 2022 / 113366 General disclosure
[0003] A first aspect of the present invention provides an image processing device comprising: an image conversion unit that converts an input image, which is a microscopic image including a biological sample acquired using a first observation method, into a pseudo image that has an appearance as if acquired using a second observation method different from the first observation method; a mask image generation unit that generates, based on the input image, at least one successive mask image, which includes a first region corresponding to a foreground including the biological sample, a second region corresponding to a background not including the biological sample, and a boundary region between the first and second regions, with pixel values in the boundary region gradually changing from the first region to the second region; and a noise reduction unit that generates, based on the successive mask images and the pseudo image, an output image in which background noise has been removed while maintaining the appearance of the biological sample.
[0004] In a second aspect of the present invention, there is provided an image processing method, comprising: an image conversion step of converting an input image, which is a microscopic image including a biological sample acquired using a first observation method, into a pseudo image that has an appearance as if acquired using a second observation method different from the first observation method; a mask image generation step of generating, based on the input image, at least one successive mask image, which includes a first region corresponding to a foreground including the biological sample, a second region corresponding to a background not including the biological sample, and a boundary region between the first and second regions, with pixel values in the boundary region gradually changing from the first region to the second region; and a noise removal step of generating, based on the successive mask images and the pseudo image, an output image in which background noise has been removed while maintaining the appearance of the biological sample.
[0005] A third aspect of the present invention provides an image processing program that causes a computer to execute: an image conversion procedure for converting an input image, which is a microscopic image containing a biological sample acquired using a first observation method, into a pseudo image that appears to be acquired using a second observation method different from the first observation method; a mask image generation procedure for generating, based on the input image, at least one continuous mask image, the continuous mask image comprising a first region corresponding to a foreground containing the biological sample, a second region corresponding to a background not containing the biological sample, and a boundary region between the first and second regions, the pixel values within the boundary region gradually changing from the first region to the second region; and a noise removal procedure for generating, based on the continuous mask image and the pseudo image, an output image from which background noise has been removed while maintaining the appearance of the biological sample.
[0006] The above summary of the invention does not list all of the necessary features of the present invention, and subcombinations of these features may also constitute inventions.
[0007] 1 shows a schematic configuration of an image processing device 100 in this embodiment. (a) shows a bright-field image that is an input image, and (b) shows a pseudo image after image conversion by the image conversion unit 11. (a) to (d) show an example of mask generation processing by the mask image generation unit 13. (a) to (e) show an example of noise removal processing by the noise removal unit 14. (a) to (c) show the effect of an edge continuity mask in this embodiment. An example of a GUI screen 200 in this embodiment is shown. A flowchart showing the operation of the image processing device 100 in this embodiment. (a) and (b) are diagrams for explaining another embodiment of the mask image generation unit 13. (a) and (b) are diagrams for explaining another embodiment of the mask image generation unit 13. A schematic configuration of an image processing device 110 in another embodiment is shown. An example of a computer 2200 is shown.
[0008] The present invention will be described below through embodiments of the invention. The following embodiments do not limit the scope of the invention. Not all of the combinations of features described in the embodiments are necessarily essential to the solution of the invention.
[0009] Fig. 1 shows a schematic configuration of an image processing device 100 according to this embodiment. As shown in Fig. 1, the image processing device 100 according to this embodiment includes an image conversion unit 11, a mask image generation unit 13, a noise removal unit 14, and a display control unit 15. The image conversion unit 11 includes a trained model 12. The image processing device 100 is a device that performs image processing on a microscopic image obtained by microscopic observation.
[0010] An input image, which is a microscope image acquired by a first observation method in microscope observation, is input to the image conversion unit 11. The input image may be the microscope image itself, or may be a microscope image that has undergone image processing (e.g., preprocessing such as shading correction or smoothing of a partial region of the image) different from the image conversion described below. In this embodiment, the first observation method is reflected bright-field observation, and the input image is, for example, a reflected bright-field image acquired by a bright-field microscope (hereinafter referred to as a bright-field image). Note that the first observation method may also be transmitted bright-field observation, fluorescent observation, or dark-field observation, in which case the input image is a transmitted bright-field image, fluorescent image, or dark-field image acquired by each observation method.
[0011] The biological sample to be observed under a microscope is not particularly limited and may be derived from multicellular organisms such as animals or plants, or from unicellular organisms such as bacteria, including, for example, unstained cultured cells, tissues, and organs. Specific examples include living cells cultured in a monolayer or multilayer in a culture vessel, or cultured in suspension as single cells or cell clumps. The region of interest within the object to be observed with particular attention may be an entire cell, or an intracellular organelle such as a nucleolus or a cell membrane. The culture vessel may be, for example, a vessel commonly used for cell culture, such as a dish or well plate, or a biofunctional chip (organ-on-a-chip).
[0012] The image converter 11 converts an input image, which is a microscopic image acquired by a first observation method, into a pseudo image that appears as if it were acquired by a second observation method that is different from the first observation method. In this embodiment, the second observation method is phase-contrast observation, and the pseudo image is a pseudo phase-contrast image.
[0013] The image conversion unit 11 has a trained model 12. The trained model 12 is a model that has learned a mapping from a source image to a target image. In this embodiment, the trained model 12 is trained by an image conversion network using bright-field images captured by bright-field observation as a source image group and phase-contrast images as target images. The image conversion network is not particularly limited, and examples include "convolutional neural networks (CNN)," "generative adversarial networks (GAN)," "CGAN (Conditional GAN)," "DCGAN (Deep Conventional GAN)," "Pix2Pix," and "CycleGAN." Details of a method for generating the trained model 12 are omitted in this specification. For example, this is described in Patent Document 1, but is not limited thereto.
[0014] The image conversion unit 11 inputs a bright-field image, which is an input image, to the trained model 12. The trained model 12 converts the bright-field image and outputs a pseudo image. By inputting the bright-field image to the trained model 12, the arrangement of objects in the bright-field image remains unchanged, and an image is obtained in which the appearance, i.e., the style, is converted into a phase-contrast image. This type of image conversion is also called style conversion. As shown in FIG. 1 , the image conversion unit 11 outputs a pseudo phase-contrast image (called a pseudo image) generated by the image conversion to the noise removal unit 14.
[0015] FIG. 2( a ) shows a bright-field image, which is the input image, and FIG. 2( b ) shows a pseudo image after image conversion by the image conversion unit 11. Phase-contrast images often have higher contrast than bright-field images, making it easier to visually observe cell structures, etc. Therefore, by converting the bright-field image of FIG. 2( a ) into a pseudo image that looks like the phase-contrast image of FIG. 2( b ), the contrast of the image is increased, as shown in the figure, making it easier to visually observe cell structures, etc. However, there is a drawback in that the contrast of background areas other than the cell structures is also increased, making noise more noticeable. The present embodiment aims to resolve this drawback.
[0016] 3(a) to 3(d) show an example of the mask image generation process by the mask image generation unit 13. Fig. 3(a) shows a bright-field image that is an input image, Fig. 3(b) shows an edge image, Fig. 3(c) shows an edge image after distance transformation, Fig. 3(d) shows a mask image for foreground extraction, and Fig. 3(e) shows a mask image for background extraction.
[0017] The mask image generation unit 13 generates, based on the input image, at least one continuous mask image, which includes a first region corresponding to the foreground including a biological sample, a second region corresponding to the background not including a biological sample, and a boundary region between the first and second regions, with pixel values within the boundary region gradually changing from the first region to the second region. The mask image generation unit 13 generates a foreground extraction mask image based on the input bright-field image, which extracts the foreground from the pseudo image while maintaining its appearance. The mask image generation unit 13 further generates a background extraction mask image based on the input bright-field image, which extracts a region corresponding to the background from the background image. Details of the processing performed by the mask image generation unit 13 are described below with reference to Figures 3(a) to 3(d). In this specification, the foreground refers to a region of interest in the input image or pseudo image that includes cells of interest, and the background refers to a region of interest that does not include cells of interest, in the input image or pseudo image.
[0018] First, the mask image generation unit 13 performs edge detection processing on the input bright-field image ( FIG. 3A ) to detect areas where pixel values change significantly. A commonly known method may be used for the edge detection processing. In this embodiment, for example, the Canny filter method is used. The Canny filter method obtains the results of first-order differential filtering in the vertical and horizontal directions within the edge detection area, and determines the gradient and direction of the edge from the two differential images. In addition to the Canny filter method, the Sobel filter method or the Prewitt filter method may also be used. As shown in FIG. 3B , the mask image generation unit 13 generates an edge image displaying the detected edges as a result of the edge detection processing.
[0019] As shown in FIG. 3C, the mask image generation unit 13 performs distance transformation processing on the edge image to generate a distance-transformed image (referred to as a distance-transformed image). In this embodiment, the distance transformation processing is a process of assigning a difference from the pixel value of the edge, one by one, depending on the distance from the edge. For example, the pixel value of the edge is set to a minimum value, and a larger pixel value is assigned as the distance from the edge increases. This makes it possible to blur the area around the edge. Here, the parameter indicating the relationship between the distance from the edge and the pixel value can be set appropriately depending on the magnification of the objective lens and the type of object to be imaged. For example, it is possible to set an image with a clear edge or, conversely, a blurred edge.
[0020] 3(d) and 3(e), based on the distance transformation image, the mask image generation unit 13 generates a foreground extraction mask image (FIG. 3(d)) in which the pixel values of the region corresponding to the foreground (referred to as the foreground-corresponding region) are set to 255, the pixel values of the region corresponding to the background (referred to as the background-corresponding region) are set to 0, and the pixel values of the boundary region between the foreground-corresponding region and the background-corresponding region are set to gradually decrease by a predetermined value from the foreground-corresponding region toward the background-corresponding region (according to the distance from the edge), and a background extraction mask image (FIG. 3(e)) in which the pixel values of the background-corresponding region are set to 255, the pixel values of the foreground-corresponding region are set to 0, and the pixel values of the boundary region between the foreground-corresponding region and the background-corresponding region are set to gradually increase by a predetermined value from the foreground-corresponding region toward the background-corresponding region (according to the distance from the edge). In FIGS. 3(d) and 3(e), pixel values of 255 are represented by white, and pixel values of 0 are represented by black.
[0021] In this case, the sum of the pixel values of each pixel in the foreground extraction mask image and the background extraction mask image is set to 255 (a constant value) (256 gradations in an 8-bit image). In this case, the foreground extraction mask image and the background extraction mask image can be said to have a relationship in which the black and white of each pixel is inverted. In response to the blurring around the edges in the distance-transformed image described above, in both the foreground extraction mask image and the background extraction mask image, pixel values change in steps of a predetermined value depending on the distance from the edge within the boundary region between the region where the pixel value is 255 and the region where the pixel value is 0. For example, in the foreground extraction mask, the pixel value decreases the farther the distance from the detected edge, and increases the pixel value the closer the distance from the detected edge. If the predetermined value is 20, the width of the boundary region will be 12 pixels, and if the predetermined value is 50, the width of the boundary region will be 5 pixels. The size of the region changes depending on the predetermined value. A mask image in which pixel values change in steps of a predetermined value depending on the distance from the edge is called an edge-continuous mask image.
[0022] A mask image in which pixel values in a boundary region between a region where pixel values are set to 255 and a region where pixel values are set to 0 change stepwise by a predetermined value from one region to the other (according to the distance from a predetermined position) is called a continuous mask image. The continuous mask image is a concept that includes an edge continuous mask image, a mask image described in Alternative Embodiment 1, which will be described later, and a mask image described in Alternative Embodiment 4. The predetermined position is an edge in the case of an edge continuous mask image, a contour of a cell segmented region (a region where cells exist) in the case of the mask image described in Alternative Embodiment 1, and a boundary where pixel values of a binary mask image differ in the case of the mask image described in Alternative Embodiment 4.
[0023] Furthermore, the predetermined value may not be a fixed value but may be a variable value (the change width is not constant but varies), and an arbitrary value or a changed pixel value may be set depending on the distance from the predetermined position. As shown in Figure 1, the mask image generation unit 13 outputs the generated foreground extraction mask image and background extraction mask image to the noise removal unit 14. Based on the continuous mask image and the pseudo image, the noise removal unit 14 generates an output image in which background noise has been removed while maintaining the appearance of the biological sample.
[0024] 4A to 4E show an example of noise removal processing by the noise removal unit 14. FIG. 4A shows a pseudo image, FIG. 4B shows a foreground extraction mask image, FIG. 4C shows a background image, FIG. 4D shows a background extraction mask image, and FIG. 4E shows an output image after noise reduction. As shown in FIG. 1 , the noise removal unit 14 receives a pseudo image from the image conversion unit 11 and a foreground extraction mask image and a background extraction mask image from the mask image generation unit 13. Using the foreground extraction mask image, the noise removal unit 14 generates a first intermediate image in which the pixel values of the foreground in the pseudo image are maintained as they are, the pixel values of the background are set to zero (pure black), and the pixel values of the boundary region between the foreground and background are set to values that take into account the pixel values of the foreground extraction mask image. The noise removal unit 14 uses the background extraction mask image to maintain the pixel values of the background in the background image, set the pixel values of the foreground to zero (pure black), and set the pixel values of the boundary area between the foreground and background to values that take into account the pixel values of the background extraction mask image. A second intermediate image is generated. The noise removal unit 14 combines the first intermediate image and the second intermediate image to generate an output image. Note that noise removal is a concept that includes not only complete noise removal but also noise reduction.
[0025] As shown in FIGS. 4( a) and 4(b), the noise removal unit 14 generates a first intermediate image by multiplying a foreground extraction mask image with a pseudo image. As shown in FIGS. 4(c) and 4(d), the noise removal unit 14 generates a second intermediate image by multiplying a background extraction mask image with a monochrome background image. In this embodiment, the noise removal process is a process of replacing pixels with monochrome pixel values, in other words, a flattening process. The background image does not have to be monochrome; it may be a phase-contrast image obtained by phase-contrast observation or a bright-field image obtained by bright-field observation of the bottom surface of a container filled only with culture solution, without any cells present. Note that moving average filtering, median filtering, or the like may also be applied as the noise removal process. The noise removal unit 14 generates the output image shown in FIG. 4(e) by summing the pixel values of the first intermediate image and the second intermediate image. As shown in FIG. 4(e), in the output image after the noise removal process, a pseudo image is displayed in the foreground, which is the region of interest, and a monochrome image with noise removed is displayed in the background, which is the region of non-interest. The boundary between the foreground and background of the output image (the overlapping region where the pixel values of the foreground extraction mask image and the background extraction mask image are both less than 255) is a linear sum of the pixel values of the pseudo image and the pixel values of the monochrome background image, depending on the pixel values of the respective mask images. For example, for a given pixel, if the pixel value of the foreground extraction mask image is a, the pixel value of the background extraction mask image is b (= 255 - a), the pixel value of the pseudo image is x, and the pixel value of the background image is y, then the pixel value z of the output image is z = (a x + b y) / 255 = (a x + (255 - a) y) / 255.
[0026] The first intermediate image and the second intermediate image do not have to be generated as "images." That is, the process of FIG. 4 may be performed for each pixel to generate an output image. Even in this case, it can be said that the processes corresponding to the generation of the first intermediate image and the second intermediate image are performed for all pixels.
[0027] 5A to 5C show the effect of the edge continuous mask image in this embodiment. FIG. 5A shows a pseudo image before noise removal processing, FIG. 5B shows an output image when noise removal processing is performed using a binary mask image created from the edge image, and FIG. 5C shows an output image when noise removal processing is performed using the edge continuous mask image. As shown in FIG. 5B, when noise removal processing is performed using a binary mask image, edges become clear, resulting in an image that looks different from an actual phase difference image. However, as shown in FIG. 5C, when noise removal processing is performed using the edge continuous mask image in this embodiment, edges can be blurred, making it possible to perform noise removal processing while maintaining an appearance close to that of an actual phase difference image.
[0028] 6 shows an example of a GUI screen 200 in this embodiment. The display control unit 15 displays a user-operable GUI (Graphical User Interface) screen 200 on a display unit such as a monitor. The GUI screen 200 may be displayed on the display unit using a dedicated application or a web browser.
[0029] The upper part of Fig. 6 displays a bright-field image, an edge image, and an output image (a noise-reduced pseudo image). The lower part of Fig. 6 displays parameters 1 to 3 that can be adjusted by the user. Note that the edge image and output image displayed first are images when parameters 1 to 3 are set to default values. For example, when the Canny filter method is used for edge detection as in this embodiment, parameter 1 is the minimum brightness threshold for determining whether or not an edge exists in the Canny filter method. Parameter 2 is the maximum brightness threshold for determining whether or not an edge exists. The lower the parameters 1 and 2, the more likely an edge is determined, and the higher the parameters 1 and 2, the more difficult it is to determine an edge.
[0030] By adjusting parameters 1 and 2, the user can, for example, set parameters that make it easier to retain detailed cellular structures in the region of interest, or set parameters that make it easier to eliminate and simplify detailed cellular structures in the region of interest. Parameters 1 and 2 may be other parameters when using other methods for edge detection.
[0031] Parameter 3 is a parameter used when generating an edge continuous mask image, and is a parameter indicating the relationship (corresponding to a predetermined value) between the distance from the edge and the pixel value. By adjusting parameter 3, for example, it is possible to sharpen or blur the boundary region in the edge continuous mask image (increase or decrease the size of the region). Parameters 1 to 3 can be adjusted appropriately depending on the type of cell observed using the first observation method (microscope) or the objective lens magnification used in the first observation method (microscope). Parameter 3 may be another parameter when using a mask image other than an edge continuous mask image (for example, a mask image created by applying a blur filter to a binary mask image).
[0032] The user can adjust parameters 1 to 3 as appropriate by operating the GUI screen 200. When the user adjusts parameters 1 to 3, the edge image and output image at the top of FIG. 6 are updated and displayed. Therefore, the user can adjust parameters 1 to 3 while checking the results of the parameter update. Note that, in addition to the updated edge image and updated output image (referred to as the updated output image), the results of the parameter update can also include, for example, a pixel value histogram of the updated output image, the density of cells detected in the updated output image, the number of cells detected in the updated output image, etc.
[0033] 7 is a flowchart showing the operation of the image processing device 100 according to this embodiment. In step S01, an input image acquired by bright-field observation is input to the image conversion unit 11 and the mask image generation unit 13. Subsequently, in step S02, the image conversion unit 11 inputs the input image to the trained model 12, performs image conversion, and outputs a pseudo image.
[0034] Next, in step S03, the mask image generation unit 13 generates a foreground extraction mask image and a background extraction mask image based on the input image. Next, in step S04, the noise removal unit 14 combines an image obtained by multiplying the foreground extraction mask image and the pseudo image with an image obtained by multiplying the background extraction mask image and the monochrome background image to generate an output image after noise removal processing. If there are other images to be processed (YES in step S05), the process returns to step S02 and repeats the processing up to step S04. If there are no other images to be processed (NO in step S05), the process ends.
[0035] According to the image processing device 100 of the above embodiment, noise removal processing is performed on the background of the pseudo image converted from the bright-field image, and an output image from which noise has been removed is generated. This solves the problem of background noise being high in contrast and conspicuous when a pseudo image with high contrast is generated, and because noise is removed only from the background, the foreground can be kept high in contrast.
[0036] According to the image processing device 100 of the above embodiment, the mask image generation unit 13 generates a mask image for foreground extraction and a mask image for background extraction based on the bright-field image, which is the input image. In this regard, if a mask image is generated based only on the pseudo image after image conversion without using the input image, it is often difficult to distinguish between cells and noise, and areas that are noise may be recognized as cells, making it impossible to generate an appropriate mask image. However, in this embodiment, a mask image is generated based on a bright-field image, which makes it easy to distinguish between cells and noise, so that an appropriate mask image can be generated.
[0037] According to the image processing device 100 of the above embodiment, the mask image generating unit 13 generates an edge continuous mask in which pixel values change stepwise by a predetermined value depending on the distance from the edge, which makes it possible to perform noise removal processing while maintaining an appearance similar to that of a phase contrast image.
[0038] [Alternative Embodiment 1] Figures 8(a) and (b) are diagrams illustrating another embodiment of the mask image generation unit 13. In this alternative embodiment, the mask image generation unit 13 generates a mask image for foreground extraction and a mask image for background extraction using cell segmentation (the outline and shape of the area where cells exist) instead of edge detection. Figure 8(a) shows a bright-field image as the input image, and Figure 8(b) shows the image after cell segmentation processing. In this alternative embodiment, the mask image generation unit 13 trains a network (U-net) for cell detection. Next, the mask image generation unit 13 detects the positions of individual cells in the bright-field image as the input image using the trained network (U-net), and estimates appropriate phase model parameters for each cell using the cell position information. The estimated parameters are used to estimate the cell region, and final segmentation is performed.
[0039] As shown in Fig. 8(b), in the image after the cell segmentation process, each cell is clearly separated. In Fig. 8(b), the area where each cell exists (foreground) is shown in white, and the area where no cell exists (background) is shown in black. The image after the cell segmentation process is subjected to the same processing as in Figs. 3(c) to 3(e) of the above embodiment, to generate a foreground extraction mask image and a background extraction mask image.
[0040] 9A and 9B are diagrams illustrating another embodiment of the mask image generator 13. In the above embodiment, the mask image generator 13 performs edge detection processing on the bright-field image, which is the input image, to generate a foreground extraction mask image and a background extraction mask image. However, the mask image generator 13 may perform edge detection processing on a separate image in addition to the bright-field image to generate a foreground extraction mask image and a background extraction mask image. The separate image may be a pseudo image or a fluorescent image after image conversion.
[0041] When generating a mask image based on a bright-field image and another image, the common portion (intersection) of a first edge image created from the bright-field image and a second edge image created from another image may be used as the edge image, as shown in FIG. 9( a). Taking the common portion of the first edge image and the second edge image can reduce false detections. Alternatively, as shown in FIG. 9( b), the union of the first edge image created from the bright-field image and the second edge image created from another image may be used as the edge image. Taking the union of the first edge image and the second edge image can reduce missed detections. Edges may be determined to be common between the first edge image and the second edge image not only when their edge positions match, but also when they are within a predetermined range. When generating a mask image based on a bright-field image and a pseudo image, it is preferable to use the common portion shown in FIG. 9( a) as the edge image. When generating a mask image based on a bright-field image and a fluorescent image, it is preferable to use the union shown in FIG. 9( b) as the edge image.
[0042] [Alternative Embodiment 3] Figure 10 shows a schematic configuration of an image processing device 110 in another embodiment. In the image processing device 100 in the above embodiment, parameters 1 to 3 were manually adjusted by a user. However, parameters 1 to 3 may be automatically set. The image processing device 110 in another embodiment further includes a parameter setting unit 16. The parameter setting unit 16 is connected to the mask image generation unit 13 and automatically sets parameters 1 to 3 when generating a mask image. The parameter setting unit 16 pre-stores a parameter set corresponding to a combination of the type of cell observed using the first observation method (microscope) and the objective lens magnification used in the first observation method (microscope), or a parameter value corresponding to the contrast of the bright-field image that is the input image, and automatically sets parameters 1 to 3 to appropriate values depending on the values actually used.
[0043] [Alternative Embodiment 4] In the above embodiment, the mask image generation unit 13 generates an edge continuous mask image in which pixel values change gradually by a predetermined value depending on the distance from the edge, and generates a foreground extraction mask image and a background extraction mask image based on this. However, instead of this, a single type of binary mask image may be generated from the input image (or edge image) in which pixel values in the region corresponding to the foreground are set to 255 and pixel values in the region corresponding to the background are set to 0, and a foreground extraction mask image and a background extraction mask image may be generated based on this. In this case, the binary mask image may be further subjected to image processing such as a Gaussian filter to blur the image, thereby forming a boundary region between the region corresponding to the foreground and the region corresponding to the background in the binary mask image. Pixel values between 255 and 0 are assigned within the boundary region.
[0044] [Alternative Embodiment 5] In the above embodiment, the image obtained by the first observation method was a reflected bright-field image, and the image obtained by the second observation method was a pseudo phase-contrast image. However, other types of image combinations are possible as long as the image obtained by the first observation method is a low-contrast image and the image obtained by the second observation method is a high-contrast image. For example, the image obtained by the second observation method may be a high-contrast image such as a relief contrast image (Hoffman modulation image) or a differential interference image, other than a phase-contrast image.
[0045] Various embodiments of the present invention may also be described with reference to flowcharts and block diagrams, where the blocks may represent (1) stages of a process in which operations are performed or (2) sections of an apparatus responsible for performing the operations. Particular stages and sections may be implemented by dedicated circuitry, programmable circuitry provided with computer-readable instructions stored on a computer-readable medium, and / or a processor provided with computer-readable instructions stored on a computer-readable medium. Dedicated circuitry may include digital and / or analog hardware circuitry, and may include integrated circuits (ICs) and / or discrete circuits. Programmable circuitry may include reconfigurable hardware circuitry including logical AND, OR, XOR, NAND, NOR, and other logic operations, flip-flops, registers, memory elements such as field programmable gate arrays (FPGAs), programmable logic arrays (PLAs), and the like.
[0046] A computer-readable medium may include any tangible device capable of storing instructions that are executed by an appropriate device, such that the computer-readable medium having instructions stored thereon comprises an article of manufacture containing instructions that can be executed to create means for performing the operations specified in the flowcharts or block diagrams. Examples of computer-readable media may include electronic, magnetic, optical, electromagnetic, and semiconductor storage media. More specific examples of computer-readable media may include floppy disks, diskettes, hard disks, random access memories (RAMs), read-only memories (ROMs), erasable programmable read-only memories (EPROMs or flash memories), electrically erasable programmable read-only memories (EEPROMs), static random access memories (SRAMs), compact disc read-only memories (CD-ROMs), digital versatile discs (DVDs), Blu-ray (RTM) discs, memory sticks, integrated circuit cards, and the like.
[0047] The computer readable instructions may include either assembler instructions, Instruction Set Architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state-setting data, or source or object code written in any combination of one or more programming languages, including object-oriented programming languages such as Smalltalk®, JAVA®, C++, etc., and conventional procedural programming languages such as the “C” programming language or similar programming languages.
[0048] The computer-readable instructions may be provided to a processor or programmable circuitry of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, either locally or over a local area network (LAN), a wide area network (WAN) such as the Internet, etc., which executes the computer-readable instructions to create means for performing the operations specified in the flowcharts or block diagrams. Examples of processors include computer processors, processing units, microprocessors, digital signal processors, controllers, microcontrollers, etc.
[0049] 11 illustrates an example of a computer 2200 in which aspects of the present invention may be embodied, in whole or in part. Programs installed on the computer 2200 may cause the computer 2200 to function as or perform operations associated with an apparatus or one or more sections of the apparatus according to embodiments of the present invention, and / or to perform a process or steps of a process according to embodiments of the present invention. Such programs may be executed by the CPU 2212 to cause the computer 2200 to perform specific operations associated with some or all of the blocks in the flowcharts and block diagrams described herein.
[0050] A computer 2200 according to this embodiment includes a CPU 2212, a RAM 2214, a graphics controller 2216, and a display device 2218, which are interconnected by a host controller 2210. The computer 2200 also includes input / output units such as a communication interface 2222, a hard disk drive 2224, a DVD-ROM drive 2226, and an IC card drive, which are connected to the host controller 2210 via an input / output controller 2220. The computer also includes legacy input / output units such as a ROM 2230 and a keyboard 2242, which are connected to the input / output controller 2220 via an input / output chip 2240.
[0051] The CPU 2212 operates according to programs stored in the ROM 2230 and RAM 2214, thereby controlling each unit. The graphics controller 2216 acquires image data generated by the CPU 2212 into a frame buffer or the like provided in the RAM 2214 or into the graphics controller 2216 itself, and causes the image data to be displayed on the display device 2218.
[0052] The communication interface 2222 communicates with other electronic devices via a network. The hard disk drive 2224 stores programs and data used by the CPU 2212 in the computer 2200. The DVD-ROM drive 2226 reads programs or data from the DVD-ROM 2201 and provides the programs or data to the hard disk drive 2224 via the RAM 2214. The IC card drive reads programs and data from an IC card and / or writes programs and data to an IC card.
[0053] ROM 2230 stores therein a boot program or the like that is executed by computer 2200 upon activation, and / or programs that depend on the hardware of computer 2200. I / O chip 2240 may also connect various I / O units to I / O controller 2220 via parallel ports, serial ports, keyboard ports, mouse ports, etc.
[0054] The programs are provided by a computer-readable medium such as a DVD-ROM 2201 or an IC card. The programs are read from the computer-readable medium, installed in the hard disk drive 2224, RAM 2214, or ROM 2230, which are also examples of computer-readable media, and executed by the CPU 2212. Information processing described in these programs is read by the computer 2200, and brings about cooperation between the programs and the various types of hardware resources described above. An apparatus or method may be configured by implementing information manipulation or processing in accordance with the use of the computer 2200.
[0055] For example, when communication is performed between computer 2200 and an external device, CPU 2212 may execute a communication program loaded into RAM 2214 and instruct communication interface 2222 to perform communication processing based on the processing described in the communication program. Under the control of CPU 2212, communication interface 2222 reads transmission data stored in a transmission buffer processing area provided in RAM 2214, hard disk drive 2224, DVD-ROM 2201, or a recording medium such as an IC card, and transmits the read transmission data to the network, or writes received data received from the network to a reception buffer processing area or the like provided on the recording medium.
[0056] Furthermore, the CPU 2212 may cause all or a necessary portion of a file or database stored on an external recording medium such as the hard disk drive 2224, the DVD-ROM drive 2226 (DVD-ROM 2201), an IC card, etc. to be read into the RAM 2214, and may perform various types of processing on the data on the RAM 2214. The CPU 2212 then writes back the processed data to the external recording medium.
[0057] Various types of information, such as various types of programs, data, tables, and databases, may be stored on the recording medium and may undergo information processing. The CPU 2212 may perform various types of processing on data read from the RAM 2214, including various types of operations, information processing, conditional judgment, conditional branching, unconditional branching, information search / replacement, etc., as described throughout this disclosure and specified by the instruction sequences of the programs, and write the results back to the RAM 2214. The CPU 2212 may also search for information in a file, database, etc. on the recording medium. For example, if multiple entries each having an attribute value of a first attribute associated with an attribute value of a second attribute are stored on the recording medium, the CPU 2212 may search for an entry that matches a condition specified by the attribute value of the first attribute from among the multiple entries, read the attribute value of the second attribute stored in the entry, and thereby obtain the attribute value of the second attribute associated with the first attribute that satisfies a predetermined condition.
[0058] The above-described programs or software modules may be stored in a computer-readable medium on or near the computer 2200. A recording medium such as a hard disk or RAM provided in a server system connected to a dedicated communication network or the Internet can also be used as a computer-readable medium, thereby providing the programs to the computer 2200 via the network.
[0059] Although the present invention has been described above using embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments. It will be apparent to those skilled in the art that various modifications and improvements can be made to the above embodiments. It is clear from the claims that such modifications and improvements can also be included within the technical scope of the present invention.
[0060] It should be noted that the order of execution of each process, such as operations, procedures, steps, and stages, in the devices, systems, programs, and methods shown in the claims, specifications, and drawings is not specifically stated as "before," "prior to," etc., and that the processes can be performed in any order unless the output of a previous process is used in a subsequent process. Even if the operational flow in the claims, specifications, and drawings is described using "first," "next," etc. for convenience, this does not mean that the processes must be performed in this order.
[0061] 11 Image conversion unit, 12 Trained model, 13 Mask image generation unit, 14 Noise removal unit, 15 Display control unit, 16 Parameter setting unit, 100 Image processing device, 110 Image processing device, 200 GUI screen, 2200 Computer, 2201 DVD-ROM, 2210 Host controller, 2212 CPU, 2214 RAM, 2216 Graphics controller, 2218 Display device, 2220 Input / output controller, 2222 Communication interface, 2224 Hard disk drive, 2226 DVD-ROM drive, 2230 ROM, 2240 Input / output chip, 2242 Keyboard
Claims
1. An image processing device having: an image conversion unit that converts an input image, which is a microscopic image including a biological sample obtained using a first observation method, into a pseudo image that has an appearance obtained using a second observation method different from the first observation method; a mask image generation unit that generates at least one continuous mask image based on the input image, the continuous mask image consisting of a first region corresponding to a foreground including the biological sample, a second region corresponding to a background not including the biological sample, and a boundary region between the first region and the second region, wherein pixel values within the boundary region change gradually from the first region to the second region; and a noise removal unit that generates an output image in which background noise has been removed while maintaining the appearance of the biological sample, based on the continuous mask images and the pseudo image.
2. The image processing device according to claim 1, wherein the mask image generation unit creates a mask image for foreground extraction and a mask image for background extraction as the continuous mask image, and the sum of the pixel value of each pixel of the mask image for foreground extraction and the pixel value of each pixel of the mask image for background extraction is constant.
3. The image processing device according to claim 2, wherein the noise removal unit generates the output image by combining a first intermediate image created based on the foreground extraction mask image and the pseudo image, and a second intermediate image created based on the background extraction mask image and the background image.
4. An image processing device according to any one of claims 1 to 3, wherein the first observation method is transmitted light bright-field observation or reflected light bright-field observation, and the second observation method is phase contrast observation.
5. An image processing device according to any one of claims 1 to 4, wherein the image conversion unit has a trained model that converts a transmitted bright-field image or a reflected bright-field image into a pseudo phase contrast image.
6. An image processing device according to any one of claims 1 to 5, wherein the mask image generating section generates the continuous mask image based only on the input image.
7. An image processing device according to any one of claims 1 to 5, wherein the mask image generating unit generates the continuous mask image based on the pseudo image in addition to the input image.
8. The image processing device according to claim 7, wherein the mask image generating section generates the continuous mask image based on a union of the input image and the pseudo image, or an intersection of the input image and the pseudo image.
9. An image processing device according to any one of claims 1 to 8, wherein the mask image generating section sets the pixel values within the boundary region by gradually varying them according to the distance from the edge of the input image.
10. An image processing device according to any one of claims 1 to 8, wherein the mask image generation unit sets the pixel values within the boundary region by gradually varying them depending on the distance from the contour of the area in which the biological sample is present in the input image.
11. The image processing device according to any one of claims 1 to 10, further comprising a display control unit that causes a display unit to display a screen for accepting parameter input for generating the continuous mask image.
12. An image processing device according to any one of claims 1 to 11, further comprising a parameter setting unit that automatically sets parameters for generating the continuous mask image based on at least one of the type of the biological sample, the objective lens magnification used in the first observation method, or the contrast of the input image.
13. An image processing method comprising: an image conversion step of converting an input image, which is a microscopic image including a biological sample obtained by a first observation method, into a pseudo image having an appearance obtained by a second observation method different from the first observation method; a mask image generation step of generating, based on the input image, at least one continuous mask image consisting of a first region corresponding to a foreground including the biological sample, a second region corresponding to a background not including the biological sample, and a boundary region between the first region and the second region, wherein pixel values within the boundary region change stepwise from the first region to the second region; and a noise removal step of generating, based on the continuous mask images and the pseudo image, an output image in which noise in the background has been removed while maintaining the appearance of the biological sample.
14. An image processing program that causes a computer to execute the following steps: an image conversion step for converting an input image, which is a microscopic image including a biological sample obtained by a first observation method, into a pseudo image having an appearance obtained by a second observation method different from the first observation method; a mask image generation step for generating, based on the input image, at least one continuous mask image consisting of a first region corresponding to a foreground including the biological sample, a second region corresponding to a background not including the biological sample, and a boundary region between the first region and the second region, wherein pixel values within the boundary region change gradually from the first region to the second region; and a noise removal step for generating, based on the continuous mask images and the pseudo image, an output image in which noise in the background has been removed while maintaining the appearance of the biological sample.
Citation Information
Patent Citations
Cashmere and wool optical microscope image recognition method based on generative adversarial network
CN112927180A
Image processing method and image processing apparatus
JP2020060822A
Modification of post-viewing parameters for digital images using image region or feature information
US20090003708A1
