Image processing device, image processing method, and image processing program

Abstract

This image processing device comprises: an image conversion unit that converts an input image, which is a microscope image including a biological sample and which has been acquired via a first observation method, into a pseudo image which has an appearance as if acquired via a second observation method differing from the first observation method; a mask image generation unit that, on the basis of the input image, generates at least one continuous mask image which is constituted by a first region corresponding to a foreground that includes the biological sample, a second region corresponding to a background that does not include the biological sample, and a boundary region between the first region and the second region, and in which pixel values in the boundary region change in stages from the first region to the second region; and a noise reduction unit that, on the basis of the continuous mask image and the pseudo image, generates an output image in which noise is removed in the background, while maintaining the appearance of the biological sample.

Description

Technical Field

[0001] The present invention relates to an image processing apparatus, an image processing method, and an image processing program.

Background Art

[0002] Patent Document 1 describes a method for generating a learned model for performing image conversion, an image processing method, an image conversion apparatus, and a program. [Prior Art Document] [Patent Document] [Patent Document 1] International Publication No. 2022 / 113366, General Disclosure

[0003] In a first aspect of the present invention, an image processing apparatus is provided. The image processing apparatus includes an image conversion unit that converts an input image, which is a microscopic image including a biological sample obtained by a first observation method, into a pseudo-image having an appearance obtained by a second observation method different from the first observation method; a mask image generation unit that generates at least one continuous mask image based on the input image, the continuous mask image including a first region corresponding to a foreground including the biological sample, a second region corresponding to a background not including the biological sample, and a boundary region between the first region and the second region, and pixel values in the boundary region changing stepwise from the first region toward the second region; and a noise removal unit that generates an output image in which noise in the background is removed while maintaining the appearance of the biological sample based on the continuous mask image and the pseudo-image.

[0004] A second aspect of the present invention provides an image processing method. The image processing method includes: an image conversion step of converting an input image, which is a microscopic image including a biological sample obtained by a first observation method, into a pseudo-image that looks as if it were obtained by a second observation method different from the first observation method; a mask image generation step of generating at least one continuous mask image based on the input image, which consists of a first region corresponding to the foreground including the biological sample, a second region corresponding to the background not including the biological sample, and a boundary region between the first and second regions, wherein the pixel values ​​in the boundary region change stepwise from the first region to the second region; and a noise reduction step of generating an output image based on the continuous mask image and the pseudo-image, in which background noise is removed while maintaining the appearance of the biological sample.

[0005] A third aspect of the present invention provides an image processing program. The image processing program causes a computer to perform the following steps: an image conversion step that converts an input image, which is a microscope image including a biological sample acquired by a first observation method, into a pseudo-image that looks as if it were acquired by a second observation method different from the first observation method; a mask image generation step that generates at least one continuous mask image based on the input image, consisting of a first region corresponding to the foreground including the biological sample, a second region corresponding to the background not including the biological sample, and a boundary region between the first and second regions, in which the pixel values ​​in the boundary region change stepwise from the first region to the second region; and a noise reduction step that generates an output image based on the continuous mask image and the pseudo-image, in which background noise is removed while maintaining the appearance of the biological sample.

[0006] It should be noted that the above summary of the invention does not enumerate all the necessary features of the present invention. Furthermore, subcombinations of these features may also constitute an invention. [Brief explanation of the drawing]

[0007] [Figure 1] The schematic configuration of the image processing device 100 in this embodiment is shown. [Figure 2](a) shows the bright-field image which is the input image, and (b) shows the pseudo-image after image conversion by the image conversion unit 11. [Figure 3] (a) to (d) show an example of the mask generation process by the mask image generation unit 13. [Figure 4] (a) to (e) show an example of noise reduction processing by the noise reduction unit 14. [Figure 5] (a) to (c) illustrate the effect of the edge continuity mask in this embodiment. [Figure 6] An example of the GUI screen 200 in this embodiment is shown. [Figure 7] This is a flowchart showing the operation of the image processing device 100 in this embodiment. [Figure 8] (a) and (b) are diagrams illustrating other embodiments of the mask image generation unit 13. [Figure 9] (a) and (b) are diagrams illustrating other embodiments of the mask image generation unit 13. [Figure 10] The schematic configuration of the image processing device 110 in another embodiment is shown. [Figure 11] An example of computer 2200 is shown. [Modes for carrying out the invention]

[0008] The present invention will be described below through embodiments of the invention. The following embodiments are not intended to limit the invention as claimed. Not all combinations of features described in the embodiments are necessarily essential to the solution of the invention.

[0009] Figure 1 shows a schematic configuration of the image processing apparatus 100 in this embodiment. As shown in Figure 1, the image processing apparatus 100 in this embodiment includes an image conversion unit 11, a mask image generation unit 13, a noise reduction unit 14, and a display control unit 15. The image conversion unit 11 has a trained model 12. The image processing apparatus 100 is a device that performs image processing on a microscope image obtained from microscopic observation.

[0010] The image conversion unit 11 receives an input image, which is a microscope image acquired by the first observation method during microscopic observation. The input image may be the microscope image itself, or it may be a microscope image that has undergone image processing different from the image conversion described later (for example, pre-processing such as shading correction or smoothing of a part of the image). In this embodiment, the first observation method is reflected bright-field observation, and the input image is, for example, a reflected bright-field image (hereinafter referred to as a bright-field image) acquired with a bright-field microscope. The first observation method may be transmitted bright-field observation, fluorescence observation, or dark-field observation, in which case the input image will be a transmitted bright-field image, fluorescence image, or dark-field image acquired by each observation method.

[0011] The biological samples used for microscopic observation are not particularly limited and may be derived from multicellular organisms such as animals and plants, or from single-celled organisms such as bacteria. Examples include unstained cultured cells, tissues, and organs. Specifically, living cells cultured in a single or multilayer structure in a culture vessel, or cultured individually or in cell aggregates in suspension, can be used as examples. The area of ​​interest within the object that is observed with particular attention may be an entire cell, or it may be an intracellular organelle such as the nucleolus or the cell membrane. The culture vessel may be a general cell culture container such as a dish or well plate, or an organ-on-a-chip.

[0012] The image conversion unit 11 converts the input image, which is a microscope image acquired by the first observation method, into a pseudo-image that appears as if it were acquired by a second observation method different from the first observation method. In this embodiment, the second observation method is phase contrast observation, and the pseudo-image is a pseudo-phase contrast image.

[0013] The image conversion unit 11 has a trained model 12. The trained model 12 is a model that has learned the mapping from a source image to a target image. In this embodiment, the trained model 12 is trained by an image conversion network using bright-field images captured by bright-field observation as the source images and phase-difference images as the target images. The image conversion network is not particularly limited, and examples include "convolutional neural networks (CNN)", "generative adversarial networks (GAN)", "CGAN (Conditional GAN)", "DCGAN (Deep Conventional GAN)", "Pix2Pix", and "CycleGAN". Details of the method for generating the trained model 12 are omitted in this specification. For example, it is described in Patent Document 1, but is not limited to that.

[0014] The image conversion unit 11 inputs the brightfield image, which is the input image, to the trained model 12. The trained model 12 then performs an image conversion on the brightfield image and outputs a pseudo-image. By inputting a brightfield image to the trained model 12, the arrangement of objects on the brightfield image remains unchanged, but an image is obtained in which the appearance, or style, has been converted into a phase-difference image. This type of image conversion is also called style conversion. As shown in Figure 1, the image conversion unit 11 outputs the pseudo-phase-difference image (called a pseudo-image) generated by the image conversion to the noise reduction unit 14.

[0015] Figure 2(a) shows the input bright-field image, and Figure 2(b) shows the pseudo-image after image conversion by the image conversion unit 11. Phase-contrast images often have higher contrast than bright-field images, making it easier to visually observe cell structures, etc. Therefore, by converting the bright-field image in Figure 2(a) into a pseudo-image that looks like the phase-contrast image in Figure 2(b), the image contrast is increased as shown in the figure, making it easier to visually observe cell structures, etc. On the other hand, the contrast of the background area other than the cell structures also increases, resulting in the drawback that noise becomes noticeable. This embodiment aims to resolve this drawback.

[0016] Figures 3(a) to 3(d) show an example of the mask image generation process by the mask image generation unit 13. Figure 3(a) shows the bright-field image which is the input image, Figure 3(b) shows the edge image, Figure 3(c) shows the edge image after distance transformation, Figure 3(d) shows the mask image for foreground extraction, and Figure 3(e) shows the mask image for background extraction.

[0017] The mask image generation unit 13 generates at least one continuous mask image based on the input image, consisting of a first region corresponding to the foreground containing a biological sample, a second region corresponding to the background not containing a biological sample, and a boundary region between the first and second regions, where the pixel values ​​in the boundary region change stepwise from the first region to the second region. The mask image generation unit 13 generates a foreground extraction mask image based on the bright-field input image, which extracts the foreground while maintaining its appearance from the pseudo-image. The mask image generation unit 13 further generates a background extraction mask image based on the bright-field input image, which extracts the region corresponding to the background from the background image. The details of the processing by the mask image generation unit 13 will be described below with reference to Figures 3(a) to (d). In this specification, the foreground is the region of interest containing cells, etc., to be observed in the input image or pseudo-image, and the background is the region of interest not containing cells, etc., to be observed in the input image or pseudo-image.

[0018] First, the mask image generation unit 13 performs an edge detection process for detecting a location with a large change in pixel value on the bright-field image (Fig. 3(a)), which is the input image. A generally known method may be used for the edge detection process. In this embodiment, for example, the Canny filter method is used. The Canny filter method obtains the results of first-order differential filtering in the vertical and horizontal directions within the edge detection area, and determines the gradient and direction of the edge from the two differential images. Instead of the Canny filter method, the Sobel filter method or the Prewitt filter method may also be used. As shown in Fig. 3(b), the mask image generation unit 13 generates an edge image in which the detected edges are displayed as a result of the edge detection process.

[0019] As shown in Fig. 3(c), the mask image generation unit 13 performs a distance conversion process on the edge image to generate an image after distance conversion (referred to as a distance conversion image). In this embodiment, the distance conversion process is a process of assigning a difference from the pixel value of the edge one by one according to the distance from the edge. For example, with the pixel value of the edge as the minimum value, a larger pixel value is assigned as the distance from the edge becomes farther. Thereby, the periphery of the edge can be blurred. Here, the parameter indicating the relationship between the distance from the edge and the pixel value can be appropriately set according to the magnification of the objective lens and the type of the imaging object. For example, those with a clear periphery or, conversely, a blurred periphery of the edge can be set.

[0020] As shown in FIGS. 3(d) and 3(e), the mask image generation unit 13 sets the pixel values of the region corresponding to the foreground (referred to as the foreground corresponding region) to 255 based on the distance conversion image, sets the pixel values of the region corresponding to the background (referred to as the background corresponding region) to 0, and sets the pixel values of the boundary region between the foreground corresponding region and the background corresponding region to decrease step by step by a predetermined value from the foreground corresponding region toward the background corresponding region (according to the distance from the edge). A foreground extraction mask image (FIG. 3(d)) is generated, and a background extraction mask image (FIG. 3(e)) is generated, in which the pixel values of the background corresponding region are set to 255, the pixel values of the foreground corresponding region are set to 0, and the pixel values of the boundary region between the background corresponding region and the foreground corresponding region are set to increase step by step by a predetermined value from the foreground corresponding region toward the background corresponding region (according to the distance from the edge). In FIGS. 3(d) and 3(e), the pixel value 255 is represented by white, and the pixel value 0 is represented by black.

[0021] At this time, the sum of the pixel values of each pixel of the foreground extraction mask image and the pixel values of each pixel of the background extraction mask image is set to 255 (a constant value) (256 gradations in an 8-bit image). In this case, it can be said that the foreground extraction mask image and the background extraction mask image are in a relationship of white-black inversion with each other. Corresponding to the fact that the edge is blurred in the distance conversion image as described above, in both the foreground extraction mask image and the background extraction mask image, the pixel values change step by step by a predetermined value according to the distance from the edge within the boundary region between the region with the pixel value of 255 and the region with the pixel value of 0. For example, in the foreground extraction mask, the farther the distance from the detected edge, the smaller the pixel value, and the closer the distance from the detected edge, the larger the pixel value. When the predetermined value is 20, the width of the boundary region is 12 pixels, and when the predetermined value is 50, the width of the boundary region is 5 pixels. Depending on the predetermined value, the size of the region changes. A mask image in which the pixel values change step by step by a predetermined value according to the distance from the edge is called an edge continuous mask image.

[0022] A continuous mask image is a mask image in which the pixel values ​​within the boundary region between a region with a pixel value set to 255 and a region with a pixel value set to 0 change in steps by a predetermined value from one region to the other (depending on the distance from a predetermined position). The concept of a continuous mask image includes edge continuous mask images, mask images described in other embodiment 1 described later, and mask images described in other embodiment 4. The predetermined position is an edge in the case of an edge continuous mask image, the contour of a cell-segmented region (a region where cells exist) in the case of a mask image described in other embodiment 1, and the boundary where the pixel values ​​of the binary mask image are different in the case of a mask image described in other embodiment 4.

[0023] Furthermore, the predetermined value may not be a fixed value but a variable value (the range of change is not constant but fluctuates), and an arbitrary value or a changed pixel value may be set according to the distance from the predetermined position. As shown in Figure 1, the mask image generation unit 13 outputs the generated foreground extraction mask image and background extraction mask image to the noise reduction unit 14. Based on the continuous mask image and the pseudo-image, the noise reduction unit 14 generates an output image in which background noise is removed while maintaining the appearance of the biological sample.

[0024] Figures 4(a) to 4(e) show an example of noise reduction processing by the noise reduction unit 14. Figure 4(a) shows a pseudo-image, Figure 4(b) shows a mask image for foreground extraction, Figure 4(c) shows a background image, Figure 4(d) shows a mask image for background extraction, and Figure 4(e) shows the output image after noise reduction. As shown in Figure 1, the noise reduction unit 14 receives a pseudo-image from the image conversion unit 11 and a mask image for foreground extraction and a mask image for background extraction from the mask image generation unit 13. Using the mask image for foreground extraction, the noise reduction unit 14 generates a first intermediate image in which the pixel values ​​of the foreground are kept as they are, the pixel values ​​of the background are set to zero (completely black), and the pixel values ​​of the boundary region between the foreground and background are set to values ​​that take into account the pixel values ​​of the mask image for foreground extraction. The noise reduction unit 14 uses a background extraction mask image to maintain the pixel values ​​of the background in the background image, set the pixel values ​​of the foreground to zero (completely black), and set the pixel values ​​of the boundary region between the foreground and background to values ​​that take into account the pixel values ​​of the background extraction mask image. A second intermediate image is generated. The noise reduction unit 14 combines the first intermediate image and the second intermediate image to generate an output image. Note that noise reduction is a concept that includes not only the complete removal of noise but also the reduction of noise.

[0025] As shown in Figures 4(a) and (b), the noise reduction unit 14 generates a first intermediate image by multiplying the foreground extraction mask image and the pseudo-image. As shown in Figures 4(c) and (d), the noise reduction unit 14 generates a second intermediate image by multiplying the background extraction mask image and the monochrome background image. In this embodiment, the noise reduction process is a process of replacing pixel values ​​with monochrome pixels, in other words, a flattening process. The background image does not have to be monochrome; it may be a phase contrast image obtained by separately observing the bottom surface of a container filled only with culture solution and no cells present, or a bright-field image obtained by separately observing bright-field pixels. Moving average filtering, median filtering, etc., may also be applied as noise reduction processing. The noise reduction unit 14 generates the output image shown in Figure 4(e) by summing the pixel values ​​of the first intermediate image and the second intermediate image. As shown in Figure 4(e), in the output image after noise reduction processing, the pseudo-image is displayed in the foreground, which is the area of ​​interest, and the monochrome image with noise removed is displayed in the background, which is the area of ​​non-interest. The boundary between the foreground and background of the output image (the overlapping region where the pixel value of the foreground extraction mask image is less than 255 and the pixel value of the background extraction mask image is less than 255) is a linear sum of the pixel value of the pseudo-image and the pixel value of the monochrome background image, depending on the pixel value of each mask image. For example, if for a certain pixel the pixel value of the foreground extraction mask image is a, the pixel value of the background extraction mask image is b (=255-a), the pixel value of the pseudo-image is x, and the pixel value of the background image is y, then the pixel value z of the output image is z = (a·x+b·y) / 255 = (a·x+(255-a)·y) / 255.

[0026] Note that the first and second intermediate images do not necessarily have to be generated as "images." In other words, the process shown in Figure 4 may be performed for each pixel to generate the output image. Even in this case, it can be said that the process corresponding to the generation of the first and second intermediate images has been performed for all pixels.

[0027] Figures 5(a) to 5(c) illustrate the effect of the edge continuity mask image in this embodiment. Figure 5(a) shows a pseudo-image before noise reduction processing, Figure 5(b) shows the output image when noise reduction processing is performed using a binary mask image created from edge images, and Figure 5(c) shows the output image when noise reduction processing is performed using an edge continuity mask image. As shown in Figure 5(b), when noise reduction processing is performed using a binary mask image, the edges become sharp, resulting in an image that differs in appearance from the actual phase difference image. However, as shown in Figure 5(c), when noise reduction processing is performed using the edge continuity mask image in this embodiment, the edges can be blurred, making it possible to perform noise reduction processing while maintaining an appearance close to that of the actual phase difference image.

[0028] Figure 6 shows an example of the GUI screen 200 in this embodiment. The display control unit 15 causes the user-operable GUI (Graphical User Interface) screen 200 to be displayed on a display unit such as a monitor. The GUI screen 200 may be displayed on the display unit using a dedicated application, or it may be displayed on the display unit using a web browser.

[0029] The top of Figure 6 displays the brightfield image, edge image, and output image (noise-reduced pseudo-image). The bottom of Figure 6 shows user-adjustable parameters 1 to 3. Note that the edge image and output image initially displayed are those when parameters 1 to 3 are set to their default values. For example, when using the Canny filter method for edge detection as in this embodiment, parameter 1 is the minimum brightness threshold for determining whether an image is an edge in the Canny filter method. Parameter 2 is the maximum brightness threshold for determining whether an image is an edge. Lowering parameters 1 and 2 makes it easier to determine an image as an edge, while raising parameters 1 and 2 makes it harder to determine an image as an edge.

[0030] By adjusting parameters 1 and 2, the user can, for example, create settings that make it easier to retain detailed cellular structures in the region of interest, or settings that make it easier to simplify the region by eliminating detailed cellular structures. Parameters 1 and 2 may be other parameters if other methods are used for edge detection.

[0031] Parameter 3 is used when generating an edge continuity mask image and indicates the relationship between the distance from the edge and the pixel value (corresponding to a predetermined value). By adjusting parameter 3, for example, the boundary region in the edge continuity mask image can be made sharper or blurred (the size of the region can be made larger or smaller). Parameters 1 to 3 can be adjusted as appropriate depending on the type of cell observed by the first observation method (microscope) or the objective lens magnification used by the first observation method (microscope). Parameter 3 may be a different parameter if other mask images other than the edge continuity mask image are used (for example, a mask image created by applying a blur filter to a binary mask image).

[0032] The user can adjust parameters 1 to 3 as appropriate by operating the GUI screen 200. When the user adjusts parameters 1 to 3, the edge image at the top of Figure 6 and the output image are updated and displayed. Therefore, the user can adjust parameters 1 to 3 while checking the results of the parameter update. In addition to the updated edge image and updated output image (referred to as the updated application image), other results of the parameter update include, for example, the pixel value histogram of the updated output image, the density of cells detected in the updated output image, and the number of cells detected in the updated output image.

[0033] Figure 7 is a flowchart showing the operation of the image processing device 100 in this embodiment. In step S01, the input image acquired by bright-field observation is input to the image conversion unit 11 and the mask image generation unit 13. Subsequently, in step S02, the image conversion unit 11 inputs the input image to the trained model 12 to perform image conversion and outputs a pseudo-image.

[0034] Next, in step S03, the mask image generation unit 13 generates a foreground extraction mask image and a background extraction mask image based on the input image. Next, in step S04, the noise reduction unit 14 generates an output image after noise reduction by combining an image obtained by multiplying the foreground extraction mask image and a pseudo-image with an image obtained by multiplying the background extraction mask image and a monochrome background image. If there are other images to be processed (YES in step S05), the process returns to step S02 and repeats up to step S04. If there are no other images to be processed (NO in step S05), the process ends.

[0035] According to the image processing device 100 in the above embodiment, noise reduction processing is performed on the background of the pseudo-image converted from the brightfield image, and an output image with the noise removed is generated. This resolves the problem that when a high-contrast pseudo-image is created, the contrast of the background noise also becomes high and noticeable, and since only the background is noise-reduced, the foreground can be kept at a high contrast.

[0036] According to the image processing apparatus 100 in the above embodiment, the mask image generation unit 13 generates a foreground extraction mask image and a background extraction mask image based on the bright-field image which is the input image. In this regard, if a mask image is generated based only on a pseudo-image after image conversion without using the input image, it is often difficult to distinguish between cells and noise, and areas that are noise may be recognized as cells, resulting in the inability to generate an appropriate mask image. However, in this embodiment, since the mask image is generated based on a bright-field image in which it is easy to distinguish between cells and noise, an appropriate mask image can be generated.

[0037] According to the image processing apparatus 100 in the above embodiment, the mask image generation unit 13 generates an edge continuous mask in which the pixel values ​​change in steps by predetermined values ​​according to the distance from the edge. This makes it possible to perform noise reduction processing while maintaining an appearance similar to a phase difference image.

[0038] [Other Embodiment 1] Figures 8(a) and 8(b) illustrate other embodiments of the mask image generation unit 13. In other embodiments, the mask image generation unit 13 generates foreground extraction mask images and background extraction mask images using cell segmentation (contours and shapes of regions where cells exist) instead of edge detection. Figure 8(a) is the input bright-field image, and Figure 8(b) is the image after cell segmentation processing. In other embodiments, the mask image generation unit 13 learns a network (U-net) for cell detection. Subsequently, the mask image generation unit 13 detects the position of individual cells in the input bright-field image using the learned network (U-net), and estimates appropriate phase model parameters for each cell using the cell position information. Cell regions are estimated using the estimated parameters, and final segmentation is performed.

[0039] As shown in Figure 8(b), in the image after cell segmentation processing, each cell is clearly separated. In Figure 8(b), the area where each cell exists (foreground) is shown in white, and the area where no cells exist (background) is shown in black. The image after cell segmentation processing is subjected to the same processing as in Figures 3(c) to (e) of the above embodiment to generate a foreground extraction mask image and a background extraction mask image.

[0040] [Another Embodiment 2] Figures 9(a) and 9(b) illustrate other embodiments of the mask image generation unit 13. In the above embodiment, the mask image generation unit 13 generates a foreground extraction mask image and a background extraction mask image by performing edge detection processing on the input brightfield image. However, the mask image generation unit 13 may also generate a foreground extraction mask image and a background extraction mask image by performing edge detection processing on another image in addition to the brightfield image. The other image may be a pseudo-image or a fluorescence image after image conversion.

[0041] When generating a mask image based on a brightfield image and another image, as shown in Figure 9(a), the intersection of a first edge image created from the brightfield image and a second edge image created from the other image may be used as the edge image. Using the intersection of the first and second edge images can suppress false detections. Alternatively, as shown in Figure 9(b), the union of the first edge image created from the brightfield image and the second edge image created from the other image may be used as the edge image. Using the union of the first and second edge images can suppress missed detections. Note that, in addition to the edge positions matching between the first and second edge images, edges within a predetermined range may also be considered common edges. When generating a mask image based on a brightfield image and a pseudo-image, it is preferable to use the intersection shown in Figure 9(a) as the edge image. Furthermore, when generating a mask image based on a brightfield image and a fluorescence image, it is preferable to use the union shown in Figure 9(b) as the edge image.

[0042] [Other Embodiment 3] Figure 10 shows a schematic configuration of the image processing device 110 in another embodiment. In the image processing device 100 in the above embodiment, the user manually adjusted parameters 1 to 3. However, parameters 1 to 3 may be set automatically. The image processing device 110 in the other embodiment further includes a parameter setting unit 16. The parameter setting unit 16 is connected to the mask image generation unit 13 and automatically sets parameters 1 to 3 when generating a mask image. The parameter setting unit 16 pre-stores parameter sets corresponding to combinations of the type of cell to be observed in the first observation method (microscope) and the objective lens magnification used in the first observation method (microscope), or parameter values ​​corresponding to the contrast of the bright-field image which is the input image, and automatically sets parameters 1 to 3 to appropriate values ​​according to the values ​​actually used.

[0043] [Other Embodiments 4] In the above embodiment, the mask image generation unit 13 generates an edge continuous mask image in which the pixel values ​​change in steps by predetermined values ​​according to the distance from the edge, and generates a foreground extraction mask image and a background extraction mask image based on this. However, instead, a single type of binary mask image may be generated from the input image (or edge image) in which the pixel values ​​of the area corresponding to the foreground are set to 255 and the pixel values ​​of the area corresponding to the background are set to 0, and a foreground extraction mask image and a background extraction mask image may be generated based on this. In this case, further image processing such as applying a Gaussian filter to the binary mask image to blur it may be applied to form a boundary region between the area corresponding to the foreground and the area corresponding to the background of the binary mask image. Within the boundary region, any pixel value from 255 to 0 is assigned.

[0044] [Other Embodiments 5] In the above embodiment, the image obtained using the first observation method was a reflected bright-field image, and the image obtained using the second observation method was a pseudo-phase-contrast image. However, other types of image combinations are also possible, provided that the image obtained using the first observation method is a low-contrast image and the image obtained using the second observation method is a high-contrast image. For example, the image obtained using the second observation method may be an image other than a phase-contrast image, such as a relief-contrast image (Hoffman modulation image) or a differential interference image, which is a high-contrast image.

[0045] Furthermore, various embodiments of the present invention may be described with reference to flowcharts and block diagrams, where a block may represent (1) a stage in a process in which an operation is performed or (2) a section of a device having the role of performing an operation. Specific stages and sections may be implemented by dedicated circuits, programmable circuits supplied with computer-readable instructions stored on a computer-readable medium, and / or processors supplied with computer-readable instructions stored on a computer-readable medium. Dedicated circuits may include digital and / or analog hardware circuits, and may include integrated circuits (ICs) and / or discrete circuits. Programmable circuits may include reconfigurable hardware circuits, including logical AND, logical OR, logical XOR, logical NAND, logical NOR, and other logic operations, flip-flops, registers, memory elements such as field-programmable gate arrays (FPGAs), programmable logic arrays (PLAs), etc.

[0046] Computer-readable media may include any tangible device capable of storing instructions to be executed by a suitable device, and as a result, computer-readable media having instructions stored therein will comprise a product containing instructions that can be executed to create means for performing operations specified in a flowchart or block diagram. Examples of computer-readable media may include electronic storage media, magnetic storage media, optical storage media, electromagnetic storage media, semiconductor storage media, etc. More specific examples of computer-readable media may include floppy disks, diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), electrically erasable programmable read-only memory (EEPROM), static random access memory (SRAM), compact disk read-only memory (CD-ROM), digital versatile disk (DVD), Blu-ray (RTM) disk, memory stick, integrated circuit card, etc.

[0047] Computer-readable instructions may include assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk®, Java®, C++, and traditional procedural programming languages ​​such as the C programming language or similar programming languages.

[0048] Computer-readable instructions may be provided locally or via a wide area network (WAN), such as a local area network (LAN) or the internet, to a processor or programmable circuit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, and these instructions may be executed to create means for performing operations specified in a flowchart or block diagram. Examples of processors include computer processors, processing units, microprocessors, digital signal processors, controllers, microcontrollers, and the like.

[0049] Figure 11 shows an example of a computer 2200 in which multiple aspects of the present invention may be embodied in whole or in part. A program installed on the computer 2200 can cause the computer 2200 to function as an operation or one or more sections of an apparatus according to an embodiment of the present invention, or to execute such operation or one or more sections, and / or to cause the computer 2200 to execute a process or a stage of such process according to an embodiment of the present invention. Such a program may be executed by the CPU 2212 to cause the computer 2200 to perform a particular operation associated with some or all of the blocks in the flowcharts and block diagrams described herein.

[0050] The computer 2200 according to this embodiment includes a CPU 2212, RAM 2214, a graphics controller 2216, and a display device 2218, which are interconnected by a host controller 2210. The computer 2200 also includes input / output units such as a communication interface 2222, a hard disk drive 2224, a DVD-ROM drive 2226, and an IC card drive, which are connected to the host controller 2210 via an input / output controller 2220. The computer also includes legacy input / output units such as a ROM 2230 and a keyboard 2242, which are connected to the input / output controller 2220 via an input / output chip 2240.

[0051] The CPU 2212 operates according to programs stored in the ROM 2230 and RAM 2214, thereby controlling each unit. The graphics controller 2216 retrieves image data generated by the CPU 2212 from a frame buffer provided in RAM 2214 or from itself, and displays the image data on the display device 2218.

[0052] The communication interface 2222 communicates with other electronic devices via a network. The hard disk drive 2224 stores programs and data used by the CPU 2212 in the computer 2200. The DVD-ROM drive 2226 reads programs or data from the DVD-ROM 2201 and provides them to the hard disk drive 2224 via the RAM 2214. The IC card drive reads programs and data from the IC card and / or writes programs and data to the IC card.

[0053] The ROM 2230 stores boot programs and / or programs that depend on the computer 2200's hardware, which are executed by the computer 2200 when activated. The input / output chip 2240 may also connect various input / output units to the input / output controller 2220 via parallel ports, serial ports, keyboard ports, mouse ports, etc.

[0054] The program is provided on a computer-readable medium such as a DVD-ROM 2201 or an IC card. The program is read from the computer-readable medium and installed on a hard disk drive 2224, RAM 2214, or ROM 2230, which are also examples of computer-readable medium, and executed by the CPU 2212. The information processing described within these programs is read by the computer 2200, resulting in coordination between the program and the various types of hardware resources described above. The apparatus or method may be configured to realize the manipulation or processing of information in accordance with the use of the computer 2200.

[0055] For example, when communication is performed between a computer 2200 and an external device, the CPU 2212 may execute a communication program loaded into RAM 2214 and, based on the processing described in the communication program, instruct the communication interface 2222 to perform communication processing. Under the control of the CPU 2212, the communication interface 2222 reads transmission data stored in a transmission buffer processing area provided in a recording medium such as RAM 2214, a hard disk drive 2224, a DVD-ROM 2201, or an IC card, transmits the read transmission data to the network, or writes received data received from the network to a reception buffer processing area provided on the recording medium.

[0056] Furthermore, the CPU 2212 may read all or necessary parts of files or databases stored on external storage media such as the hard disk drive 2224, DVD-ROM drive 2226 (DVD-ROM 2201), or IC card into the RAM 2214, and perform various types of processing on the data in the RAM 2214. The CPU 2212 then writes the processed data back to the external storage media.

[0057] Various types of information, such as various types of programs, data, tables, and databases, may be stored on the recording medium and subjected to information processing. The CPU 2212 may perform various types of processing on the data read from RAM 2214, including various types of operations, information processing, conditional judgments, conditional branching, unconditional branching, information retrieval / replacement, etc., as described throughout this disclosure and specified by the program instruction sequence, and write the results back to RAM 2214. The CPU 2212 may also retrieve information in files, databases, etc., within the recording medium. For example, if multiple entries are stored in the recording medium, each having an attribute value of a first attribute associated with an attribute value of a second attribute, the CPU 2212 may search among the multiple entries for an entry that matches the condition for which the attribute value of the first attribute is specified, read the attribute value of the second attribute stored in that entry, and thereby obtain the attribute value of the second attribute associated with the first attribute that satisfies a predetermined condition.

[0058] The programs or software modules described above may be stored on or near computer 2200 on a computer-readable medium. Alternatively, recording media such as hard disks or RAM provided within a server system connected to a dedicated communication network or the Internet can be used as computer-readable media, thereby providing programs to computer 2200 via the network.

[0059] Although the present invention has been described above using embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments. It will be apparent to those skilled in the art that various modifications or improvements can be made to the above embodiments. It will be clear from the claims that such modified or improved forms may also be included in the technical scope of the present invention.

[0060] It should be noted that the execution order of operations, procedures, steps, and stages in the devices, systems, programs, and methods shown in the claims, specifications, and drawings is not explicitly stated as "before," "prior to," etc., and that these can be performed in any order unless the output of a previous process is used in a later process. Even if the operation flow in the claims, specifications, and drawings is described using phrases such as "first," "next," etc. for convenience, this does not mean that it is mandatory to perform the operations in that order. [Explanation of Symbols]

[0061] 11 Image conversion unit, 12 Trained model, 13 Mask image generation unit, 14 Noise reduction unit, 15 Display control unit, 16 Parameter setting unit, 100 Image processing unit, 110 Image processing unit, 200 GUI screen, 2200 Computer, 2201 DVD-ROM, 2210 Host controller, 2212 CPU, 2214 RAM, 2216 Graphics controller, 2218 Display device, 2220 Input / output controller, 2222 Communication interface, 2224 Hard disk drive, 2226 DVD-ROM drive, 2230 ROM, 2240 Input / output chip, 2242 Keyboard

Claims

1. An image conversion unit converts an input image, which is a microscope image containing a biological sample obtained by a first observation method, into a pseudo-image that appears as if it were obtained by a second observation method different from the first observation method. A mask image generation unit generates at least one continuous mask image based on the input image, which consists of a first region corresponding to the foreground including the biological sample, a second region corresponding to the background not including the biological sample, and a boundary region between the first and second regions, wherein the pixel values ​​in the boundary region change stepwise from the first region to the second region. A noise reduction unit generates an output image in which background noise is removed while maintaining the appearance of the biological sample, based on the continuous mask image and the pseudo-image. An image processing device having

2. The mask image generation unit creates a foreground extraction mask image and a background extraction mask image as the continuous mask images. The image processing apparatus according to claim 1, wherein the sum of the pixel values ​​of each pixel in the foreground extraction mask image and the pixel values ​​of each pixel in the background extraction mask image is constant.

3. The image processing apparatus according to claim 2, wherein the noise reduction unit generates the output image by combining a first intermediate image created based on the foreground extraction mask image and the pseudo-image and a second intermediate image created based on the background extraction mask image and the background image.

4. The image processing apparatus according to claim 1, wherein the first observation method is transmitted bright-field observation or reflected bright-field observation, and the second observation method is phase-contrast observation.

5. The image processing apparatus according to claim 1, wherein the image conversion unit has a trained model that converts a transmitted brightfield image or a reflected brightfield image as the input image into a pseudo-image that is a pseudo-phase difference image.

6. The image processing apparatus according to claim 1, wherein the mask image generation unit creates the continuous mask image based only on the input image.

7. The image processing apparatus according to claim 1, wherein the mask image generation unit generates the continuous mask image based on the input image and the pseudo-image.

8. The image processing apparatus according to claim 7, wherein the mask image generation unit generates the continuous mask image based on the union of the input image and the pseudo-image, or the intersection of the input image and the pseudo-image.

9. The image processing apparatus according to claim 1, wherein the mask image generation unit sets the pixel values ​​within the boundary region in steps according to the distance from the edge of the input image.

10. The image processing apparatus according to claim 1, wherein the mask image generation unit sets the pixel values ​​in the boundary region in steps according to the distance from the contour of the region where the biological sample exists in the input image.

11. The image processing apparatus according to claim 1, further comprising a display control unit that causes a screen for receiving parameter inputs for generating the continuous mask image to be displayed on the display unit.

12. The image processing apparatus according to any one of claims 1 to 11, further comprising a parameter setting unit that automatically sets parameters for generating the continuous mask image based on at least one of the type of biological sample, the magnification of the objective lens used in the first observation method, or the contrast of the input image.

13. An image conversion step in which a computer converts an input image, which is a microscopic image including a biological sample acquired by a first observation method, into a pseudo-image that looks as if it were acquired by a second observation method different from the first observation method, A mask image generation step in which the computer generates at least one continuous mask image based on the input image, the continuous mask image comprising a first region corresponding to the foreground including the biological sample, a second region corresponding to the background not including the biological sample, and a boundary region between the first region and the second region, wherein the pixel values ​​in the boundary region change stepwise from the first region to the second region. The computer performs a noise reduction step, based on the continuous mask image and the pseudo-image, to generate an output image in which background noise is removed while maintaining the appearance of the biological sample. An image processing method having the following characteristics.

14. It is an image processing program, An image conversion procedure that converts an input image, which is a microscopic image containing a biological sample obtained by a first observation method, into a pseudo-image that appears as if it were obtained by a second observation method different from the first observation method, A mask image generation procedure that generates at least one continuous mask image based on the input image, comprising a first region corresponding to the foreground containing the biological sample, a second region corresponding to the background not containing the biological sample, and a boundary region between the first and second regions, wherein the pixel values ​​within the boundary region change stepwise from the first region to the second region. A noise reduction procedure that generates an output image in which background noise is removed while maintaining the appearance of the biological sample, based on the continuous mask image and the pseudo-image, An image processing program that causes a computer to perform an image processing operation.

Images