Image processing apparatus, image processing system, image sharpening method, and program

Abstract

To sharpen the whole image appropriately.SOLUTION: An image processing device 100 comprises a classification unit 130 which classifies a plurality of areas constituting an image into a plurality of classes, and a sharpening processing unit 140 which sharpens the plurality of areas with strength determined for each classified class.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The disclosure of this specification relates to an image processing apparatus, an image processing system, a method for sharpening an image, and a program.

Background Art

[0002] In recent years, the use of digital images obtained by microscopes has been advancing in pathological diagnosis and inspection of industrial parts. However, in observations using digital images obtained by microscopes, compared with observations with the naked eye, blurring caused by the shallow depth of field of the optical system and slight optical inhomogeneities occurring between the object point and the image plane are likely to be visually recognized as a decrease in sharpness and contrast.

[0003] Image processing techniques related to such problems are described in, for example, Patent Document 1. By using the unsharp masking process described in Patent Document 1, high-frequency components of the spatial frequency can be added to the original image at an arbitrary intensity to obtain a sharpened image.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] By the way, in the technique described in Patent Document 1, the sharpening process is uniformly performed on the entire image. For this reason, in regions with high sharpness in the image, the sharpness may be overly emphasized, and the granularity may deteriorate (that is, the granular noise becomes conspicuous). In addition, regions with low sharpness in the image may be blurred without sufficient improvement in sharpness.

[0006] Note that although the microscopic images have been described as an example above, the adverse effects of the edge enhancement process that is uniformly performed can occur not only in microscopic images but also in any digital image.

[0007] Based on the above circumstances, an object according to one aspect of the present invention is to provide a technique for appropriately enhancing the sharpness of the entire image.

Means for Solving the Problem

[0009] An image processing system according to one embodiment of the present invention includes a microscope device that scans a specimen to acquire a plurality of images, the aforesaid a classification unit that classifies a plurality of regions constituting an image into a plurality of classes, a sharpening processing unit that sharpens the plurality of regions with intensities determined for each of the classified classes, a display control unit that displays a sharpened image, which is an image sharpened by the sharpening processing unit, on a display device, and a pasting unit that pastes a plurality of sharpened images, each of which is an image sharpened by the sharpening processing unit, to generate a pasted image wider than the field of view of the microscope device. The sharpening processing unit changes the intensity for sharpening the plurality of regions to a higher intensity as the display magnification of the sharpened image displayed on the display device is lower. An image processing system according to another embodiment of the present invention includes a microscope device that scans a specimen to acquire a plurality of images, the aforesaid a classification unit that classifies a plurality of regions constituting an image into a plurality of classes, a sharpening processing unit that sharpens the plurality of regions with intensities determined for each of the classified classes, a display control unit that displays a sharpened image, which is an image sharpened by the sharpening processing unit, on a display device, and a pasting unit that pastes a plurality of sharpened images, each of which is an image sharpened by the sharpening processing unit, to generate a pasted image wider than the field of view of the microscope device. The sharpening processing unit changes the intensity for sharpening the plurality of regions to an intensity corresponding to the display magnification of the sharpened image determined based on the contrast sensitivity characteristic.

[0010] The image sharpening method according to one aspect of the present invention is applicable to each of a plurality of images obtained by scanning a specimen with a microscope device, the aforesaid classify a plurality of regions constituting the image into a plurality of classes, sharpen the plurality of regions with intensities determined for each of the classified classes, display a sharpened image, which is the sharpened image, on a display device, paste together a plurality of sharpened images, each of which is a sharpened image, to generate a pasted image wider than the field of view of the microscope device, and change the intensity for sharpening the plurality of regions to a higher intensity as the display magnification of the sharpened image displayed on the display device is lower. Another sharpening method according to another aspect of the present invention is applicable to each of a plurality of images obtained by scanning a specimen with a microscope device, the aforesaid classify a plurality of regions constituting the image into a plurality of classes, sharpen the plurality of regions with intensities determined for each of the classified classes, display a sharpened image, which is the sharpened image, on a display device, paste together a plurality of sharpened images, each of which is a sharpened image, to generate a pasted image wider than the field of view of the microscope device, and change the intensity for sharpening the plurality of regions to an intensity corresponding to the display magnification of the sharpened image determined based on contrast sensitivity characteristics.

[0011] A program according to one aspect of the present invention causes a computer to execute a process of, for each of a plurality of images obtained by scanning a specimen with a microscope device, the aforesaid classify a plurality of regions constituting the image into a plurality of classes, sharpen the plurality of regions with intensities determined for each of the classified classes, display a sharpened image, which is the sharpened image, on a display device, paste together a plurality of sharpened images, each of which is a sharpened image, to generate a pasted image wider than the field of view of the microscope device, and change the intensity for sharpening the plurality of regions to a higher intensity as the display magnification of the sharpened image displayed on the display device is lower. Another program according to another aspect of the present invention causes a computer to execute a process of, for each of a plurality of images obtained by scanning a specimen with a microscope device, the aforesaidClassify a plurality of regions constituting an image into a plurality of classes, sharpen the plurality of regions with intensities determined for each of the classified classes, display a sharpened image, which is the sharpened image, on a display device, generate a composite image wider than the field of view of the microscope device by pasting a plurality of sharpened images, each of which is a sharpened image, and change the intensity for sharpening the plurality of regions to an intensity corresponding to the display magnification of the sharpened image determined based on the contrast sensitivity characteristic.

Advantages of the Invention

[0012] According to the above aspect, the entire image can be appropriately sharpened.

Brief Description of the Drawings

[0013]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Modes for Carrying Out the Invention

[0014] FIG. 1 is a diagram exemplifying the configuration of an image processing system 1 according to an embodiment. First, with reference to FIG. 1, the configuration of the image processing system 1 shown in FIG. 1 will be described. As shown in FIG. 1, the image processing system 1 includes an image processing apparatus 100, one or more imaging devices 10, and one or more client terminals 20 connected to each other via a network 30.

[0015] The type of the network is not particularly limited. The network may be, for example, a public line such as the Internet, a dedicated line, or a LAN (Local Area Network). The connection of the image processing apparatus 100, the imaging device 10, and the client terminal 20 may be a wired connection or a wireless connection.

[0016] The image processing apparatus 100 is an apparatus that performs a process of enhancing the sharpness of an input image and outputs a sharpened image (hereinafter referred to as a sharpened image). In the image processing apparatus 100, instead of uniformly sharpening the entire image, the regions constituting the image are classified, and each region is sharpened with an intensity corresponding to the class. Thereby, even when there are regions with high sharpness and regions with low sharpness mixed in the input image, a sharpened image in which the entire image is appropriately sharpened can be generated. Details of the image processing apparatus 100 will be described later.

[0017] The imaging device 10 is an apparatus that captures a specimen and acquires an image to be input to the image processing apparatus 100. The image of the specimen generated by the imaging device 10 is output from the imaging device 10 to the image processing apparatus 100 via the network 30, for example. However, the image generated by the imaging device 10 may be input to the image processing apparatus 100 via other devices or a recording medium.

[0018] The imaging device 10 may be any apparatus that captures a specimen and generates a digital image. The imaging device 10 may include, for example, a microscope device 11. Further, the imaging device 10 may include an observation device 12 that is placed in an incubator and performs time-lapse imaging of a specimen in a culture vessel. The imaging device 10 may include a digital camera 13, and further may include any other arbitrary imaging device such as a smartphone or an endoscope.

[0019] The client terminal 20 is a terminal operated by a user and has an input device and a display device. The image processing apparatus 100 may output a sharpened image to the client terminal 20, for example, in response to a request from the client terminal 20.

[0020] The client terminal 20 may be, for example, a portable type client terminal 21 such as a notebook personal computer, or a mobile type client terminal 22 such as a tablet or a smartphone. Further, it may be a stationary type client terminal 23 such as a desktop personal computer.

[0021] In FIG. 1, the case where the imaging device 10, the client terminal 20, and the image processing device 100 are separate devices located at different locations is shown as an example. However, the image processing system 1 only needs to have these functions, and the specific configuration is not limited to this example. The image processing system 1 may be, for example, a microscope system including a microscope device and a microscope control device. The image acquired by the microscope device may be sharpened by the microscope control device, and the generated sharpened image may be displayed on a display device provided in the microscope control device.

[0022] FIG. 2 is a functional block diagram showing the functional configuration of the image processing device 100 according to an embodiment. FIG. 3 is an example of a flowchart of the processing performed by the image processing device 100. FIG. 4 is a diagram for explaining a method of generating a sharpened image. FIG. 5 is a diagram for explaining the size of the region R. Hereinafter, with reference to FIGS. 2 to 5, the configuration common to the image processing devices according to the following embodiments and the method of sharpening an image will be described by taking the image processing device 100 as an example.

[0023] As shown in FIG. 2, the image processing device 100 includes an acquisition unit 110, a division unit 120, a classification unit 130, a sharpening processing unit 140, and an output unit 150.

[0024] When the processing shown in FIG. 3 is started, in the image processing device 100, first, the acquisition unit 110 acquires an image of a specimen photographed by the imaging device 10 (step S1). Here, for example, the acquisition unit 110 reads the image 41 in FIG. 4 stored in an auxiliary storage device (not shown) of the image processing device 100 into a main storage device (not shown). The image 41 is, for example, an image of a pathological specimen photographed by the microscope device 11. The image 41 shows a state in which a large number of cells are densely present.

[0025] When an image is acquired by the acquisition unit 110, the division unit 120 divides the image acquired by the acquisition unit 110 into a plurality of regions (step S2). Here, for example, the division unit 120 divides the image 41 into a plurality of regions R with a certain size. The certain size is not particularly limited, but as shown in the region information 42 of FIG. 4, region division may be performed on the image 41 so that each region is a square region (region R) with a size of n×n pixels. Also, the certain size is preferably, for example, a size that includes at least one or more cells, as shown in FIG. 5. This will be described in detail later.

[0026] When the image is divided into a plurality of regions, the classification unit 130 classifies the plurality of regions constituting the image into a plurality of classes (step S3). Here, for example, the classification unit 130 assigns one of the plurality of classes to each of the plurality of regions R as shown in the class information 43 of FIG. 4. The number of classes classified by the classification unit 130 is not particularly limited. However, as will be described later, the number of times of the sharpening process depends on the number of classes, and the more the number of classes, the more the number of processing times increases. Therefore, it is desirable that the number of classes is not too large. Note that the class information 43 of FIG. 4 shows an example in which a plurality of regions R are classified into four classes.

[0027] When the regions are classified into classes, the sharpening unit 140 sharpens the plurality of regions with the intensity determined for each of the classified classes (step S4). Here, for example, the sharpening unit 140 performs contour enhancement processing such as unsharp masking on the image 41 so that each region R of the image 41 is sharpened with the intensity corresponding to the class to which the region R is assigned, and generates a sharpened image 44. Note that the specific method of sharpening the image 41 with different intensities according to the regions in the image 41 is not particularly limited. For example, if a plurality of regions R are classified into four classes, the image 41 is sharpened with four different intensities (for example, strong, medium, weak, very weak) corresponding to each class to generate four class images, and information on the regions assigned to the class is extracted and synthesized from each of the four generated class images to generate one sharpened image 44. Note that the intensity determined by the sharpening unit 140 includes the intensity for sharpening by a factor of 1. When sharpening by a factor of 1, the sharpening unit 140 may omit the sharpening process.

[0028] When the sharpened image is generated, the output unit 150 outputs the sharpened image (step S5). Here, for example, the output unit 150 outputs the sharpened image 44 to the client terminal 20. In the client terminal 20 that has received the sharpened image from the image processing apparatus 100, for example, the sharpened image 44 is displayed on the display device of the client terminal 20.

[0029] In the image processing apparatus 100 configured as described above, a plurality of regions constituting the image are classified into a plurality of classes, and the plurality of regions are sharpened with the intensity corresponding to the classified classes. Thereby, even when regions with high sharpness and regions with low sharpness are mixed in the image, each region can be sharpened with an appropriate intensity. Therefore, it is possible to avoid a situation where a specific region is oversharpened or not sharpened sufficiently, and appropriately sharpen the entire image.

[0030] Also, in the image processing apparatus 100, classification into classes is performed in units of regions rather than in units of pixels, which are the minimum units of an image. This feature greatly contributes to the image processing apparatus 100 appropriately sharpening an image. Hereinafter, this point will be specifically described.

[0031] Differences in sharpness in an image are caused by various factors. For example, in the case of a microscopic image, generally, the vicinity of the optical axis has good aberration correction and high sharpness, while in the peripheral part of the field of view, the image is likely to be blurred and the sharpness is low due to the influence of aberration. Such differences in sharpness due to the performance of the apparatus usually occur in units larger than the pixel unit. Also, when the specimen is placed inclined with respect to the optical system, regions with high and low sharpness can occur due to differences in the degree of focus. Similar to the differences in sharpness caused by the performance of the apparatus, the differences in sharpness caused by the imaging environment typified by such a specimen placement also occur in units larger than the pixel unit. In view of the purpose of observing the specimen well, it is desirable that these differences in sharpness are corrected so as to be reduced during the sharpening process.

[0032] On the other hand, it is not always desirable to correct the differences in sharpness caused by the specimen itself, which is the observation target. Rather, it may be desirable to maintain the differences as they are in some cases. Taking the cells shown in FIG. 5 as an example, differences in sharpness can occur within the cell depending on the cell structure. Specifically, regions with high and low sharpness occur within the cell, such as the cell nucleus tending to be visualized with good contrast compared to the cell membrane. However, since such differences in sharpness caused by the cell structure are considered to represent the cell structure itself, it is not always desirable to correct these differences by sharpness processing. Note that the differences in sharpness due to the structure also depend on the observation magnification but occur in pixel units or units close thereto.

[0033] Therefore, the image processing apparatus 100 makes good correction of only the difference in sharpness to be corrected by using the difference in units, that is, the spatial size, between the difference in sharpness to be corrected and the difference in sharpness not to be corrected. Specifically, the relationship is utilized that the difference in sharpness not to be corrected due to the structure of the specimen or the like occurs in a smaller unit than the difference in sharpness to be corrected due to the apparatus or the environment or the like. More specifically, the image processing apparatus 100 divides an image into intermediate units that are smaller than the above-described large unit and larger than the above-described small unit, classifies each of a plurality of regions obtained by dividing the image into classes, and performs a sharpening process on the regions classified into the classes with an intensity corresponding to the class.

[0034] In this way, by performing the sharpening process with a region rather than a pixel as the minimum unit, the inside of at least one region is uniformly sharpened with the same intensity. For this reason, for an object smaller than a region such as a cell, it is possible to appropriately sharpen while maintaining the difference in sharpness due to its structure. In addition, the difference in sharpness due to the apparatus performance or the imaging environment can be corrected so that the difference in sharpness becomes small by changing the intensity in units of regions and performing the sharpening process.

[0035] Hereinafter, each embodiment of the present invention will be specifically described. [First Embodiment] FIG. 6 is a functional block diagram showing a functional configuration of an image processing apparatus 101 according to the present embodiment. FIG. 7 is an example of a flowchart of processing performed by the image processing apparatus 101.

[0036] As shown in FIG. 6, the image processing apparatus 101 is different from the image processing apparatus 100 in that it includes a detection unit 160 that detects the sharpness of a plurality of regions based on an image. The detection unit 160 further includes a calculation unit 161 that calculates a sharpness distribution of the image, and a determination unit 162 that determines the sharpness of a plurality of regions based on the sharpness distribution calculated by the calculation unit 161.

[0037] When the image processing apparatus 101 configured as described above starts the process shown in FIG. 7, first, the acquisition unit 110 acquires an image of the specimen photographed by the photographing apparatus 10 (step S11). This process is the same as the process of step S1 in FIG. 3.

[0038] Next, the division unit 120 divides the image acquired in step S11 into a plurality of regions of a predetermined size (step S12). This predetermined size is a size determined in advance, for example, n×n pixels.

[0039] Thereafter, the calculation unit 161 calculates the sharpness distribution of the image acquired in step S11 (step S13). Here, the calculation unit 161 performs, for example, a smoothing differentiation process on the image. By extracting edges while smoothing, it is possible to calculate a sharpness distribution that suppresses the influence of noise. Specifically, a Sobel filter can be used, but for example, other edge detection methods such as Canny's edge detection algorithm may be used to calculate the sharpness distribution.

[0040] When the image is a color image, the color image may be converted into a grayscale image, and the sharpness distribution may be calculated for the obtained grayscale image.

[0041] When the sharpness distribution is detected, the determination unit 162 determines the sharpness of the plurality of regions based on the sharpness distribution (step S14). Here, the determination unit 162 performs, for example, an averaging process on the sharpness within each of the plurality of regions specified by the division unit 120. That is, the determination unit 162 determines the average value of the sharpness at each position within the region as the sharpness of the region. Note that the sharpness of the region may be a value representing the sharpness distribution of the region, and is not necessarily limited to the average value. For example, the median value or the mode value may be used.

[0042] When the sharpness for each region is determined, the classification unit 130 classifies the plurality of regions into a plurality of classes according to the sharpness of the plurality of regions determined in step S14 (step S15). Here, the classification unit 130 classifies the plurality of regions into a plurality of classes, for example, based on a predetermined threshold value. That is, by comparing the threshold value with the sharpness of the region, the region is classified into one of the plurality of classes. As the threshold value, an absolute value (for example, sharpness 100, etc.) may be used, or a relative value (for example, 1 / 2 of the difference between the maximum sharpness and the minimum sharpness) may be used.

[0043] Note that, as an example of the output result of step S15, for example, a filter composed of a plurality of pixels having the same number as the number of classes and each corresponding to a plurality of regions may be generated. In this case, the pixels of each filter are binary data that distinguish between the classified regions (0) of the corresponding class and the others (1).

[0044] When the classification into classes is performed, the sharpening processing unit 140 sharpens the plurality of regions for each class with an intensity corresponding to the sharpness of the regions classified into that class (step S16). Here, the sharpening processing unit 140 generates a sharpened image, for example, by sharpening the plurality of regions for each classified class with a higher intensity as the sharpness of the regions classified into that class is lower. More specifically, each region may be sharpened with an intensity inversely proportional to the sharpness.

[0045] Note that in step S16, by sharpening the entire image with an intensity corresponding to the class, a plurality of images having the same number as the number of classes may be generated. Then, for the image corresponding to each class, only the information of the regions classified into that class may be extracted using the filter corresponding to each class generated in step S15. Specifically, for example, the image corresponding to each class may be filtered with the filter corresponding to each class, and the filtered image may be subtracted from the image corresponding to the class to extract only the information of the regions classified into that class. A sharpened image may be generated by synthesizing the information extracted for each filter class.

[0046] When the sharpened image is generated, the output unit 150 outputs the sharpened image (step S17). This process is the same as the process of step S5 in FIG. 3.

[0047] Also by the image processing apparatus 101 according to the present embodiment, similar to the image processing apparatus 100, it is possible to appropriately sharpen the entire image. Further, according to the image processing apparatus 101, since a plurality of regions are classified based on the sharpness distribution of the image, it is possible to perform classification reflecting the sharpness of the image to be sharpened. Furthermore, by sharpening each region with an intensity corresponding to the sharpness of the region, it is possible to satisfactorily suppress the sharpness difference between the regions.

[0048] [Second Embodiment] FIG. 8 is a functional block diagram showing the functional configuration of the image processing apparatus 102 according to the present embodiment. As shown in FIG. 8, the image processing apparatus 102 is different from the image processing apparatus 101 according to the first embodiment in that it includes a setting unit 170 that sets operation parameters for the operations of the division unit 120, the classification unit 130, and the sharpening processing unit 140. Other configurations are the same as those of the image processing apparatus 101.

[0049] The setting unit 170 sets operation parameters according to an input from the user, for example, sets the values specified by the user for each operation parameter. Specifically, the region size, which is an operation parameter used in the division process in the division unit 120, the number of classes and threshold values, which are operation parameters used in the classification process in the classification unit 130, and a coefficient for determining the sharpening intensity, which is an operation parameter used in the sharpening process in the sharpening processing unit 140, may be set according to the user's specification. Also, it may be set to omit the sharpening process itself for a specific class.

[0050] Also, the image processing apparatus 102 according to the present embodiment can appropriately sharpen the entire image and can suppress the sharpness difference between regions as well as the image processing apparatus 101. Further, according to the image processing apparatus 102, since the operation parameters can be adjusted by the setting unit 170, the image can be sharpened according to the characteristics of the image (specimen) and the user's preference.

[0051] Note that, in the image processing apparatus 102, an example in which the setting unit 170 sets the operation parameters of the division unit 120, the classification unit 130, and the sharpening processing unit 140 is shown, but the setting unit 170 may set at least any of these operation parameters.

[0052] [Third Embodiment] FIG. 9 is a functional block diagram showing the functional configuration of the image processing apparatus 103 according to the present embodiment. As shown in FIG. 9, the image processing apparatus 103 is the same as the image processing apparatus 101 according to the first embodiment in that it includes a detection unit 160 that detects the sharpness of a plurality of regions based on an image, and the sharpness of the plurality of regions detected by the detection unit 160 is also used by the classification unit 130 in the same manner as the image processing apparatus 101. However, the image processing apparatus 103 is different from the image processing apparatus 101 in that the sharpness of the plurality of regions detected by the detection unit 160 is not used by the sharpening processing unit 140.

[0053] In the image processing apparatus 103, the sharpening processing unit 140 sharpens a plurality of regions with a predetermined intensity for each class classified according to the sharpness of the plurality of regions. Note that, although the sharpening processing unit 140 does not directly reflect the sharpness of the region, such as being inversely proportional to the sharpness of the region, it is desirable to sharpen the region with a lower predetermined intensity as the sharpness of the region is higher.

[0054] Also, the image processing apparatus 103 according to the present embodiment can appropriately sharpen the entire image and can suppress the sharpness difference between regions as well as the image processing apparatus 101.

[0055] [Fourth Embodiment] FIG. 10 is a functional block diagram showing the functional configuration of the image processing apparatus 104 according to the present embodiment. As shown in FIG. 10, the image processing apparatus 104 is the same as the image processing apparatus 101 according to the first embodiment in that it includes a detection unit 160 that detects the sharpness of a plurality of regions based on an image, and the sharpness of the plurality of regions detected by the detection unit 160 is also used by the sharpness processing unit 140, the same as the image processing apparatus 101. However, the image processing apparatus 104 is different from the image processing apparatus 101 in that the sharpness of the plurality of regions detected by the detection unit 160 is not used by the classification unit 130.

[0056] In the image processing apparatus 104, the classification unit 130 classifies a plurality of regions into a plurality of classes according to, for example, the distance from a predetermined field of view center. More specifically, from the closer to the field of view center, it may be classified into a class that sharpens strongly, a class that sharpens with medium strength, and a class that sharpens weakly. Then, the sharpness processing unit 140 determines the intensity of sharpening specifically according to the sharpness of the regions classified into the classes and performs sharpening.

[0057] Also by the image processing apparatus 104 according to the present embodiment, as with the image processing apparatus 101, it is possible to appropriately sharpen the entire image and to satisfactorily suppress the sharpness difference between regions.

[0058] [Fifth Embodiment] FIG. 11 is a functional block diagram showing the functional configuration of the image processing apparatus 105 according to the present embodiment. As shown in FIG. 11, the image processing apparatus 105 is different from the image processing apparatus 101 according to the first embodiment in that it includes a stitching unit 180 that stitches a plurality of sharpened images to generate a stitched image. Other configurations are the same as those of the image processing apparatus 101.

[0059] In this embodiment, the microscope apparatus 11 scans a specimen by sequentially moving the field of view with respect to the specimen using an objective lens with a relatively high magnification to obtain a plurality of images. The image processing apparatus 105 sharpens each of the plurality of images corresponding to a range wider than the field of view of the microscope apparatus 11 thus obtained, and then stitches them together to generate a stitched image wider and with higher resolution than the field of view of the microscope apparatus 11.

[0060] Also, with the image processing apparatus 105 according to this embodiment as well, similar to the image processing apparatus 101, it is possible to appropriately sharpen the entire image, and it is possible to satisfactorily suppress the sharpness difference between regions. Further, due to the general characteristic of a microscope apparatus that the image quality of the peripheral part of the field of view deteriorates compared to the central part, if the images obtained by the microscope apparatus are stitched together as they are, periodic stripe patterns depending on the stitching period may appear in the stitched image. According to the image processing apparatus 105, by stitching together sharpened images with the sharpness difference between regions suppressed to generate a stitched image, it is possible to prevent or suppress the occurrence of such artifacts, and it is possible to generate a high-quality stitched image in which the stitching part is not conspicuous. Also, an improvement in the stitching accuracy itself can be expected.

[0061] FIGS. 12 and 13 are flowcharts of the processing performed by the image processing system including the image processing apparatus 105. When generating a stitched image, the sharpening process may be performed in synchronization with image acquisition, or may be performed asynchronously with image acquisition.

[0062] For example, as shown in FIG. 12, when the image processing system receives a stitched image generation instruction from the user (step S21), the process of the microscope apparatus 11 acquiring an image (step S22) and the process of the image processing apparatus 105 performing a sharpening process on the image acquired by the microscope apparatus 11 (step S23) are repeated the required number of times. Then, when the repetitive process ends (NO in step S24), the image processing apparatus 105 may generate a stitched image using the plurality of sharpened images sharpened in step S23 (step S25).

[0063] Also, for example, as shown in FIG. 13, in the image processing system, an image may be appropriately acquired by the microscope apparatus 11 (steps S31 and S32). Thereafter, when a stitching image generation instruction including designation of an image to be used is received from the user (step S33), the imaging device 10 sequentially sharpens the designated image (steps S34 and S35), and may generate a stitching image using the plurality of generated sharpened images (step S36).

[0064] In the above description, the relationship between the timing of image acquisition and the timing of the sharpening process has been explained. However, when the user gives an instruction to generate a stitching image, a check box or the like for setting whether to perform the sharpening process may be provided on the user interface so that the user can select whether to perform the sharpening process before stitching the images. Thus, when degradation of the image quality can be tolerated, the user can select to omit the sharpening process in order to shorten the processing time for generating the stitching image. Further, a check box for setting whether to save the image before the sharpening process may be provided. Thereby, it is possible to save the storage capacity by omitting the saving of the image before the sharpening process.

[0065] [Sixth Embodiment] FIG. 14 is a functional block diagram showing the functional configuration of the image processing apparatus 106 according to the present embodiment. As shown in FIG. 14, the image processing apparatus 106 is different from the image processing apparatus 101 according to the first embodiment in that it includes a display control unit 151 instead of the output unit 150 and there is feedback from the display control unit 151 to the sharpening unit 140. Other configurations are the same as those of the image processing apparatus 101.

[0066] The display control unit 151 displays the sharpened image on a display device such as the client terminal 20, for example. The display control unit 151 further feeds back the display magnification of the sharpened image displayed on the display device to the sharpening processing unit 140. The sharpening processing unit 140 changes the intensity of sharpening a plurality of regions according to the display magnification fed back from the display control unit 151. At this time, the sharpening processing unit 140 may determine the sharpening intensity using, for example, a mathematical model of contrast sensitivity characteristics. Note that the fed-back display magnification may be the overall magnification or the magnification of digital zoom by the display control unit 151 for the sharpened image output from the sharpening processing unit 140.

[0067] Also, with the image processing apparatus 106 according to the present embodiment, similar to the image processing apparatus 101, it is possible to appropriately sharpen the entire image and to satisfactorily suppress the sharpness difference between regions. Further, by feeding back the display magnification to the sharpening processing unit 140, an image sharpened with an intensity optimized for the display magnification can be displayed on the display device.

[0068] For example, taking the bonded image as an example, it is known that the boundary of bonding is more prominent when displayed at a low magnification, and the boundary of bonding is not prominent when displayed at a high magnification. This is related to the fact that the contrast sensitivity of the human eye is generally maximized at a spatial frequency of about 10 cycles / deg. Thus, the necessity of sharpening processing may vary depending on the display magnification. Among users who obtain analysis results from images, such as researchers using microscope images, there are also users who do not prefer image processing. However, according to the image processing apparatus 105, it is also possible to provide the user with an unprocessed sharpened image when the display magnification is high.

[0069] [Seventh Embodiment] FIG. 15 is a functional block diagram showing the functional configuration of the image processing apparatus 107 according to the present embodiment. As shown in FIG. 15, the image processing apparatus 107 is different from the image processing apparatus 101 according to the first embodiment in that it includes a detection unit 190 instead of the detection unit 160. Other configurations are the same as those of the image processing apparatus 101.

[0070] The detection unit 190 detects the sharpness characteristics of the imaging device 10 that acquires the image to be sharpened. The sharpness characteristics of the imaging device 10 may be, for example, aberration characteristics such as image curvature that the imaging device 10 has. These sharpness characteristics may be calculated from an image of a calibration specimen with a flat surface taken by the imaging device 10.

[0071] In the image processing apparatus 107, the classification unit 130 classifies a plurality of regions into a plurality of classes according to the sharpness characteristics of the imaging device 10 detected by the detection unit 190, rather than the sharpness distribution of the image. Further, the sharpness processing unit 140 sharpens a plurality of regions with an intensity corresponding to a portion of the sharpness characteristics corresponding to the regions classified into that class, for each of the classified classes, rather than the sharpness of the regions classified into that class.

[0072] Also with the image processing apparatus 107 according to the present embodiment, as with the image processing apparatus 101, it is possible to appropriately sharpen the entire image and to satisfactorily suppress the sharpness difference between regions. In particular, it is suitable for observing a specimen with ensured flatness, such as those used in industrial applications, and it is possible to appropriately sharpen the entire image in a shorter time by omitting the step of detecting the sharpness distribution from the image.

[0073] Note that, in the image processing apparatus 107, an example in which both the classification unit 130 and the sharpness processing unit 140 use the sharpness characteristics has been shown, but the sharpness characteristics may be used by only one of the classification unit 130 and the sharpness processing unit 140.

[0074] FIG. 16 is a diagram illustrating the hardware configuration of a computer 200 for realizing the above-described image processing apparatus. The hardware configuration shown in FIG. 16 includes, for example, a processor 201, a memory 202, a storage device 203, a reading device 204, a communication interface 206, and an input / output interface 207. Note that the processor 201, the memory 202, the storage device 203, the reading device 204, the communication interface 206, and the input / output interface 207 are connected to each other via, for example, a bus 208.

[0075] The processor 201 may be, for example, a single processor, a multi-processor, or a multi-core processor. The processor 201 reads and executes a program stored in the storage device 203, thereby operating as the above-described acquisition unit 110, division unit 120, classification unit 130, sharpness processing unit 140, output unit 150, detection unit 160, setting unit 170, pasting unit 180, and detection unit 190.

[0076] The memory 202 is, for example, a semiconductor memory and may include a RAM area and a ROM area. The storage device 203 is, for example, a semiconductor memory such as a hard disk or a flash memory, or an external storage device.

[0077] The reading device 204 accesses a storage medium 205 in accordance with an instruction from the processor 201, for example. The storage medium 205 is realized by, for example, a semiconductor device, a medium in which information is input / output by a magnetic action, a medium in which information is input / output by an optical action, or the like. Note that the semiconductor device is, for example, a USB (Universal Serial Bus) memory. Also, the medium in which information is input / output by a magnetic action is, for example, a magnetic disk. The medium in which information is input / output by an optical action is, for example, a CD (Compact Disc)-ROM, a DVD (Digital Versatile Disk), a Blu-ray Disc, etc. (Blu-ray is a registered trademark).

[0078] The communication interface 206 communicates with other devices, for example, according to the instructions of the processor 201. The input / output interface 207 is an interface, for example, between an input device and an output device. The input device is a device such as a keyboard, a mouse, or a touch panel that receives instructions from a user. The output device is a display device such as a display and an audio device such as a speaker. The above-described acquisition unit 110 and output unit 150 may include at least one of the communication interface 206 or the input / output interface 207.

[0079] The program executed by the processor 201 is provided to the computer 200, for example, in the following forms. (1) It is pre-installed in the storage device 203. (2) It is provided by the storage medium 205. (3) It is provided from a server such as a program server.

[0080] Note that the hardware configuration of the computer 200 for realizing the image processing apparatus described with reference to FIG. 16 is an example, and the embodiments are not limited thereto. For example, a part of the above-described configuration may be deleted, or a new configuration may be added. Further, in another embodiment, for example, some or all of the functions of the above-described electric circuit may be implemented as hardware by an FPGA (Field Programmable Gate Array), an SoC (System-on-a-Chip), an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or the like.

[0081] The above-described embodiments are presented with specific examples to facilitate understanding of the invention, and the present invention is not limited to these embodiments. Modified forms of the above-described embodiments and alternative forms substituting the above-described embodiments may be included. That is, each embodiment can be configured by modifying the components without departing from the spirit and scope thereof. Further, by appropriately combining a plurality of components disclosed in one or more embodiments, a new embodiment can be implemented. Also, some components may be deleted from the components shown in each embodiment, or some components may be added to the components shown in the embodiment. Furthermore, the processing procedures shown in each embodiment may be performed in a swapped order as long as there is no contradiction. That is, the image processing apparatus, image processing system, image sharpening method, and program of the present invention can be variously modified and changed without departing from the description of the claims.

Explanation of Signs

[0082] 1 Image processing system 10 Photographing device 11 Microscope device 12 Observation device 13 Digital camera 20 - 23 Client terminal 30 Network 41 Image 42 Region information 43 Class information 44 Sharpened image 100 - 107 Image processing apparatus 110 Acquisition unit 120 Division unit 130 Classification unit 140 Sharpening processing unit 150 Output unit 151 Display control unit 160, 190 Detection unit 161 Calculation unit 162 Decision unit 170 Setting unit 180 Stitching unit 200 Computer 201 Processor 202 Memory 203 Storage Device 204 Reading Device 205 Memory Medium 206 Communication Interface 207 Input / Output Interface 208 Bus R Region

Claims

1. A microscope apparatus that scans a specimen to obtain a plurality of images, a classification unit that classifies a plurality of regions constituting the images into a plurality of classes, a sharpening processing unit that sharpens the plurality of regions with intensities determined for each of the classified classes, a display control unit that displays a sharpened image, which is an image sharpened by the sharpening processing unit, on a display device, and a stitching unit that stitches a plurality of sharpened images, each of which is an image sharpened by the sharpening processing unit, to generate a stitched image wider than the field of view of the microscope apparatus, and an image processing apparatus comprising: the sharpening processing unit changes the intensity for sharpening the plurality of regions to a higher intensity as the display magnification of the sharpened image displayed on the display device is lower An image processing system characterized by this.

2. A microscope apparatus that scans a specimen to obtain a plurality of images, a classification unit that classifies a plurality of regions constituting the images into a plurality of classes, a sharpening processing unit that sharpens the plurality of regions with intensities determined for each of the classified classes, a display control unit that displays a sharpened image, which is an image sharpened by the sharpening processing unit, on a display device, and a stitching unit that stitches a plurality of sharpened images, each of which is an image sharpened by the sharpening processing unit, to generate a stitched image wider than the field of view of the microscope apparatus, and an image processing apparatus comprising: the sharpening processing unit changes the intensity for sharpening the plurality of regions to an intensity corresponding to the display magnification of the sharpened image determined based on the contrast sensitivity characteristic An image processing system characterized by this.

3. In the image processing system according to claim 1 or claim 2, the image processing apparatus further comprises a detection unit that detects the sharpness of the plurality of regions based on the image, the classification unit classifies the plurality of regions into the plurality of classes according to the sharpness of the plurality of regions detected by the detection unit An image processing system characterized by this.

4. In the image processing system according to claim 3, the sharpening processing unit sharpens the plurality of regions for each of the classified classes with an intensity corresponding to the sharpness of the regions classified into the class detected by the detection unit An image processing system characterized by this.

5. In the image processing system according to claim 1 or claim 2, the image processing apparatus further comprises a detection unit that detects the sharpness of the plurality of regions based on the image, The sharpness processing unit sharpens the plurality of regions for each classified class with an intensity corresponding to the sharpness detected by the detection unit of the regions classified into the class. An image processing system characterized by this. **Claim 6** In the image processing system according to any one of Claims 3 to 5, the sharpness processing unit sharpens the plurality of regions for each classified class with a higher intensity as the sharpness of the regions classified into the class is lower. An image processing system characterized by this. **Claim 7** In the image processing system according to any one of Claims 3 to 6, the detection unit includes a calculation unit that calculates the sharpness distribution of the image, and a determination unit that determines the sharpness of the plurality of regions based on the sharpness distribution calculated by the calculation unit. An image processing system characterized by this. **Claim 8** In the image processing system according to Claim 7, the calculation unit performs a smoothing differentiation process on the image. An image processing system characterized by this. **Claim 9** In the image processing system according to Claim 7 or Claim 8, the determination unit performs an averaging process on the sharpness within each of the plurality of regions. An image processing system characterized by this. **Claim 10** In the image processing system according to Claim 1 or Claim 2, the image processing apparatus further includes a detection unit that detects the sharpness characteristics of an imaging device that acquires the image, which is the microscope device, and the classification unit classifies the plurality of regions into the plurality of classes according to the sharpness characteristics detected by the detection unit. An image processing system characterized by this. **Claim 11** In the image processing system according to Claim 10, the sharpness processing unit sharpens the plurality of regions for each classified class with an intensity corresponding to a portion of the sharpness characteristics detected by the detection unit that corresponds to the regions classified into the class. An image processing system characterized by this. **Claim 12** In the image processing system according to Claim 1 or Claim 2, the image processing apparatus further includes a detection unit that detects the sharpness characteristics of an imaging device that acquires the image, which is the microscope device, and the sharpness processing unit sharpens the plurality of regions for each classified class with an intensity corresponding to a portion of the sharpness characteristics detected by the detection unit that corresponds to the regions classified into the class. An image processing system characterized by this. **Claim 13** In the image processing system according to any one of claims 1 to 12, the image processing apparatus further includes a dividing unit that divides the image into the plurality of regions in a predetermined size The image processing system is characterized in that.

14. In the image processing system according to claim 13, the image processing apparatus further includes a setting unit that sets at least one operation parameter among the classification unit, the sharpening processing unit, and the dividing unit, the setting unit sets the operation parameter according to an input from a user The image processing system is characterized in that.

15. For each of a plurality of images obtained by scanning a specimen with a microscope apparatus, classify a plurality of regions constituting the image into a plurality of classes, sharpen the plurality of regions with an intensity determined for each classified class, display a sharpened image, which is the sharpened image, on a display device, paste together a plurality of sharpened images, each of which is a sharpened image, to generate a pasted image wider than the field of view of the microscope apparatus, according to the display magnification of the sharpened image displayed on the display device, change the intensity for sharpening the plurality of regions to a higher intensity as the display magnification is lower A method for sharpening an image, characterized in that.

16. For each of a plurality of images obtained by scanning a specimen with a microscope apparatus, classify a plurality of regions constituting the image into a plurality of classes, sharpen the plurality of regions with an intensity determined for each classified class, display a sharpened image, which is the sharpened image, on a display device, paste together a plurality of sharpened images, each of which is a sharpened image, to generate a pasted image wider than the field of view of the microscope apparatus, change the intensity for sharpening the plurality of regions to an intensity corresponding to the display magnification of the sharpened image determined based on the contrast sensitivity characteristic A method for sharpening an image, characterized in that.

17. On a computer, For each of a plurality of images obtained by scanning a specimen with a microscope apparatus, classify a plurality of regions constituting the image into a plurality of classes, sharpen the plurality of regions with an intensity determined for each classified class, display a sharpened image, which is the sharpened image, on a display device, paste together a plurality of sharpened images, each of which is a sharpened image, to generate a pasted image wider than the field of view of the microscope apparatus, According to the display magnification of the sharpened image displayed on the display device, change the intensity of sharpening the plurality of regions to a higher intensity as the display magnification is lower. A program characterized by causing the above processing to be executed.

18. On a computer, For each of a plurality of images obtained by scanning a specimen with a microscope device, Classify a plurality of regions constituting the image into a plurality of classes, Sharpen the plurality of regions with an intensity determined for each classified class, Display a sharpened image, which is a sharpened image, on a display device, Stitch together a plurality of sharpened images, each of which is a sharpened image, to generate a stitched image wider than the field of view of the microscope device, Change the intensity of sharpening the plurality of regions to an intensity corresponding to the display magnification of the sharpened image determined based on the contrast sensitivity characteristic. A program characterized by causing the above processing to be executed.

Images