Image processing device and computer program

Abstract

An image processing unit (30) provided to an image processing device includes: a machine learning model (300) that outputs, for each of a plurality of classification classes, an output value indicating the likelihood of individual pixels of an image (Di) of interest falling under the classification class; a region segmentation image creation unit (32) that, on the basis of an output (Do) of the machine learning model (300), creates a region segmentation image in which the pixels of the image (Di) of interest are segmented for individual regions belonging to any of the plurality of classification classes; an applicability probability calculation unit (33) that, on the basis of the output (Do) of the machine learning model (300), calculates, for each pixel of the image (Di) of interest, the probability of falling under each of the plurality of classification classes; and an evaluation image creation unit (34) that creates, for each of the classification classes, an evaluation image (Dv) which is displayed in a different display mode in accordance with the applicability probability. This makes it possible to visualize the magnitude of a reaction to the selected classification class in an image region segmentation task in which a machine learning model is used.

Description

Image processing apparatus and computer program

[0001] The present invention relates to a technique for determining a classification class to which each pixel of an image belongs using a machine learning model and dividing the image into regions.

[0002] Conventionally, machine learning models for performing region division of images, such as semantic segmentation (region division), instance segmentation (individual object region division), panoptic segmentation (whole image region division), etc., are known. A conventional machine learning model for performing region division of images is described in, for example, Patent Document 1.

[0003] In other machine learning models for image processing, there are some that can output the reaction position of a deep learning model with respect to an input image. For example, Grad-CAM for image classification can visualize the reaction position of a deep learning model with a heat map or the like. By knowing the reaction position in the image, the machine learning model has interpretability, which is very effective in judging the superiority and inferiority and characteristics of the machine learning model.

[0004] Japanese Patent Application Laid-Open No. 2016-99734

[0005] However, in a machine learning model for performing region division of an image, a general means for converting the reaction position of a deep learning model into a heat map has not been established. For this reason, the machine learning model has no interpretability, and it is difficult to judge the superiority and inferiority and characteristics of the machine learning model.

[0006] The present invention has been made in view of such circumstances, and an object thereof is to provide a technique for visualizing the magnitude of a reaction to a selected classification class in an image region division task using a machine learning model.

[0007] To solve the above problems, the first invention of the present application is an image processing apparatus comprising: a storage unit for storing a target image to be processed; an image processing unit for performing image processing on the target image; and a display unit for displaying the processed image created by the image processing unit, wherein the image processing unit includes: a machine learning model that outputs an output value indicating the probability that each pixel of the input target image belongs to a classification class for each of a plurality of classification classes; a region segmentation image creation unit that creates a region segmentation image by dividing each pixel of the target image into regions belonging to any of the plurality of classification classes based on the output of the machine learning model; a probability calculation unit that calculates the probability that each pixel of the target image belongs to each of the plurality of classification classes based on the output of the machine learning model; and an evaluation image creation unit that creates an evaluation image displayed in a different display manner according to the probability for each of the classification classes, either partially or entirely.

[0008] The second invention of this application is an image processing apparatus of the first invention, wherein the evaluation image is a heat map in which the color gradually changes according to the magnitude of the corresponding probability.

[0009] The third invention of this application is an image processing apparatus of the first invention, wherein the evaluation image is a monochromatic gradient image in which the color becomes darker as the probability of the corresponding value increases.

[0010] The fourth invention of the present application is an image processing apparatus of the third invention, wherein the evaluation image creation unit further creates an integrated evaluation image by superimposing two or three of the monochrome gradient images corresponding to two or three of the classification classes.

[0011] The fifth invention of this application is an image processing apparatus according to any one of the first to fourth inventions, wherein the corresponding probability calculation unit calculates the corresponding probability by applying a softmax function to the output value indicating the probability of each pixel of the target image belonging to one of the plurality of classification classes.

[0012] The sixth invention of this application is a computer program for image processing, which causes the computer to perform the following steps: A) input a target image to be processed into a machine learning model and obtain an output value as output for each of a plurality of classification classes that indicates the probability that each pixel of the input target image belongs to the classification class; B) create a region segmented image by dividing each pixel of the target image into regions belonging to one of the plurality of classification classes based on the output of the machine learning model; C) calculate the probability that each pixel of the target image belongs to each of the plurality of classification classes output by the machine learning model; and D) create an evaluation image that displays some or all of the classification classes in different display modes according to the probability of matching.

[0013] According to the first to sixth inventions of this application, in an image region segmentation task using a machine learning model, the magnitude of the response to a selected classification class can be visualized.

[0014] In particular, according to the fourth invention of this application, the probability of each pixel belonging to multiple classification classes can be visualized simultaneously in a single image.

[0015] This is a schematic diagram of the image processing apparatus according to the first embodiment. This is a functional block diagram of the image processing apparatus according to the first embodiment. This is a flowchart showing the flow of region segmentation processing in the image processing apparatus according to the first embodiment. This is an example of a target image converted to grayscale. This is an example of a region segmented image converted to grayscale. This is an example of a first evaluation image converted to grayscale. This is an example of a second evaluation image converted to grayscale.

[0016] Hereinafter, preferred embodiments of the present invention will be described with reference to the drawings.

[0017] <1. First Embodiment> <1-1. Configuration of Image Processing Device> Figure 1 is a schematic diagram of the image processing device 1 according to the first embodiment. Figure 2 is a functional block diagram of the image processing device 1. This image processing device 1 uses a machine learning model to determine the classification class to which each pixel of an image belongs and performs region segmentation processing to divide the image into regions.

[0018] As shown in Figure 1, this image processing device 1 comprises a computer body 11, a display unit 12, and an input unit 13. The computer body 11 includes a processor 111 such as a CPU, memory 112 such as RAM, and a storage unit 113 such as a hard disk drive. The display unit 12 displays images output from the computer body 11. The display unit 12 is, for example, a liquid crystal display such as a PC monitor. The input unit 13 can receive commands to the computer body 11. The input unit 13 is, for example, a keyboard and a mouse.

[0019] The memory 112 and the storage unit 113 are connected to the processor 111 via bus wiring, which is not shown in the diagram. The storage unit 113 stores the computer program P. The processor 111 loads the computer program P stored in the storage unit 113 into the memory 112 and executes the code contained in the computer program P sequentially. As a result, the image processing device 1 performs region division processing on the target image Di, which is the object to be processed, and classifies each pixel of the target image Di into one of several classes.

[0020] Computer program P is application software that causes the computer main unit 11 to execute various processes related to area partitioning. Computer program P is read from a storage medium M such as a CD or DVD and installed on the computer main unit 11. However, computer program P may also be downloaded to the computer main unit 11 via a network N such as the Internet.

[0021] In a typical region segmentation AI model, a target image Di is classified into multiple classes, and region segmentation images such as mask images are output, each colored according to its respective class. This image processing device 1 not only produces region segmentation images Ds obtained by dividing the target image Di into multiple classification classes, but also creates evaluation images Dv for each classification class, displayed in different display modes according to the probability of matching. These evaluation images Dv allow the probability of matching each classification class to be visually confirmed.

[0022] Figure 2 is a functional block diagram of the image processing device 1. The computer body 11 includes an image storage unit 20 and an image processing unit 30.

[0023] The image storage unit 20 is a storage unit provided within the computer main body 11. The image storage unit 20 may use the storage unit 113, or it may be a storage unit provided separately from the storage unit 113. The image storage unit 20 stores the target image Di that is to be processed.

[0024] The image processing unit 30 performs image processing on the target image Di. The image processing unit 30 includes a machine learning model 300, a region division execution unit 31, a region division image creation unit 32, a corresponding probability calculation unit 33, and an evaluation image creation unit 34. The region division execution unit 31, the region division image creation unit 32, the corresponding probability calculation unit 33, and the evaluation image creation unit 34 are functions realized by the processor 111 executing a computer program P.

[0025] The machine learning model 300 takes an image as an input variable and outputs output data Do, which contains output values ​​indicating the probability that each pixel of the input image belongs to one of several classification classes. For example, the machine learning model 300 takes a landscape photograph as an input image and outputs output values ​​indicating the probability that each pixel of the input image belongs to one of five classification classes: road class, plant class, building class, sky class, and water class.

[0026] In this embodiment, there are no restrictions on the range of output values ​​output by the machine learning model 300; they can be positive, zero, or negative. A larger output value indicates a higher probability of belonging to that classification class. Therefore, in conventional segmentation AI models, for a given pixel, the classification class with the largest output value among all classification classes was output as the classification result.

[0027] The region segmentation execution unit 31 reads the target image Di from the multiple images stored in the image storage unit 20 according to the input command from the input unit 13 and inputs it to the machine learning model 300. Then, it passes the output data Do output from the machine learning model 300 to the region segmentation image creation unit 32 and the corresponding probability calculation unit 33.

[0028] The region segmentation image creation unit 32 creates a region segmentation image Ds by dividing each pixel of the target image Di into regions belonging to one of several classification classes, based on the output of the machine learning model 300. Specifically, the region segmentation image creation unit 32 compares each pixel of the target image Di with the output values ​​of several classification classes included in the output data Do and classifies it to the classification class with the largest output value. Then, it creates a region segmentation image Ds that is the same size as the target image Di and assigns a color set for each classification class to each pixel.

[0029] The probability calculation unit 33 calculates the probability of each pixel of the target image Di belonging to each of the multiple classification classes based on the output data Do output by the machine learning model 300. Then, it passes the probability data Dp, which includes the probability of each pixel of the target image Di and all classification classes, to the evaluation image creation unit 34.

[0030] The evaluation image creation unit 34 creates evaluation images Dv corresponding to the target image Di for each of the classification classes, either partially or entirely. The evaluation image Dv is an image of the same size as the target image Di, and each pixel is evaluated in a different display manner according to the probability of it belonging to the classification class.

[0031] The image processing unit 30 transmits the processed image created by the image processing unit 30 to the display unit 12 in accordance with the command from the input unit 13. As a result, the display unit 12 displays the processed image created by the image processing unit 30. The processed image created by the image processing unit 30 includes the region segmentation image Ds created by the region segmentation image creation unit 32, one or more evaluation images Dv created by the evaluation image creation unit 34, and an integrated evaluation image Dt if created.

[0032] <1-2. Image Processing Flow> Next, the region segmentation process in the image processing device 1 will be explained with reference to Figure 3. Figure 3 is a flowchart showing the flow of the region segmentation process in the image processing device 1.

[0033] As shown in Figure 3, first, in accordance with the command from the input unit 13, the image processing unit 30 reads the target image Di to be subjected to region division processing from the image storage unit 20 (Step S101: Target image acquisition step).

[0034] Next, the region segmentation execution unit 31 inputs the target image Di to be processed into the machine learning model 300 and obtains output data Do from the machine learning model 300 (step S102: probability acquisition step). The output data Do includes output values ​​indicating the probability that each pixel of the target image Di belongs to one of several classification classes. That is, for each of the multiple classification classes, the region segmentation execution unit 31 obtains an output value indicating the probability that each pixel of the input target image Di belongs to that classification class. The region segmentation execution unit 31 then passes the output data Do to the region segmentation image creation unit 32 and the corresponding probability calculation unit 33.

[0035] The region segmentation image creation unit 32 creates a region segmentation image Ds by dividing each pixel of the target image Di into regions belonging to one of several classification classes based on the output of the machine learning model 300 (Step S103: Region Segmentation Image Acquisition Step). Specifically, the region segmentation image creation unit 32 compares the output values ​​of several classification classes included in the output data Do for each pixel of the target image Di and classifies the pixel to the classification class with the largest output value. Then, it creates a region segmentation image Ds of the same size as the target image Di, with each pixel assigned a color set for each classification class.

[0036] Meanwhile, the probability calculation unit 33 calculates the probability of each pixel of the target image Di belonging to each of the multiple classification classes based on the output data Do output by the machine learning model 300 (step S104: probability calculation step). At this time, as mentioned above, there is no limit to the range of the output value indicating probability output by the machine learning model 300, and it can take any value, such as positive, zero, or negative. The probability of belonging calculated by the probability calculation unit 33 is a normalized version of this output value. That is, for each pixel, the sum of the probabilities of belonging to all classification classes is 1. The probability calculation unit 33 then passes the probability data Dp, which includes the probabilities of belonging to each pixel of the target image Di and all classification classes, to the evaluation image creation unit 34.

[0037] In this embodiment, the probability calculation unit 33 calculates the probability of each pixel of the target image Di belonging to one of several classification classes by applying a softmax function in the direction of the classification class to the output value indicating the probability of belonging to one of the classification classes. For example, if the number of classification classes is n, and the output value of the i-th classification class is x ｉ The probability of the i-th classification class being represented is f ｉ If we let (x), then the probability f is given by the following formula. ｉ Calculate (x).

[0038]

[0039] Next, the evaluation image creation unit 34 creates evaluation images Dv corresponding to the target image Di for each of the classification classes, either partially or completely (Step S105: Evaluation Image Creation Process). The evaluation image Dv is an image of the same size as the target image Di, and each pixel is evaluated in a different display manner according to the probability of it belonging to the classification class.

[0040] The evaluation image Dv is, for example, a heatmap whose color gradually changes according to the magnitude of the probability of matching. Alternatively, the evaluation image Dv is, for example, a monochrome gradient image whose color becomes darker as the probability of matching increases. In this way, the probability of matching for each pixel in each classification class can be visualized.

[0041] Incidentally, the evaluation image creation unit 34 may further create an integrated evaluation image Dt by overlapping two or three monochromatic gradient images corresponding to two or three classification classes. Specifically, for example, evaluation images Dv that are monochromatic gradient images of the first classification class, the second classification class, and the third classification class are created in each color of RGB or CMK. Then, these three evaluation images Dv are overlapped. As a result, for example, if the corresponding probabilities of the R color classification class, G color classification class, and B color classification class are 1:0:0, it is red, if 0.5:0.5:0, it is yellow, if 0.5:0:0.5, it is cyan, and if 0.33:0.33:0.34, it is gray. By doing so, the corresponding probabilities of multiple classification classes in each pixel can be visualized simultaneously according to the color tone of each pixel.

[0042] Then, according to a command from the input unit 13, the image processing unit 30 transmits a part or all of the region division image Ds, which is a processed image created by the image processing unit 30, one or more evaluation images Dv, and the integrated evaluation image Dt if created, to the display unit 12 and causes the display unit 12 to display these images (step S106: display step).

[0043] In step S106, for example, the evaluation image Dv and the region division image Ds are displayed on the display unit 12. Then, when one of the regions of the color indicating a certain classification class shown in the region division image Ds is clicked by an input from the input unit 13, the evaluation image Dv of the said classification class may be displayed.

[0044] By obtaining the evaluation image Dv in such an image division process, in the region division task of an image using a machine learning model, the magnitude of the reaction to the selected classification class can be visualized.

[0045] Note that in the above image segmentation process, the region-divided image acquisition step of step S103 is performed before the evaluation image creation step of step S104 and the evaluation image creation step of step S105. However, the present invention is not limited to this. The region-divided image acquisition step of step S103 may be performed after the evaluation image creation step of step S104 and the evaluation image creation step of step S105, or the region-divided image acquisition step of step S103 may be performed in parallel with the evaluation image creation step of step S104 and the evaluation image creation step of step S105.

[0046] <1-3. Example> Hereinafter, examples of a region-divided image Ds and an evaluation image Dv obtained by a region division process using the image processing apparatus 1 according to the first embodiment for a certain target image Di will be shown. FIG. 4 is an image obtained by converting the target image Di into grayscale. FIG. 5 is an image obtained by converting the region-divided image Ds into grayscale. FIG. 6 is an image obtained by converting the first evaluation image Dv into grayscale. FIG. 7 is an image obtained by converting the second evaluation image Dv into grayscale.

[0047] The target image Di shown in FIG. 4 is an image in which a background with a water surface in light blue to blue and two black-and-white penguins are photographed. The image processing apparatus 1 according to the first embodiment performed a region division process on this target image Di. As a result, a region-divided image Ds shown in FIG. 5, a first evaluation image Dv which is a heat map showing the probability of correspondence of the classification class "penguin" shown in FIG. 6, and a second evaluation image Dv which is a heat map showing the probability of correspondence of the classification class "background" shown in FIG. 7 were obtained.

[0048] In the original color images shown in Figures 6 and 7, the colors are displayed in the following order from areas with a high probability of occurrence to areas with a low probability of occurrence: dark red, light red, orange, yellow, yellow-green, light blue, light blue, navy blue, and dark navy blue. Therefore, in Figures 6 and 7, dark red and dark navy blue are displayed as similarly blackish, while light blue and yellow are displayed as particularly whitish. Consequently, in the grayscale versions of Figures 6 and 7, areas with a high probability of occurrence and areas with a low probability of occurrence are displayed in similar color tones. Therefore, the color tones of some of the points circled in Figures 6 and 7 are described below.

[0049] In the segmented image Ds shown in Figure 5, the "Penguin" class is shown in light gray, and the "Background" class is shown in black. In the segmented image Ds, each pixel is assigned to the segmentation class with the highest calculated probability of matching. At first glance, in the segmented image Ds shown in Figure 5, it appears that the machine learning model 300 does not understand the boundary between the "Penguin" class region and the "Background" class region at all. However, when compared with the true boundary between classes that can be recognized from the target image Di in Figure 4, the shape closer to the true boundary can be seen from the two evaluation images Dv shown in Figures 6 and 7 than from the segmented image in Figure 5.

[0050] In the two evaluation images Dv shown in Figures 6 and 7, the true boundary is not well recognized near the beak and wings of the penguin on the left. However, in other areas, the true boundary in Figure 5 is located slightly inside the light blue area that appears whitish in Figure 6, and slightly inside the yellow area that appears whitish in Figure 7. From this, it is thought that by performing additional training on the machine learning model 300 that output probability in this embodiment, it may be possible to perform highly accurate class classification.

[0051] On the other hand, if the boundaries between classes in the segmented image Ds appear to be unrecognizable compared to the target image Di, and similarly, the evaluation image Dv, such as a heatmap, does not allow for the recognition of a shape close to the true boundary, then it is highly likely that even with additional training, it will be difficult to perform accurate class classification.

[0052] In this way, by obtaining an evaluation image Dv during the image segmentation process, it becomes possible to visualize the magnitude of the response to the selected classification class in an image segmentation task using a machine learning model. This then makes it possible to judge the superiority or inferiority and characteristics of the machine learning model.

[0053] <2. Modifications> Although embodiments have been described above, the present invention is not limited to those described above, and various modifications are possible.

[0054] In the above embodiment, the display unit 12 and the input unit 13 of the image processing device 1 were configured separately, but the present invention is not limited thereto. The display unit 12 and the input unit 13 may be touch panels.

[0055] Although this invention has been described in detail, the above description is illustrative in all respects and does not limit the invention. It is understood that countless variations not illustrated can be envisioned without falling outside the scope of this invention. The components described in each of the above embodiments and variations can be combined or omitted as appropriate, as long as they do not contradict each other.

[0056] 1: Image processing device 11: Computer main unit 12: Display unit 13: Input unit 20: Image storage unit 30: Image processing unit 31: Region division execution unit 32: Region division image creation unit 33: Probability calculation unit 34: Evaluation image creation unit 300: Machine learning model Di: Target image Do: Output data Dp: Probability data Ds: Region division image Dt: Integrated evaluation image Dv: Evaluation image M: Storage medium P: Computer program

Claims

1. An image processing apparatus comprising: a storage unit for storing a target image to be processed; an image processing unit for performing image processing on the target image; and a display unit for displaying the processed image created by the image processing unit, wherein the image processing unit includes: a machine learning model that outputs an output value indicating the probability that each pixel of the input target image belongs to a classification class for each of a plurality of classification classes; a region segmentation image creation unit that creates a region segmentation image by dividing each pixel of the target image into regions belonging to any of the plurality of classification classes based on the output of the machine learning model; a probability calculation unit that calculates the probability that each pixel of the target image belongs to each of the plurality of classification classes based on the output of the machine learning model; and an evaluation image creation unit that creates an evaluation image displayed in a different display manner according to the probability for each of the classification classes, either partially or entirely.

2. An image processing apparatus according to claim 1, wherein the evaluation image is a heat map whose color gradually changes according to the magnitude of the corresponding probability.

3. An image processing apparatus according to claim 1, wherein the evaluation image is a monochromatic gradient image in which the color becomes darker as the probability of the corresponding value increases.

4. An image processing apparatus according to claim 3, wherein the evaluation image creation unit further creates an integrated evaluation image by superimposing two or three monochrome gradient images corresponding to two or three classification classes.

5. An image processing apparatus according to any one of claims 1 to 4, wherein the corresponding probability calculation unit calculates the corresponding probability by applying a softmax function to the output value indicating the probability of each pixel of the target image belonging to one of the plurality of classification classes.

6. A computer program for performing image processing, which causes a computer to perform the following steps: A) input a target image to be processed into a machine learning model and obtain an output value indicating the probability that each pixel of the input target image belongs to the classification class for each of a plurality of classification classes; B) create a region segmented image by dividing each pixel of the target image into regions belonging to one of the plurality of classification classes based on the output of the machine learning model; C) calculate the probability that each pixel of the target image belongs to each of the plurality of classification classes output by the machine learning model; and D) create an evaluation image displayed in a different display manner according to the probability for each of the classification classes, either partially or entirely.

Images