Image processing device and computer program

Abstract

This technology provides a way to visualize the magnitude of the response to a selected classification class in an image segmentation task using a machine learning model. [Solution] The image processing unit 30 of this image processing device includes a machine learning model 300 that outputs an output value indicating the probability that each pixel of the target image Di belongs to a classification class for each of a plurality of classification classes; a region segmentation image creation unit 32 that creates a region segmentation image by dividing each pixel of the target image Di into regions belonging to one of the plurality of classification classes based on the output Do of the machine learning model 300; a probability calculation unit 33 that calculates the probability that each pixel of the target image Di belongs to each of the plurality of classification classes based on the output Do of the machine learning model 300; and an evaluation image creation unit 34 that creates an evaluation image Dv that is displayed in a different display manner according to the probability for each classification class.

Description

Technical Field

[0001] The present invention relates to a technique for determining a classification class to which each pixel of an image belongs using a machine learning model and dividing the image into regions.

Background Art

[0002] Conventionally, machine learning models for performing region segmentation of images, such as semantic segmentation (region segmentation), instance segmentation (individual object region segmentation), panoptic segmentation (whole image region segmentation), etc., are known. A conventional machine learning model for performing region segmentation of images is described in, for example, Patent Document 1.

[0003] In other machine learning models for image processing, for example, Grad-CAM for performing image classification can visualize the reaction positions of a deep learning model with a heat map or the like. By knowing the reaction positions in the image, the machine learning model has interpretability, which is very effective in determining the superiority, inferiority, and characteristics of the machine learning model.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, in a machine learning model for performing region segmentation of an image, a general means for converting the reaction positions of a deep learning model into a heat map has not been established. For this reason, the machine learning model has no interpretability, and it is difficult to determine the superiority, inferiority, and characteristics of the machine learning model.

[0006] This invention has been made in view of these circumstances, and aims to provide a technique for visualizing the magnitude of the response to a selected classification class in an image segmentation task using a machine learning model. [Means for solving the problem]

[0007] To solve the above problems, the first invention of the present application is an image processing apparatus comprising: a storage unit for storing a target image to be processed; an image processing unit for performing image processing on the target image; and a display unit for displaying the processed image created by the image processing unit, wherein the image processing unit includes: a machine learning model that outputs an output value indicating the probability that each pixel of the input target image belongs to a classification class for each of a plurality of classification classes; a region segmentation image creation unit that creates a region segmentation image by dividing each pixel of the target image into regions belonging to any of the plurality of classification classes based on the output of the machine learning model; a probability calculation unit that calculates the probability that each pixel of the target image belongs to each of the plurality of classification classes based on the output of the machine learning model; and an evaluation image creation unit that creates an evaluation image displayed in a different display manner according to the probability for each of the classification classes, either partially or entirely.

[0008] The second invention of this application is an image processing apparatus of the first invention, wherein the evaluation image is a heat map in which the color gradually changes according to the magnitude of the corresponding probability.

[0009] The third invention of this application is an image processing apparatus of the first invention, wherein the evaluation image is a monochromatic gradient image in which the color becomes darker as the probability of the corresponding value increases.

[0010] The fourth invention of this application is an image processing apparatus of the third invention, wherein the evaluation image creation unit further creates an integrated evaluation image by superimposing two or three of the monochrome gradient images corresponding to two or three of the classification classes.

[0011] The fifth invention of this application is an image processing apparatus according to any one of the first to fourth inventions, wherein the corresponding probability calculation unit calculates the corresponding probability by applying a softmax function to the output value indicating the probability of each pixel of the target image belonging to one of the plurality of classification classes.

[0012] The sixth invention of this application is a computer program for image processing, which causes the computer to perform the following steps: A) input a target image to be processed into a machine learning model and obtain an output value as output for each of a plurality of classification classes that indicates the probability that each pixel of the input target image belongs to the classification class; B) create a region segmented image by dividing each pixel of the target image into regions belonging to one of the plurality of classification classes based on the output of the machine learning model; C) calculate the probability that each pixel of the target image belongs to each of the plurality of classification classes output by the machine learning model; and D) create an evaluation image that displays some or all of the classification classes in different display modes according to the probability of matching. [Effects of the Invention]

[0013] According to the first to sixth inventions of this application, the magnitude of the response to a selected classification class can be visualized in an image region segmentation task using a machine learning model.

[0014] In particular, according to the fourth invention of this application, the probability of each pixel belonging to multiple classification classes can be visualized simultaneously in a single image. [Brief explanation of the drawing]

[0015] [Figure 1] This is a schematic diagram of the image processing device according to the first embodiment. [Figure 2] This is a functional block diagram of the image processing apparatus according to the first embodiment. [Figure 3] This is a flowchart showing the flow of region segmentation processing in the image processing apparatus according to the first embodiment. [Figure 4]It is an image obtained by converting an example of the target image into grayscale. [Figure 5] It is an image obtained by converting an example of the region-divided image into grayscale. [Figure 6] It is an image obtained by converting an example of the first evaluation image into grayscale. [Figure 7] It is an image obtained by converting an example of the second evaluation image into grayscale.

Embodiments for Carrying Out the Invention

[0016] Hereinafter, preferred embodiments of the present invention will be described with reference to the drawings.

[0017] <1. First Embodiment> <1-1. Configuration of the Image Processing Apparatus><U+ FIG. 1 is a schematic diagram of an image processing apparatus 1 according to the first embodiment. FIG. 2 is a functional block diagram of the image processing apparatus 1. This image processing apparatus 1 performs a region division process of determining a classification class to which each pixel of an image belongs and dividing the image into regions using a machine learning model.

[0018] As shown in FIG. 1, this image processing apparatus 1 includes a computer main body 11, a display unit 12, and an input unit 13. The computer main body 11 includes a processor 111 such as a CPU, a memory 112 such as a RAM, and a storage unit 113 such as a hard disk drive. The display unit 12 displays an image output from the computer main body 11. The display unit 12 is, for example, a liquid crystal display such as a PC monitor. The input unit 13 can input commands to the computer main body 11. The input unit 13 is, for example, a keyboard and a mouse.

[0019] Memory 112 and storage unit 113 are connected to the processor 111 via bus wiring (not shown). Storage unit 113 stores the computer program P. The processor 111 loads the computer program P stored in storage unit 113 into memory 112 and executes the code contained in the computer program P sequentially. As a result, the image processing device 1 performs region division processing on the target image Di to be processed and classifies each pixel of the target image Di into one of several classes.

[0020] Computer program P is application software that causes the computer main unit 11 to execute various processes related to area partitioning. Computer program P is read from a storage medium M such as a CD or DVD and installed on the computer main unit 11. However, computer program P may also be downloaded to the computer main unit 11 via a network N such as the Internet.

[0021] In a typical segmentation AI model, a target image Di is classified into multiple classes, and segmentation images such as mask images are output, each colored according to its respective class. This image processing device 1 not only produces segmentation images Ds obtained by segmenting the target image Di into multiple classification classes, but also creates evaluation images Dv for each classification class, displayed in different ways according to the probability of matching. These evaluation images Dv allow the probability of matching each classification class to be visually confirmed.

[0022] Figure 2 is a functional block diagram of the image processing device 1. The computer body 11 includes an image storage unit 20 and an image processing unit 30.

[0023] The image storage unit 20 is a storage unit provided within the computer main body 11. The image storage unit 20 may use the storage unit 113, or it may be a storage unit provided separately from the storage unit 113. The image storage unit 20 stores the target image Di to be processed.

[0024] The image processing unit 30 performs image processing on the target image Di. The image processing unit 30 includes a machine learning model 300, a region segmentation execution unit 31, a region segmentation image creation unit 32, a corresponding probability calculation unit 33, and an evaluation image creation unit 34. The region segmentation execution unit 31, the region segmentation image creation unit 32, the corresponding probability calculation unit 33, and the evaluation image creation unit 34 are functions realized by the processor 111 executing a computer program P.

[0025] The machine learning model 300 takes an image as its input variable and outputs output data Do, which contains output values ​​indicating the probability that each pixel of the input image belongs to one of several classification classes. For example, the machine learning model 300 takes a landscape photograph as its input image and outputs output values ​​indicating the probability that each pixel of the input image belongs to one of five classification classes: road, plant, building, sky, and water.

[0026] In this embodiment, there are no restrictions on the range of output values ​​output by the machine learning model 300; they can be positive, zero, or negative. A larger output value indicates a higher probability of belonging to that classification class. Therefore, in conventional segmentation AI models, for a given pixel, the classification class with the largest output value among all classification classes was output as the classification result.

[0027] The region segmentation execution unit 31 reads the target image Di from multiple images stored in the image storage unit 20 according to the input command from the input unit 13 and inputs it to the machine learning model 300. Then, it passes the output data Do output from the machine learning model 300 to the region segmentation image creation unit 32 and the corresponding probability calculation unit 33.

[0028] The segmented image creation unit 32 creates segmented images Ds by dividing each pixel of the target image Di into regions belonging to one of several classification classes, based on the output of the machine learning model 300. Specifically, the segmented image creation unit 32 compares each pixel of the target image Di with the output values ​​of several classification classes included in the output data Do and classifies it to the classification class with the largest output value. Then, it creates segmented images Ds of the same size as the target image Di, with each pixel assigned a color set for each classification class.

[0029] The probability calculation unit 33 calculates the probability of each pixel of the target image Di belonging to each of several classification classes based on the output data Do output by the machine learning model 300. Then, it passes the probability data Dp, which includes the probability of each pixel of the target image Di and all classification classes, to the evaluation image creation unit 34.

[0030] The evaluation image creation unit 34 creates evaluation images Dv corresponding to the target image Di for each of the classification classes, either partially or completely. The evaluation image Dv is an image of the same size as the target image Di, and each pixel is evaluated in a different display manner according to the probability of it belonging to the classification class.

[0031] The image processing unit 30 transmits the processed image created by the image processing unit 30 to the display unit 12 in accordance with the command from the input unit 13. As a result, the display unit 12 displays the processed image created by the image processing unit 30. The processed image created by the image processing unit 30 includes the region segmentation image Ds created by the region segmentation image creation unit 32, one or more evaluation images Dv created by the evaluation image creation unit 34, and an integrated evaluation image Dt if created.

[0032] <1-2. Image Processing Flow> Next, the region segmentation process in the image processing device 1 will be explained with reference to Figure 3. Figure 3 is a flowchart showing the flow of the region segmentation process in the image processing device 1.

[0033] As shown in Figure 3, first, in accordance with the command from the input unit 13, the image processing unit 30 reads the target image Di to be subjected to region division processing from the image storage unit 20 (Step S101: Target image acquisition step).

[0034] Next, the region segmentation execution unit 31 inputs the target image Di to be processed into the machine learning model 300 and obtains output data Do from the machine learning model 300 (step S102: probability acquisition step). The output data Do contains output values ​​indicating the probability that each pixel of the target image Di corresponds to one of several classification classes. That is, for each of the multiple classification classes, the region segmentation execution unit 31 obtains output values ​​indicating the probability that each pixel of the input target image Di corresponds to that classification class. The region segmentation execution unit 31 then passes the output data Do to the region segmentation image creation unit 32 and the corresponding probability calculation unit 33.

[0035] The region segmentation image creation unit 32 creates a region segmentation image Ds by dividing each pixel of the target image Di into regions belonging to one of several classification classes based on the output of the machine learning model 300 (Step S103: Region segmentation image acquisition step). Specifically, the region segmentation image creation unit 32 compares each pixel of the target image Di with the output values ​​of several classification classes included in the output data Do and classifies it to the classification class with the largest output value. Then, it creates a region segmentation image Ds that is the same size as the target image Di and assigns a color set for each classification class to each pixel.

[0036] Meanwhile, the probability calculation unit 33 calculates the probability of each pixel of the target image Di belonging to each of the multiple classification classes based on the output data Do output by the machine learning model 300 (Step S104: Probability Calculation Process). At this time, as mentioned above, there is no limit to the range of the output value indicating probability output by the machine learning model 300, and it can take any value, such as positive, zero, or negative. The probability calculated by the probability calculation unit 33 is a normalized version of this output value. That is, for each pixel, the sum of the probabilities of all classification classes is 1. The probability calculation unit 33 then passes the probability data Dp, which includes the probabilities of each pixel of the target image Di and all classification classes, to the evaluation image creation unit 34.

[0037] In this embodiment, the probability calculation unit 33 calculates the probability of each pixel of the target image Di belonging to one of several classification classes by applying a softmax function in the direction of the classification class to the output value indicating the probability of belonging to one of the classification classes. For example, if the number of classification classes is n, and the output value of the i-th classification class is x i The probability of the i-th classification class being applicable is f i Let (x) be the case, then the probability f is given by the following formula. i Calculate (x).

[0038]

number

[0039] Next, the evaluation image creation unit 34 creates evaluation images Dv corresponding to the target image Di for each of the classification classes, either partially or completely (Step S105: Evaluation Image Creation Process). The evaluation image Dv is an image of the same size as the target image Di, and each pixel is evaluated in a different display manner according to the probability of it belonging to the classification class.

[0040] The evaluation image Dv is, for example, a heatmap whose color gradually changes according to the magnitude of the probability of matching. Alternatively, the evaluation image Dv is, for example, a monochrome gradient image whose color becomes darker as the probability of matching increases. In this way, the probability of matching for each pixel in each classification class can be visualized.

[0041] The evaluation image creation unit 34 may further create an integrated evaluation image Dt by superimposing two or three monochrome gradient images corresponding to two or three classification classes. Specifically, for example, evaluation images Dv, which are monochrome gradient images of the first, second, and third classification classes, are created in RGB or CMK colors, respectively. Then, these three evaluation images Dv are superimposed. As a result, for example, if the probability of the R-colored classification class, G-colored classification class, and B-colored classification class is 1:0:0, it will be red; if it is 0.5:0.5:0, it will be yellow; if it is 0.5:0:0.5, it will be cyan; and if it is 0.33:0.33:0.34, it will be gray. In this way, the probability of multiple classification classes at each pixel can be simultaneously visualized by the color tone at each pixel.

[0042] Subsequently, the image processing unit 30 transmits to the display unit 12, in accordance with the command from the input unit 13, some or all of the processed images created by the image processing unit 30, namely the region segmentation image Ds, one or more evaluation images Dv, and, if created, the integrated evaluation image Dt, to the display unit 12, causing the display unit 12 to display these images (step S106: display step).

[0043] In step S106, for example, the display unit 12 displays the evaluation image Dv and the region segmentation image Ds. Then, based on input from the input unit 13, clicking on one of the colored regions in the region segmentation image Ds that represents a certain classification class may display the evaluation image Dv for that classification class.

[0044] By obtaining an evaluation image Dv in this image segmentation process, it becomes possible to visualize the magnitude of the response to the selected classification class in an image segmentation task using a machine learning model.

[0045] In the image segmentation process described above, the region segmentation image acquisition step in step S103 is performed before the evaluation image creation step in step S104 and the evaluation image creation step in step S105, but the present invention is not limited thereto. The region segmentation image acquisition step in step S103 may be performed after the evaluation image creation step in step S104 and the evaluation image creation step in step S105, or the region segmentation image acquisition step in step S103 may be performed in parallel with the evaluation image creation step in step S104 and the evaluation image creation step in step S105.

[0046] <1-3. Examples> Below, we show examples of region segmentation images Ds and evaluation images Dv obtained by region segmentation processing using the image processing apparatus 1 according to the first embodiment for a given target image Di. Figure 4 is an image obtained by converting the target image Di to grayscale. Figure 5 is an image obtained by converting the region segmentation image Ds to grayscale. Figure 6 is an image obtained by converting the first evaluation image Dv to grayscale. Figure 7 is an image obtained by converting the second evaluation image Dv to grayscale.

[0047] The target image Di shown in Figure 4 is an image in which a background of light blue to blue water and two black and white penguins are photographed. The image processing device 1 according to the first embodiment performed region segmentation processing on this target image Di. As a result, the region segmentation image Ds shown in Figure 5, the first evaluation image Dv which is a heatmap showing the probability of belonging to the classification class "penguin" shown in Figure 6, and the second evaluation image Dv which is a heatmap showing the probability of belonging to the classification class "background" shown in Figure 7 were obtained.

[0048] In the original color images shown in Figures 6 and 7, the colors are displayed in the following order from areas with a high probability of occurrence to areas with a low probability of occurrence: dark red, light red, orange, yellow, yellow-green, light blue, light blue, navy blue, and dark blue. Therefore, in Figures 6 and 7, dark red and dark blue appear similarly black, while light blue and yellow appear particularly whitish. Consequently, in the grayscale versions of Figures 6 and 7, areas with a high probability of occurrence and areas with a low probability of occurrence appear in similar color tones. Therefore, the color tones of some of the points circled in Figures 6 and 7 are described below.

[0049] In the segmented image Ds shown in Figure 5, the "Penguin" class is shown in light gray, and the "Background" class is shown in black. In segmented image Ds, each pixel is assigned to the segmentation class with the highest calculated probability of being included. At first glance, in the segmented image Ds shown in Figure 5, it appears that the machine learning model 300 does not understand the boundary between the "Penguin" class region and the "Background" class region at all. However, when compared to the true boundary between classes that can be recognized from the target image Di in Figure 4, the two evaluation images Dv shown in Figures 6 and 7 show a shape closer to the true boundary than the segmented image in Figure 5.

[0050] In the two evaluation images Dv shown in Figures 6 and 7, the true boundary is not well recognized near the beak and feathers of the penguin on the left. However, in other areas, the true boundary in Figure 5 is located slightly inside the light blue area that appears whitish in Figure 6, and slightly inside the yellow area that appears whitish in Figure 7. From this, it is thought that by performing additional training on the machine learning model 300 that output probability in this embodiment, it may be possible to perform highly accurate class classification.

[0051] On the other hand, if the boundaries between classes in the segmented image Ds appear to be unrecognizable compared to the target image Di, and similarly, the evaluation image Dv, such as a heatmap, does not allow for the recognition of shapes close to the true boundaries, then it is highly likely that even with additional training, it will be difficult to perform accurate class classification.

[0052] In this way, by obtaining an evaluation image Dv during the image segmentation process, it becomes possible to visualize the magnitude of the response to the selected classification class in an image segmentation task using a machine learning model. This then makes it possible to judge the superiority or inferiority and characteristics of the machine learning model.

[0053] <2. Variant> Although embodiments have been described above, the present invention is not limited to those described above, and various modifications are possible.

[0054] In the above embodiment, the display unit 12 and the input unit 13 of the image processing device 1 were configured separately, but the present invention is not limited thereto. The display unit 12 and the input unit 13 may be touch panels.

[0055] Although this invention has been described in detail, the above description is illustrative in all respects and does not limit the invention. It is understood that countless variations not illustrated can be envisioned without falling outside the scope of this invention. The components described in each of the above embodiments and variations can be combined or omitted as appropriate, as long as they do not contradict each other. [Explanation of symbols]

[0056] 1: Image processing device 11: Computer main unit 12:Display section 13: Input section 20: Image storage unit 30: Image Processing Unit 31: Area partitioning execution unit 32: Region segmentation image creation unit 33: Probability calculation unit 34: Evaluation Image Creation Department 300: Machine Learning Models Di: Target image Do: Output data Dp: Probability data Ds: Region segmentation image Dt: Integrated evaluation image Dv: Evaluation image M:Storage medium P: Computer program

Claims

1. An image processing device, A storage unit that stores the target image to be processed, An image processing unit that performs image processing on the aforementioned target image, A display unit that displays the processed image created by the image processing unit, It has, The aforementioned image processing unit, A machine learning model that outputs an output value for each of the multiple classification classes, indicating the probability that each pixel of the input target image belongs to the classification class. A region segmentation image creation unit creates a region segmentation image by dividing each pixel of the target image into regions belonging to one of the multiple classification classes based on the output of the machine learning model. A probability calculation unit calculates the probability that each pixel of the target image belongs to one of the multiple classification classes based on the output of the machine learning model, An evaluation image creation unit that creates evaluation images for each of the classification classes, some or all of the classification classes, displayed in different display modes according to the probability of applicability; Image processing device, including

2. An image processing apparatus according to claim 1, The aforementioned evaluation image is a heatmap in which the color gradually changes according to the magnitude of the corresponding probability, in an image processing device.

3. An image processing apparatus according to claim 1, The aforementioned evaluation image is a monochrome gradient image in which the color becomes darker as the probability of the occurrence increases, in the image processing apparatus.

4. An image processing apparatus according to claim 3, The aforementioned evaluation image creation unit further creates an integrated evaluation image by superimposing two or three of the aforementioned monochrome gradient images corresponding to two or three of the aforementioned classification classes, which is an image processing device.

5. An image processing apparatus according to any one of claims 1 to 4, The aforementioned probability calculation unit calculates the probability by applying a softmax function to the output value indicating the probability of each pixel of the target image belonging to one of the multiple classification classes.

6. A computer program for image processing, which is used by a computer. A) A step of inputting the target image to be processed into a machine learning model and obtaining an output value as the output for each of the multiple classification classes, which indicates the probability that each pixel of the input target image belongs to the classification class. B) A step of creating a segmented image by dividing each pixel of the target image into regions belonging to one of the multiple classification classes based on the output of the machine learning model, C) A step of calculating the probability that each pixel of the target image corresponds to one of the multiple classification classes output by the machine learning model, D) A step of creating evaluation images for each of the classification classes, some or all of which are displayed in different display modes according to the probability of applicability; A computer program designed to execute something.

Images