Information processing device and image processing method

Abstract

To provide an information processing apparatus capable of indicating the basis of classification of images classified by image recognition using a model learned by machine learning, for the entire images.SOLUTION: An information processing apparatus 100 includes an analysis target acquisition unit 101, an image processing unit 102, an inference unit 103, an inference result extraction unit 104, and a basis generation unit 105. The image processing unit 102 generates a plurality of masked images by masking each of the images using a plurality of masks. The inference result extraction unit 104 extracts an inference result at target coordinates designated in the image from an inference result of each masked image. Based on the inference result at the target coordinates extracted by the inference result extraction unit 104 and the plurality of masks, the basis generation unit 105 generates a basis map visualizing the determination basis for a classification result of the image by a model.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an information processing device and an image processing method. [Background technology]

[0002] In recent years, information processing devices that perform image processing such as image recognition using machine learning have been widely used. In information processing devices using machine learning, in addition to improving the accuracy of recognition, it is required to improve the reliability of the recognition.

[0003] Regarding the improvement of the reliability of image recognition by machine learning, for example, the technology of Patent Document 1 is known. Patent Document 1 discloses an information processing device having a trained first neural network and a second neural network in which an initial value is set for a weight parameter, generating a mask based on the second neural network, and updating either the first neural network or the second neural network based on an evaluation result of an inference value based on the first neural network and composite data obtained by combining input data and the mask. This improves the explainability of the neural network while suppressing deterioration of the accuracy of the output by the neural network. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 6801751 Summary of the Invention [Problem to be solved by the invention]

[0005] In Patent Document 1, it is explained that since the first neural network is a model that performs inference from outside the mask region, by visualizing the attention region of this model, it is possible to grasp the attention region as the region used for inference. In other words, by applying the technology of Patent Document 1, it is possible to show which part of the input image was used to classify the image by the neural network.

[0006] However, while the technology of Patent Document 1 makes it possible to visualize the region of interest of an inference model, it is not possible to show the basis for image classification for the entire image.

[0007] The present invention has been made in consideration of the above-mentioned problems, and its main objective is to provide an information processing device and an image processing method that are capable of showing the basis for classification of an entire image that has been classified by image recognition using a model trained by machine learning. [Means for solving the problem]

[0008] The information processing device according to the present invention comprises an analysis target acquisition unit that acquires an image to be analyzed, an image processing unit that sets a plurality of masks for the image and generates a plurality of masked images by masking each of the image using the plurality of masks, an inference unit that performs inference on each of the plurality of masked images using a model trained by machine learning and acquires an inference result regarding the classification of the image for each of the plurality of masked images, an inference result extraction unit that extracts an inference result at a target coordinate specified in the image from the inference result of each masked image acquired by the inference unit, and a basis generation unit that generates a basis map that visualizes the basis for the judgment on the classification result of the image by the model, based on the inference result at the target coordinate extracted by the inference result extraction unit and the plurality of masks. An image processing method according to the present invention uses an information processing device, and includes the steps of: acquiring an image to be analyzed; setting a plurality of masks for the image; masking each of the image using the plurality of masks to generate a plurality of masked images; performing inference on each of the plurality of masked images using a trained model through machine learning to acquire an inference result regarding the classification of the image for each of the plurality of masked images; extracting an inference result at a target coordinate specified within the image from the inference result for each acquired masked image; and generating a basis map that visualizes the basis for the judgment of the classification result of the image by the model based on the inference result at the extracted target coordinates and the plurality of masks. Effect of the Invention

[0009] According to the present invention, it is possible to provide an information processing device and an image processing method that are capable of showing the basis for classification of an entire image that has been classified by image recognition using a model that has been trained by machine learning. [Brief description of the drawings]

[0010] [Figure 1] 1 is a block diagram showing an example of the configuration of an information processing device according to a first embodiment of the present invention. [Diagram 2] 4 is a flowchart showing an example of processing contents of the information processing device according to the first embodiment of the present invention. [Diagram 3] FIG. 13 is a diagram illustrating an example of mask processing. [Figure 4] FIG. 13 is a diagram for explaining an example of extraction of an inference result. [Diagram 5] FIG. 13 is a diagram illustrating an example of generating a basis map. [Figure 6] 13 is a flowchart showing an example of processing contents of an information processing device according to a second embodiment of the present invention. [Figure 7] FIG. 13 is a diagram illustrating an example of generating a basis map. [Figure 8] 13 is a flowchart showing an example of processing contents of an information processing device according to a third embodiment of the present invention. [Figure 9] FIG. 13 is a block diagram showing an example of the configuration of an information processing device according to a fourth embodiment of the present invention. [Figure 10] 13 is a flowchart showing an example of processing contents of an information processing device according to a fourth embodiment of the present invention. [Figure 11] FIG. 13 is a diagram showing an example of an image from which a learning image is generated. [Figure 12] FIG. 13 is a diagram illustrating an example of determining a template region. [Figure 13] FIG. 11 is a diagram illustrating an example of learning image generation. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. The following description and drawings are examples for explaining the present invention, and appropriate omissions and simplifications have been made for clarity of explanation. The present invention can be implemented in various other forms. Unless otherwise specified, each component may be singular or plural.

[0012] An example of the information processing device of the present invention described in each of the following embodiments is used for supporting the learning of an analysis device to which machine learning is applied. Machine learning includes learning a neural network using learning data (teacher data). Such an information processing device can be configured using a general computer such as a PC (personal computer) or a server. That is, like a general PC or server, the information processing device according to the present invention includes an arithmetic processing device configured using a CPU, a ROM, a RAM, etc., a storage device configured using a HDD (Hard Disk Drive) or an SSD (Solid State Drive), etc., and various peripheral devices. The program executed by this information processing device is assumed to be pre-installed in the storage device. In the following description, these components that an information processing device naturally includes are not illustrated, and the description focuses on the functions realized by the information processing device of each embodiment.

[0013] Specifically, the functions of the information processing device of each embodiment are realized by a program stored in a storage device and executed by an arithmetic processing device. That is, the functions of calculation, control, etc. described in each embodiment are realized by software and hardware working together as a program stored in a storage device is executed by an arithmetic processing device. In the following description, a program executed by a computer or the like, its function, or a means for realizing the function may be referred to as a "function," "means," "part," "unit," "module," etc.

[0014] The information processing device of each embodiment may be configured as a single computer, or may be configured as multiple computers connected to each other via a network. The concept of the invention is equivalent and remains the same.

[0015] In addition, in the information processing device of each embodiment, the present invention is described with functions realized by software, but equivalent functions can also be realized by hardware such as FPGA (Field Programmable Gate Array) and ASIC (Application Specific Integrated Circuit). Also, various software and hardware may be combined to realize the functions. These aspects are also included in the scope of the present invention.

[0016] [First embodiment] Fig. 1 is a block diagram showing an example of the configuration of an information processing device 100 according to a first embodiment of the present invention. As shown in Fig. 1, the information processing device 100 according to this embodiment includes the following functional blocks: an analysis target acquisition unit 101, an image processing unit 102, an inference unit 103, an inference result extraction unit 104, a basis generation unit 105, an input interface 106, an output interface 107, and an external interface 108. These functional blocks are connected to each other via a bus 109. The bus 109 holds data, control information, analysis information, etc. handled by each functional block, and mediates information transmission between each functional block.

[0017] As mentioned at the beginning, each functional block in Fig. 1 is realized by software, hardware, or a combination of these. The information processing device 100 may include various hardware, interfaces, etc. that are normally included in a computer in addition to those shown in Fig. 1.

[0018] The information processing device 100 is connected to an input device 110, a display device 111, and an information appliance 112. The information processing device 100 may be connected to these devices by wire or wirelessly. Note that, although an example in which the input device 110 and the display device 111 are provided outside the information processing device 100 is shown in FIG. 1, these may be built into the information processing device 100.

[0019] The analysis target acquisition unit 101 acquires an image to be analyzed by the information processing device 100. This image may be, for example, an image selected by a user's input operation input from the input device 110 via the input interface 106 among images stored in a storage device (not shown), or may be an image input from an external information device 112 via the external interface 108. The analysis target acquisition unit 101 can acquire any image as long as it is an image that can be classified by a model trained by machine learning and is an image to be analyzed by an analysis device (not shown).

[0020] The image processing unit 102 generates a masked image by performing image processing using a mask on the image acquired by the analysis target acquisition unit 101. The image processing unit 102 can generate a plurality of masked images by setting a plurality of masks for one image and masking the image for each mask.

[0021] The inference unit 103 performs inference using a model learned by machine learning for each of the masked images generated by the image processing unit 102 from one image. As a result, it is possible to determine what object is in each of the masked images, and obtain the determination result as an inference result regarding the classification of the original unmasked image. The classification of the image obtained by the inference performed by the inference unit 103 is hereinafter referred to as a "class." That is, the inference unit 103 can obtain a class representing the classification of each object as an inference result regarding the unmasked image by determining the type of various objects in the masked image. When there are multiple types of objects in the masked image, or when there are background parts other than objects in the masked image, a class corresponding to each of the image regions corresponding to these is obtained for each image region as an inference result regarding the unmasked image.

[0022] The inference result extraction unit 104 extracts an inference result at a target coordinate specified in the original unmasked image from the inference results of each masked image acquired by the inference unit 103. Note that the target coordinate is specified by a user's input operation input from the input device 110 via the input interface 106, for example.

[0023] The evidence generating unit 105 generates an evidence map based on the inference result at the target coordinates extracted by the inference result extracting unit 104 and a plurality of masks set when the image processing unit 102 generates a masked image. This evidence map visualizes the judgment basis for the image classification result executed by an analysis device (not shown) using a trained model. A specific example of the evidence map generated by the evidence generating unit 105 will be described later.

[0024] The input interface 106 is connected to the input device 110 and accepts input operations performed by the user using the input device 110. The input device 110 is configured using, for example, a mouse, a keyboard, and the like. When the user inputs various instruction operations and selection operations to the information processing device 100 using the input device 110, the contents of the input operation are transmitted to each functional block in the information processing device 100 via the input interface 106. This allows each functional block to perform processing according to the user's input operation. For example, the analysis target acquisition unit 101 can acquire an image to be analyzed and target coordinates specified in the image based on the user's input operation performed via the input interface 106.

[0025] The output interface 107 is connected to the display device 111, and outputs various images and information to the display device 111 to display the contents on the display device 111. The display device 111 is configured using, for example, a liquid crystal display. The information processing device 100 can provide information to the user by displaying, for example, a basis map generated by the basis generating unit 105 on the display device 111 via the output interface 107. At this time, the output interface 107 may display the basis map as it is, or may display a screen in which the basis map is superimposed on the image to be analyzed.

[0026] The external interface 108 is connected to an external information device 112, and relays communication data transmitted and received between the information processing device 100 and the information device 112. The information device 112 corresponds to, for example, a PC or a server present in the same network as the information processing device 100, or a server present on the cloud. The information processing device 100 can acquire various information and data used in each functional block in the information processing device 100 by receiving communication data from the information device 112 via the external interface 108. For example, the analysis target acquisition unit 101 can acquire an image to be analyzed, target coordinates specified in the image, and the like from the information device 112 via the external interface 108.

[0027] Next, a description will be given of a method for generating a basis map in the information processing device 100 of this embodiment. Fig. 2 is a flowchart showing an example of the processing contents of the information processing device 100 according to the first embodiment of the present invention.

[0028] First, the analysis target acquisition unit 101 acquires an image to be analyzed, and also acquires target coordinates and target class in the target image (step S201). Here, for example, as described above, based on information input from the input device 110 or the external information device 112, the analysis target image and target coordinates are acquired, and the target class is acquired. The target class is a class designated as a target for generating a grounds map among the above-mentioned classes acquired for each image region of the masked image by the inference unit 103. Like the target coordinates, the target class can also be designated by a user's input operation input from the input device 110 via the input interface 106, or information input from the information device 112 via the external interface 108. When the target coordinates or the target class is designated by the user's input operation, for example, a graphical input operation in which the target image is displayed on the display device 111 and the user selects coordinates therein, or a character-based input operation may be used. In addition to the above, any input operation method can be adopted.

[0029] In the process of step S201, the target coordinates and the target class may be acquired based on the information of the target image and the inference result of the inference unit 103 for the target image. For example, when the contrast difference between an object and the background in the target image is small, the coordinates near the boundary may be acquired as the target coordinates. In addition, the inference unit 103 may perform inference on the target image in advance, and the inference result may be presented to the user to acquire the coordinates of a part determined to be incorrect, or the coordinates of a part that differs from the inference result obtained by another analysis method, as the target coordinates. Furthermore, the class of the image area corresponding to these target coordinates may be acquired as the target class, or the class corresponding to all image areas in the target image may be acquired as the target class. In addition to this, the target coordinates and the target class can be acquired by any method.

[0030] Next, the image processing unit 102 performs mask processing on the target image acquired in step S201 to generate a masked image (step S202). Here, for example, the target image is duplicated to generate multiple copy images, and a different mask is set for each copy image, and multiple masked images are generated by performing mask processing using the set mask for each copy image. Each mask is divided into a processed portion (masked portion) and an unprocessed portion (non-masked portion), and in the processing of step S202, the portion of each copy image corresponding to the processed portion is masked. That is, the portion of each copy image corresponding to the processed portion of the mask is subjected to a predetermined image processing, and the portion corresponding to the unprocessed portion of the mask is used as is to generate a masked image.

[0031] An example of the mask processing performed in step S202 will be described with reference to Fig. 3. For example, in step S201, a target image 301 is acquired as an image to be analyzed, and in step S202, a mask processing is performed on an image obtained by duplicating this target image 301 by applying a mask 302, thereby generating a masked image 303. Two fish 311 and 312 are shown in the target image 301, and the mask 302 has an unprocessed portion 302a and a processed portion 302b. In this case, in the masked image 303, the area of ​​the target image 301 that corresponds to the processed portion 302b is masked, and only a part of the fish 311 existing in the area that corresponds to the unprocessed portion 302a remains.

[0032] In the process of step S202, the area of ​​the copied image of the target image that overlaps with the processing portion of the mask may be filled with, for example, the background color of the target image, or may be filled with a single color such as white or black. Alternatively, a predetermined image filter such as a blurring filter may be applied. In addition, mask processing can be performed using any image processing. There is no limit to the shape or number of masks set during mask processing, and masks of various shapes such as circles and squares can be used. In this case, multiple types of mask shapes may be mixed.

[0033] Furthermore, in step S202, the mask position may be determined randomly or may be biased. An example of biasing the mask position is to place many masks based on the target coordinate position so that the boundary between the processed and unprocessed parts of the mask is located near the target coordinate, thereby creating a difference in the mask placement density. Alternatively, it is possible to bias the mask position by any method, such as generating many masks near a part where there is a difference with the inference result obtained by another analysis method.

[0034] As described above, in the processing of step S202, the image processing unit 102 can adjust at least one of the position, shape, and density of multiple masks set for the target image based on the target coordinates or other coordinates specified in the target image.

[0035] Returning to the explanation of Fig. 2, the inference unit 103 performs inference on each of the masked images generated in step S202 (step S203). Here, inference is performed on each masked image using a model trained by machine learning, thereby determining the class of an object appearing in each masked image.

[0036] In step S203, by the process described above, for each of the masked images generated in step S202, a class representing the classification of the object determined using a model trained by machine learning is acquired for each image region corresponding to the object or background in each masked image as an inference result of the inference unit 103. Note that the inference result for each image region may be acquired in units of pixels in the image region, or may be acquired by thinning out an arbitrary number of pixels. Alternatively, one inference result may be acquired for each image region.

[0037] Next, the inference result extraction unit 104 extracts the inference results at the target coordinates acquired in step S201 from the inference results of each masked image acquired in step S203 (step S204). Here, the inference results at the target coordinates can be extracted by extracting the class of the image region corresponding to the target coordinates from the classes obtained for each image region of each masked image.

[0038] An example of extraction of an inference result performed in step S204 will be described with reference to FIG. 4. For example, in step S202, three masks 401, 411, and 421 are applied to the target image 301 in FIG. 3, thereby generating masked images 402, 412, and 422. In step S203, the inference unit 103 performs inference on each of these masked images 402, 412, and 422, thereby acquiring a class for each image region. For ease of explanation, in the following, a case will be described in which the inference unit 103 performs a semantic segmentation task in which each pixel on each masked image is classified into three classes, namely, a "fish class," a "background class," and a "dog class," in step S203. Here, in general, in class classification judgment, a reliability (score value) that indicates the likelihood of the classification judgment result is obtained for each class in the range from 0 to 1, and the class with the maximum score value is acquired as the classification judgment result.

[0039] Inference results 403, 413, and 423 in Fig. 4 represent the results of inference performed on masked images 402, 412, and 422, respectively. In the inference results 403, 413, and 423, image regions 403a, 413a, and 423a represent regions in the masked images 402, 412, and 422 that have the highest background class score value and are therefore determined to be the background class, respectively. Image regions 403b and 413b represent regions in the masked images 402 and 412 that have the highest fish class score value and are therefore determined to be the fish class, respectively. Image regions 403c and 413c represent regions in the masked images 402 and 412 that have the highest dog class score value and are therefore determined to be the dog class, respectively.

[0040] In the inference results 403, 413, and 423, the coordinates indicated by the reference symbols 403d, 413d, and 423d respectively indicate the object coordinates acquired by the analysis object acquisition unit 101. The object coordinates 403d and 413d belong to the image regions 403b and 413b respectively which have been determined to be the fish class as described above. Therefore, in the process of step S204, the fish class is extracted as the inference result for the object coordinates 403d and 413d respectively. On the other hand, the object coordinates 423d belong to the image region 423a which has been determined to be the background class. Therefore, in the process of step S204, the background class is extracted as the inference result for the object coordinates 423d.

[0041] Returning to the explanation of FIG. 2, the basis generating unit 105 selects one of the multiple masked images generated in step S202 (step S205).

[0042] Next, the evidence generating unit 105 judges whether or not the inference result at the target coordinates extracted in step S204 for the masked image selected in step S205, i.e., the class at the target coordinates, matches the target class acquired in step S201 (step S206). If the class at the target coordinates of the selected masked image matches the target class, the evidence generating unit 105 extracts the mask used to generate the masked image in step S202 as a synthesis target mask and temporarily stores it in a storage device (not shown) (step S207). After performing the process of step S207, the evidence generating unit 105 proceeds to the next step S208. On the other hand, if the class at the target coordinates of the selected masked image does not match the target class, the evidence generating unit 105 proceeds to step S208 without performing the process of step S207.

[0043] Next, the basis generating unit 105 judges whether or not all the masked images have been selected in step S205 (step S208). If all the masked images generated in step S202 have been selected, the process proceeds to step S209, and if there are any unselected masked images remaining, the process returns to step S205. As a result, the processes of steps S206 and S207 are performed on each masked image, and a mask whose class at the target coordinates matches the target class is saved as a synthesis target mask.

[0044] In the example of FIG. 4 described above, the following masks are stored as synthesis target masks according to the object class by the processing of steps S205 to S208. That is, when the object class is a fish class, the masks 401 and 411 used in generating the masked images 402 and 412 from which the inference results 403 and 413, in which the inference results at the object coordinates 403d and 413d are the fish class, are obtained are stored as synthesis target masks. When the object class is a background class, the mask 421 used in generating the masked image 422 from which the inference result 423, in which the inference result at the object coordinates 423d is the background class, is obtained is stored as synthesis target masks. When the object class is a dog class, the inference results 403, 413, and 423 from which the inference result at the object coordinates is the dog class do not exist, so no mask is stored as synthesis target masks.

[0045] Returning to the explanation of FIG. 2, the evidence generating unit 105 generates a composite mask image by superimposing and synthesizing each of the composite target masks stored in step S207, and generates an evidence map based on this composite image mask (step S209). Here, for example, when all of the composite target masks are superimposed, the ratio of the number of overlaps of unprocessed parts (non-masked parts) to the total number is calculated to calculate the evidence rate for each region. Then, the evidence map is generated by visualizing the obtained evidence rate for each region.

[0046] An example of generating a basis map in step S209 will be described with reference to Fig. 5. For example, when two masks 501 and 502 are stored as masks to be synthesized in step S207, a basis map 503 is generated by superimposing these two masks.

[0047] The evidence map 503 has regions 503a, 503b, 503c, and 503d. In the region 503a, the processed portions (masked portions) of the masks 501 and 502 are overlapped, and the evidence ratio in the region 503a is calculated as 0 / 2=0%. In the region 503b, the unprocessed portions of the masks 501 and 502 are overlapped, and the evidence ratio in the region 503b is calculated as 2 / 2=100%. In the regions 503c and 503d, the processed portions of one of the masks 501 and 502 and the unprocessed portions of the other are overlapped, and the evidence ratio in the regions 503c and 503d is calculated as 1 / 2=50%.

[0048] When the generation of the basis map is completed in step S209, the information processing device 100 of this embodiment completes the flowchart of FIG.

[0049] The generated evidence map is presented to the user by being displayed on the display device 111 via the output interface 107, for example. At this time, the display device 111 changes the display form (e.g., color, brightness, etc.) of the evidence map for each region, for example, according to the value of the evidence rate described above. This makes it possible to show the user the basis for classification of the entire target image, which has been classified by image recognition using a model trained by machine learning. At this time, the evidence map may be displayed superimposed on the target image to facilitate comparison with the target image. Also, the target coordinates may be shown on the evidence map.

[0050] According to the first embodiment of the present invention described above, the following advantageous effects are achieved.

[0051] (1) The information processing device 100 includes an analysis target acquisition unit 101 that acquires an image to be analyzed, an image processing unit 102 that generates a plurality of masked images by setting a plurality of masks for the image and masking each of the images with the plurality of masks, an inference unit 103 that performs inference on each of the masked images using a model trained by machine learning and acquires an inference result regarding the classification of the image for each of the masked images, an inference result extraction unit 104 that extracts an inference result at a target coordinate specified in the image from the inference result of each masked image acquired by the inference unit 103, and a basis generation unit 105 that generates a basis map that visualizes the basis for the judgment on the classification result of the image by the model based on the inference result at the target coordinate extracted by the inference result extraction unit 104 and the plurality of masks. As a result of the above, it is possible to provide an information processing device 100 that can show the basis for classification of the entire image for an image classified by image recognition using a model trained by machine learning.

[0052] (2) The inference unit 103 acquires a class representing the classification of the image determined by inference for each of the multiple masked images as an inference result for each image region (step S203). The inference result extraction unit 104 extracts the class of the image region corresponding to the target coordinates from the classes for each image region of each masked image acquired by the inference unit 103 (step S204). The evidence generation unit 105 extracts the mask used to generate the masked image as a synthesis target mask for each masked image in which the class extracted by the inference result extraction unit 104 matches the target class specified for the image from among the multiple masked images (steps S206, S207), generates a synthesis mask image by superimposing and synthesizing the extracted synthesis target masks, and generates an evidence map based on the generated synthesis mask image (step S209). In this way, for any target class, it is possible to generate an evidence map indicating the basis for which the target class was obtained as a classification result for the image.

[0053] (3) The information processing device 100 includes an input interface 106 that accepts an input operation by a user. The analysis target acquisition unit 101 can acquire target coordinates based on an input operation by the user performed via the input interface 106 (step S201). In this way, it is possible to generate a basis map for any target coordinates designated by the user.

[0054] (4) The information processing device 100 includes an output interface 107 that is connected to the display device 111 and provides information to the user by displaying a basis map on the display device 111. As a result, information regarding the basis for classifying an image can be provided to the user in an easily understandable manner using the basis map.

[0055] (5) The output interface 107 can also display a screen in which the basis map is superimposed on the image to be analyzed on the display device 111. In this way, it is possible to provide information to the user in a form in which the image to be analyzed and the basis map can be easily compared.

[0056] (6) The information processing device 100 includes an external interface 108 that is connected to an external information device 112. The analysis target acquisition unit 101 can also acquire target coordinates via the external interface 108 (step S201). In this way, it becomes possible to generate a basis map for the specified target coordinates by utilizing inference results obtained by other analysis methods.

[0057] (7) The image processing unit 102 can adjust at least one of the position, shape, and density of the multiple masks set for the image based on the target coordinates or other coordinates specified in the image (step S202). In this way, it is possible to automatically obtain multiple masks required for generating a basis map for the image to be analyzed in an appropriate manner.

[0058] (8) The image processing unit 102 generates a masked image by using the unmasked portion of the image as is, and generates a masked image by performing a predetermined image processing on the masked portion of the image (step S202). In this way, a masked image can be easily generated from the image to be analyzed.

[0059] [Second embodiment] Next, an information processing device according to a second embodiment of the present invention will be described with reference to Fig. 6 and Fig. 7. The information processing device of this embodiment has a similar configuration to the information processing device 100 of Fig. 1 described in the first embodiment. Therefore, hereinafter, this embodiment will be described using the configuration of the information processing device 100 of Fig. 1.

[0060] A method for generating a basis map in the information processing device 100 of this embodiment will be described below. Fig. 6 is a flowchart showing an example of the processing contents of the information processing device 100 according to the second embodiment of the present invention. In the flowchart of Fig. 6, the same step numbers as in Fig. 2 are assigned to the parts performing the same processing as in the flowchart of Fig. 2 described in the first embodiment. In the following, a description of the processing with the same step numbers will be omitted.

[0061] The analysis target acquisition unit 101 acquires an image to be analyzed and acquires target coordinates in the target image (step S201A). Note that, unlike the first embodiment, the present embodiment acquires a target image and target coordinates, but does not need to acquire a target class.

[0062] After the image processing unit 102 executes the process of step S202, the inference unit 103 performs inference on each of the masked images generated in step S202 (step S203A). As in the first embodiment, inference is performed on each masked image using a model trained by machine learning to determine the class of an object appearing in each masked image. Furthermore, in this embodiment, a score value is calculated representing the reliability of the class determined for each object in each masked image. This score value changes depending on the degree of learning of the model used by the inference unit 103 in inference, and generally the more advanced the model's learning is, the higher the score value is.

[0063] Next, the inference result extraction unit 104 extracts the inference results at the target coordinates acquired in step S201A from the inference results of each masked image acquired in step S203A (step S204A). Here, the inference results at the target coordinates can be extracted by extracting the score value of the image region corresponding to the target coordinates from the score values ​​obtained for each image region of each masked image.

[0064] Next, the evidence generation unit 105 sets each mask used to generate the masked image in step S202 as a synthesis target mask, combines it with the inference result at the target coordinates extracted in step S204A, i.e., the score value at the target coordinates, and temporarily stores it in a storage device not shown (step S207A).

[0065] Thereafter, the basis generating unit 105 weights each of the composite target masks stored in step S207A at a ratio according to the score value, and generates a composite mask image by superimposing and composing these. A basis map is generated based on the composite image mask thus generated (step S209A). That is, weighting values ​​according to the score value are set for unprocessed parts (non-masked parts) of all masks, and the weighting values ​​of the unprocessed parts that overlap when the masks are superimposed are summed up and divided by the number of masks to calculate a basis coefficient for each region. Then, the basis coefficient of each region thus obtained is visualized to generate a basis map.

[0066] An example of generating a basis map in step S209A will be described with reference to Fig. 7. For example, when two masks 601 and 602 are saved as composite target masks in step S207A, the basis map 603 is generated by superimposing these two masks. For example, the score value of 0.9 extracted in step S204A is set as a weighting value to the unprocessed portion of the mask 601, and the score value of 0.8 extracted in step S204A is set as a weighting value to the unprocessed portion of the mask 602.

[0067] The evidence map 603 has regions 603a, 603b, 603c, and 603d. In the region 603a, the processed portions (masked portions) of the masks 601 and 602 are overlapped, and the evidence coefficient in the region 603a is calculated as (0×0.9+0×0.8) / 2=0%. In the region 603b, the unprocessed portions of the masks 601 and 602 are overlapped, and the evidence coefficient in the region 603b is calculated as (1×0.9+1×0.8) / 2=85%. In the region 603c, the unprocessed portion of the mask 601 and the processed portion of the mask 602 are overlapped, and the evidence coefficient in the region 603c is calculated as (1×0.9+0×0.8) / 2=45%. In region 603d, the processed portion of mask 601 and the unprocessed portion of mask 602 are overlapped, and the basis coefficient in this region 603d is calculated to be (0×0.9+1×0.8) / 2=40%.

[0068] When the generation of the basis map is completed in step S209A, the information processing device 100 of this embodiment completes the flowchart of FIG.

[0069] According to the second embodiment of the present invention described above, the inference unit 103 acquires a score value representing the reliability of inference for the classification of the target image for each of the multiple masked images as an inference result for each image region (step S203A). The inference result extraction unit 104 extracts the score value of the image region corresponding to the target coordinates from the score values ​​for each image region of each masked image acquired by the inference unit 103 (step S204A). The evidence generation unit 105 generates a composite mask image by superimposing and synthesizing multiple masks at a ratio according to the score value extracted by the inference result extraction unit 104, and generates an evidence map based on the generated composite mask image (step S209A). In this way, it is possible to generate an evidence map indicating the evidence obtained as the classification result of the image for all classes.

[0070] [Third embodiment] Next, an information processing device according to a third embodiment of the present invention will be described with reference to Fig. 8. Note that, like the second embodiment, the information processing device of this embodiment also has a similar configuration to the information processing device 100 of Fig. 1 described in the first embodiment. Therefore, hereinafter, this embodiment will be described using the configuration of the information processing device 100 of Fig. 1.

[0071] A method for generating a basis map in the information processing device 100 of this embodiment will be described below. Fig. 8 is a flowchart showing an example of the processing contents of the information processing device 100 according to the third embodiment of the present invention. In the flowchart of Fig. 8, the same step numbers as in Fig. 2 and Fig. 6 are assigned to parts performing the same processing as in the flowcharts of Fig. 2 and Fig. 6 described in the first and second embodiments, respectively.

[0072] First, the analysis target acquisition unit 101 acquires an image to be analyzed, and acquires target coordinates and target class in the target image, as in the first embodiment (step S201). Next, the image processing unit 102 masks the target image acquired in step S201, as in the first embodiment, to generate a masked image (step S202). After that, the inference unit 103 performs inference on each of the multiple masked images generated in step S202 (step S203A). Here, as in the second embodiment, the class of the object in each masked image is determined, and a score value is calculated.

[0073] Next, the inference result extraction unit 104 extracts the inference results at the target coordinates acquired in step S201 from the inference results of each masked image acquired in step S203A (step S204B). Here, the inference result at the target coordinates can be extracted by extracting the class and score value of the image region corresponding to the target coordinates from the class and score value obtained for each image region for each masked image.

[0074] Next, as in the first embodiment, the evidence generating unit 105 selects one of the multiple masked images generated in step S202 (step S205), and determines whether the class at the target coordinates extracted in step S204B for the selected masked image matches the target class acquired in step S201 (step S206). As a result, if the class at the target coordinates of the selected masked image matches the target class, the evidence generating unit 105 extracts the mask used to generate the masked image in step S202 as a synthesis target mask, combines it with the score value at the target coordinates extracted in step S204B, and temporarily stores it in a storage device (not shown) (step S207B). After performing the process of step S207B, the evidence generating unit 105 proceeds to the next step S208. On the other hand, if the class at the target coordinates of the selected masked image does not match the target class, the evidence generating unit 105 proceeds to step S208 without performing the process of step S207B.

[0075] Next, the basis generating unit 105 judges whether or not all the masked images have been selected in step S205 (step S208). If all the masked images generated in step S202 have been selected, the process proceeds to step S209A, and if there are any unselected masked images remaining, the process returns to step S205. As a result, the processes of steps S206 and S207B are performed on each masked image, and a mask whose class at the target coordinates matches the target class is stored as a synthesis target mask together with a score value.

[0076] The basis generating unit 105 generates a composite mask image by superimposing and synthesizing each of the composite target masks stored in step S207B, and generates a basis map based on this composite image mask (step S209A). Here, as in the second embodiment, each of the composite target masks stored in step S207B is weighted at a ratio according to the score value, and these are superimposed and synthesized to generate a composite mask image. A basis map is generated based on the composite image mask generated in this way.

[0077] When the generation of the basis map is completed in step S209A, the information processing device 100 of this embodiment completes the flowchart of FIG.

[0078] According to the third embodiment of the present invention described above, the inference unit 103 further acquires, for each of the multiple masked images, a score value representing the reliability of the inference for the classification of the target image as an inference result for each class (step S203A). The inference result extraction unit 104 extracts a class and a score value corresponding to the target coordinates of each masked image acquired by the inference unit 103 (step S204B). The evidence generation unit 105 superimposes and synthesizes each synthesis target mask at a ratio according to the score value extracted by the inference result extraction unit 104 to generate a synthetic mask image (step S209A). In this way, it is possible to generate an evidence map showing more detailed evidence for any target class.

[0079] Note that each of the first to third embodiments described above may be preset in the information processing device 100, or may be arbitrarily selectable by the user through an input operation input from the input device 110 via the input interface 106. For example, in step S201 in Fig. 2 and Fig. 8 or step S201A in Fig. 6, when a target image, target coordinates, or target class is acquired in response to an input operation by the user, the user may be allowed to select a method for generating a basis map, thereby determining which embodiment to apply.

[0080] [Fourth embodiment] Next, an information processing device according to a fourth embodiment of the present invention will be described with reference to FIGS.

[0081] Fig. 9 is a block diagram showing an example of the configuration of an information processing device 100A according to a fourth embodiment of the present invention. As shown in Fig. 9, the information processing device 100A according to this embodiment further includes a training image generating unit 121 and an additional candidate image storage unit 122 in addition to the elements of the information processing device 100 according to the first embodiment shown in Fig. 1. The training image generating unit 121 is realized by, for example, a CPU executing a predetermined program, and the additional candidate image storage unit 122 is configured using a storage device such as an HDD or SSD.

[0082] The training image generation unit 121 generates training images used in machine learning of the model. This model is used for image classification in an analysis device (not shown), and is also used for inference performed by the inference unit 103. The training images generated by the training image generation unit 121 are input to, for example, a learning device (not shown), and used in machine learning of the model performed by the learning device. Note that a machine learning unit may be provided within the information processing device 100A, and machine learning of the model may be performed in this machine learning unit.

[0083] The additional candidate image storage unit 122 stores one or more additional candidate images registered in advance. Each additional candidate image stored in the additional candidate image storage unit 122 is, for example, an image showing an object that is the same as or similar to an object to be analyzed by the analysis device, and is used when the training image generation unit 121 generates training images. In other words, the training image generation unit 121 can generate training images for machine learning based on the additional candidate images stored in the additional candidate image storage unit 122.

[0084] FIG. 10 is a flowchart showing an example of the processing contents of the information processing device 100A according to the fourth embodiment of the present invention.

[0085] In step S200, a basis map generation process is executed. Here, a basis map is generated for a target image according to any one of the flowcharts in Fig. 2, Fig. 6, and Fig. 8 described in the first to third embodiments. In the information processing device 100A of this embodiment, a learning image is generated using this basis map.

[0086] 11 is a diagram showing an example of an image from which a learning image is generated in the information processing device 100A of this embodiment. In this embodiment, an example of generating a learning image in order to improve the accuracy of an analysis process performed in an analysis device (not shown) will be described.

[0087] Images 701 and 711 in FIG. 11 are examples of images captured by an electron microscope during semiconductor inspection. The analysis device executes a task of recognizing the tips of needles 701a and 711a captured in these images using semantic segmentation. Here, image 701 captures only needle 701a, which is the detection target, while image 711 captures not only needle 711a, which is the detection target, but also dust 711b, which is not the detection target. It is assumed that a semantic segmentation model has already been trained in advance in the analysis device using predetermined training data.

[0088] When the results of executing a task by an analysis device are superimposed on images 701 and 711, inference results 702 and 712, for example, are obtained. In the inference results 702 and 712, circles 702a and 712a are drawn around the recognized tips of needles 701a and 711a, respectively. In the inference result 712, the tip of dust 711b is also mistakenly recognized as the tip of a needle, resulting in a circle 712b being drawn.

[0089] Here, the task executed on images 701 and 711 aims to recognize the tip of the needle and to determine that the other parts are background class. However, in the inference results 702 and 712 in FIG. 11, only the parts recognized as the tip of the needle are shown as circles, and the background class is not explicitly shown because it is a wide range. In the example of FIG. 11, the inference result 702 is ideal because a circle 702a is correctly drawn around the tip of the needle 701a and the other parts are determined to be background class. On the other hand, the inference result 712 is not preferable because a circle 712a is correctly drawn around the tip of the needle 711a, but a circle 712b is incorrectly drawn around the dust 711b.

[0090] In the information processing device 100A of this embodiment, for example, an image that is expected to have a high effect of suppressing erroneous recognition of such dust 711b is selected, and a learning image is generated using the selected image. The generated learning image is provided from the information processing device 100A to a learning device (not shown) and is used in the machine learning of the model performed by the learning device.

[0091] Returning to the explanation of Fig. 10, the training image generating unit 121 determines a template region based on the evidence map generated by the evidence map generating process in step S200 (step S301). Here, for example, based on the distribution of evidence levels (evidence rates or evidence coefficients) for classification results on the target image represented by the evidence map, a part of the target image used to generate the evidence map is extracted as the template region. Specifically, for example, a threshold value for the evidence level is set for the evidence map, and an area of ​​the target image corresponding to an area of ​​the evidence map having an evidence level value greater than the threshold is extracted as the template region.

[0092] An example of the template region determination performed in step S301 will be described with reference to FIG. 12. An image 711 shown in FIG. 12 is the same as the image 711 exemplified in FIG. 11. When the image 711 is used as a target image and the tip of dust 711b is specified as target coordinates 801b to perform the basis map generation process in step S209, masks 802 and 803, for example, are set, and a basis map 804 is generated by overlapping these masks. In the process of step S301, when a threshold value of 80% is set for the basis map 804, for example, a region 804a whose basis exceeds the threshold value of 80% is selected, and a region 805 of the image 711 corresponding to the region 804a is extracted as the template region. The template region 805 extracted in this way includes the dust 711b specified by the target coordinates 801b.

[0093] The threshold value for determining the template region in step S301 may be specified, for example, according to a user's input operation input from the input device 110 via the input interface 106, or may be automatically specified by the information processing device 100A with reference to the quartiles or average value of the evidence level in the entire evidence map. The size and shape of the template region can be set arbitrarily. For example, a portion of the evidence map whose evidence level satisfies the threshold value may be set as the template region in pixel units, or a rectangular, circular, or other region large enough to include those pixels may be set as the template region.

[0094] Returning to the explanation of Fig. 10, the training image generation unit 121 selects one of the additional candidate images stored in the additional candidate image storage unit 122 (step S302). Next, the training image generation unit 121 performs template matching on the additional candidate image selected in step S302 using the template region determined in step S301 (step S303). Here, for example, a portion of the additional candidate image that has the highest similarity to the template region is determined, and the similarity of that portion is extracted as a matching result.

[0095] In the template matching in step S303, the template region determined in step S301 may be subjected to image transformation such as change in size or angle, inversion, or binarization. In this case, whether or not to apply image transformation to the template region may be selected depending on the type of object targeted by the task. For example, as described in each of the first to third embodiments, in the case of a task targeting a fish, it is considered that the size and orientation of the fish may change within the image. Therefore, it is assumed that the similarity of the template region is appropriately obtained by performing template matching using the template region to which the above-mentioned image transformation has been applied. On the other hand, the examples of Figs. 11 and 12 described in this embodiment are tasks targeting artificial objects in an image captured by a microscope. In such a task, it is considered that there is little change in size or orientation within the image, so that if the above-mentioned image transformation is applied, there is a possibility that a high similarity will be erroneously obtained in a location different from that expected. Therefore, in these examples, it is considered necessary to perform template matching without applying image transformation to the template region. In this way, when performing template matching in step S303, it is preferable to select whether or not to apply image transformation in consideration of the characteristics of the template region and the image to be compared. In this case, the type of image transformation to be applied may be selected.

[0096] After performing template matching, the learning image generation unit 121 determines whether or not all additional candidate images have been selected in step S302 (step S304). If all additional candidate images stored in the additional candidate image storage unit 122 have been selected, the process proceeds to step S305, and if unselected additional candidate images remain, the process returns to step S302. As a result, template matching is performed on each additional candidate image in step S303, and as a result, a matching result for each additional candidate image is extracted.

[0097] Finally, the training image generating unit 121 generates a training image based on each additional candidate image for which template matching was performed in step S303 (step S305). Here, for example, among the matching results for each additional candidate image, an additional candidate image for which a matching result with the highest similarity to the template region was obtained is selected and set as the training image. This makes it possible to generate a training image that is estimated to have a high effect of improving accuracy in machine learning based on the template region determined based on the evidence map. Note that the training image may be generated by using the selected additional candidate image as it is, or the training image may be generated by performing a predetermined image processing on the selected additional candidate image.

[0098] An example of the learning image generation performed in step S305 will be described with reference to FIG. 13. Here, it is assumed that the additional candidate images 901 and 911 are stored in the additional candidate image storage unit 122, and that the regions 901a and 911a having the highest similarity to the template region 805 in the additional candidate images 901 and 911 are extracted by performing template matching on these additional candidate images 901 and 911 using the template region 805 in FIG. 12. The region 901a of the additional candidate image 901 contains dust having a shape similar to the dust 711b in the image 711 in FIG. 12 from which the template region 805 was extracted, and therefore the similarity is obtained as a relatively high value. On the other hand, the additional candidate image 911 does not contain dust, and the region 911a having the highest similarity to the template region 805 is extracted from the additional candidate image 911, but the similarity value of this region 911a is smaller than that of the region 901a of the additional candidate image 901.

[0099] In the above situation, when the process of step S305 is executed by the learning image generating unit 121, the additional candidate image 901 from which the region 901a is obtained is selected, and the learning image 902 is set based on this. The learning image 902 is generated by superimposing a circle 902a representing an annotation as teacher data on the tip portion of the needle shown in the additional candidate image 901. Note that a background class is set to the portion of the learning image 902 other than the circle 902a for annotation.

[0100] As described above, in the learning image 902, the part corresponding to the area 901a containing dust is set to the background class. Therefore, if machine learning is further performed using the learning image 902 as training data and image analysis is performed using a model reflecting the learning result, it is possible to prevent dust from being erroneously determined to be the tip of a needle. That is, in the inference result 712 in FIG. 11, it is possible to prevent a circle 712b from being erroneously drawn for the tip of dust 711b.

[0101] In the process of step S305, not only the additional candidate image with the highest matching result of similarity with the template region is selected, but also a threshold value for the matching result may be set, and all additional candidate images with similarity with the template region exceeding this threshold may be selected, and these may be used to generate learning images. Learning images may also be generated based on additional candidate images that satisfy other conditions. For example, learning images may be generated using additional candidate images that show unique characteristics, such as an additional candidate image whose similarity value with the template region is significantly different from that of other additional candidate images. Furthermore, additional candidate images selected based on the result of template matching may be presented to the user by displaying them on the display device 111 via the output interface 107, and learning images may be generated using additional candidate images that the user has permitted or specified among them.

[0102] When the generation of the learning images is completed in step S305, the information processing apparatus 100A of this embodiment completes the flow chart of FIG.

[0103] According to the fourth embodiment of the present invention described above, the information processing device 100A includes a training image generating unit 121 that extracts a part of a target image as a template region based on the evidence map generated by the evidence generating unit 105, and generates a training image to be used in machine learning based on the extracted template region. As a result, it is possible to improve the accuracy of image analysis processing performed using a machine-learned model by utilizing the evidence map.

[0104] According to the fourth embodiment of the present invention described above, the basis map represents the distribution of basis levels for the classification results on the target image. The learning image generating unit 121 extracts a template region based on a threshold value of the basis level specified for the basis map (step S301). In this manner, it is possible to extract an appropriate portion of the target image as the template region by using the basis map.

[0105] Furthermore, according to the fourth embodiment of the present invention described above, the training image generating unit 121 generates training images by extracting portions of pre-acquired additional candidate images whose similarity to the template region satisfies a predetermined condition (steps S303 and S305). This makes it possible to easily generate appropriate training images based on the template region.

[0106] The present invention is not limited to the above-described embodiment, and various modifications are possible without departing from the spirit of the present invention. Each embodiment may be implemented alone, or any combination of multiple embodiments may be applied. [Explanation of symbols]

[0107] 100, 100A: information processing device, 101: analysis target acquisition unit, 102: image processing unit, 103: inference unit, 104: inference result extraction unit, 105: evidence generation unit, 106: input interface, 107: output interface, 108: external interface, 109: bus, 110: input device, 111: display device, 112: information device, 121: learning image generation unit, 122: additional candidate image storage unit

Claims

1. an analysis target acquisition unit that acquires an image to be analyzed; an image processing unit that sets a plurality of masks for the image and generates a plurality of masked images by masking the image using the plurality of masks; An inference unit that performs inference on each of the masked images using a model trained by machine learning, and obtains an inference result regarding the classification of each of the masked images; an inference result extraction unit that extracts an inference result at a target coordinate specified in each masked image from the inference result of each masked image acquired by the inference unit; and a basis generation unit that generates a basis map that visualizes a judgment basis for the classification result of the image by the model based on the inference result at the target coordinates extracted by the inference result extraction unit and the multiple masks.

2. 2. The information processing device according to claim 1, The inference unit obtains, for each of the plurality of masked images, a class representing a classification of the image determined by the inference, as the inference result for each image region; The inference result extraction unit extracts a class of an image region corresponding to the target coordinates from among classes for each image region of each masked image acquired by the inference unit; The information processing device, wherein the basis generation unit extracts, for each masked image among the plurality of masked images in which the class extracted by the inference result extraction unit matches a target class specified for the image, a mask used to generate the masked image as a synthesis target mask, generates a composite mask image by overlaying and synthesizing each of the extracted synthesis target masks, and generates the basis map based on the generated composite mask image.

3. 3. The information processing device according to claim 2, The inference unit further obtains, for each of the plurality of masked images, a score value representing a reliability of the inference for the classification of the image as the inference result for each of the classes; The inference result extraction unit extracts the class and the score value at the target coordinates of each masked image, The information processing device, wherein the basis generating unit overlays and synthesizes each of the synthesis target masks at a ratio according to the score value extracted by the inference result extracting unit, thereby generating the synthetic mask image.

4. 2. The information processing device according to claim 1, the inference unit acquires, for each of the plurality of masked images, a score value representing a reliability of the inference for the classification of the image, as the inference result for each image region; The inference result extraction unit extracts a score value of an image region corresponding to the target coordinates from among the score values ​​for each image region of each masked image acquired by the inference unit; The information processing device, wherein the basis generation unit generates a composite mask image by overlaying and synthesizing the multiple masks in a ratio corresponding to the score value extracted by the inference result extraction unit, and generates the basis map based on the generated composite mask image.

5. 2. The information processing device according to claim 1, An information processing device comprising: a training image generation unit that extracts a portion of the image as a template region based on the basis map, and generates a training image to be used in the machine learning based on the extracted template region.

6. 6. The information processing device according to claim 5, the evidence map represents a distribution of evidence levels for the classification results on the image; The learning image generation unit extracts the template region based on a threshold value of the degree of evidence specified for the evidence map.

7. 6. The information processing device according to claim 5, The training image generation unit generates the training image by extracting a portion of an additional candidate image acquired in advance, the portion having a similarity to the template region that satisfies a predetermined condition.

8. 2. The information processing device according to claim 1, An input interface for receiving an input operation from a user; The analysis target acquisition unit acquires the target coordinates based on an input operation performed by the user via the input interface.

9. 2. The information processing device according to claim 1, An information processing device comprising an output interface connected to a display device and providing information to a user by causing the display device to display the basis map.

10. 10. The information processing device according to claim 9, The output interface is an information processing device that causes the display device to display a screen in which the grounds map is superimposed on the image.

11. 2. The information processing device according to claim 1, An external interface is provided for connection to an external information device; The analysis target acquisition unit acquires the target coordinates via the external interface.

12. 2. The information processing device according to claim 1, The image processing unit adjusts at least one of the positions, shapes, and densities of the plurality of masks set for the image based on the target coordinates or other coordinates specified within the image.

13. 2. The information processing device according to claim 1, The image processing unit generates the masked image by using unmasked parts of the image as is, and generates the masked image by performing a predetermined image processing on masked parts of the image.

14. 2. The information processing device according to claim 1, The analysis object acquisition unit is an information processing device that acquires an image captured by an electron microscope as the image to be analyzed.

15. An image processing method using an information processing device, comprising: Acquire the image to be analyzed. Setting a plurality of masks for the image; masking the image with the plurality of masks to generate a plurality of masked images; By performing inference on each of the masked images using a trained model by machine learning, an inference result regarding a classification of the image is obtained for each of the masked images; Extracting an inference result at a target coordinate specified in each of the acquired masked images from the inference result of the masked images; An image processing method that generates a basis map that visualizes the basis for the judgment of the classification result of the image by the model based on the inference result at the extracted target coordinates and the multiple masks.

Images