Material analysis support device, material analysis support method, and program

Abstract

To provide a material analysis support device or the like for immediately analyzing factors causing quality differences among multiple groups.SOLUTION: A material analysis support device 10 includes: a classification model creation unit 21 which creates a classification model 3 for classifying a material image 1 by using multiple material images 1 and group information 2 assigned to each of the images; an explanation model creation unit 31 which creates an explanation model 4 for explaining the classification basis of the material images 1 by the classification model 3; an explanation image creation unit 33 which creates an explanation image 5 that quantifies the classification basis of the material images 1 for one or more material images 1 by using the explanation model 4; and an explanation image output unit 35 which visualizes the explanation image 5. In addition, the material analysis support device 10 includes a classification model verification unit 23 which verifies the classification results of the material images 1 by the classification model 3 using the chi-square test.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a material analysis support device, a material analysis support method, and a program, and more particularly to a technique for supporting analysis in quality control of materials. [Background technology]

[0002] The Seven QC Tools, a statistically based analytical method, is widely used in the quality control of various products. When there are differences in quality between groups, such as multiple manufacturing facilities, multiple production lots, multiple workers, or multiple vendors, stratification and histograms, two of the Seven QC Tools, are often used. Stratification involves classifying data acquired for quality control into groups and comparing them. Histograms, on the other hand, are a method for visualizing data variability. Furthermore, statistical hypothesis testing is used to statistically analyze differences between data across multiple groups by combining stratification and histograms. When analyzing differences between two groups, tests of mean differences and median differences are often used. Analysis of variance (ANOVA) is often used to analyze differences between three or more groups.

[0003] In the quality control of new development and mass production of materials such as metals, resins, magnets, and ceramics, it is common to image and observe microstructures using electron microscopes, optical microscopes, electron microanalyzers, etc. In order to analyze differences between groups in the imaged material images, it is necessary to extract several types of feature quantities using image processing technology and analyze the data, as described in Patent Document 1, for example. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 7147974 Summary of the Invention [Problem to be solved by the invention]

[0005] The method in Patent Document 1 extracts feature quantities from material images by making full use of image processing technology. However, because the feature quantities to be extracted differ depending on the material being analyzed, it is necessary to develop a dedicated image processing technology for each material, which makes it unsuitable for tasks where immediate results are important, such as analysis in quality control.

[0006] The present invention has been made in consideration of the above circumstances, and aims to provide a material analysis support device etc. for instantly analyzing the causes of quality differences between multiple groups. [Means for solving the problem]

[0007] The first invention for solving the above-mentioned problems is a material analysis support device comprising: a classification model creation unit that creates a classification model for classifying material images using a plurality of material images and group information assigned to each of the material images; an explanatory model creation unit that creates an explanatory model that explains the classification basis of the material images using the classification model; an explanatory image creation unit that uses the explanatory model to create an explanatory image for one or more material images that quantifies the classification basis of the material image; and an explanatory image output unit that visualizes the explanatory image.

[0008] According to the first invention, a classification model for classifying ingredient images is created using multiple ingredient images and group information assigned to each. Classifying ingredient images using this classification model makes it possible to immediately analyze the factors that cause quality differences between multiple groups. Furthermore, by using an explanatory model to create and visualize explanatory images that quantify the basis for the classification of ingredient images, the basis for the classification of ingredient images can be easily understood. Since this invention does not require the development of dedicated image processing technology for each ingredient, rapid quality control analysis is realized, significantly improving the quality control process.

[0009] In the first invention, the explanatory image creation unit may create explanatory images that quantify the classification grounds of the ingredient images for each group, and the explanatory image output unit may visualize the created explanatory images for each group. By creating and visualizing explanatory images that quantify the classification grounds for each group, it is possible to easily understand the factors that cause quality differences between groups.

[0010] The image processing system may further include a region dividing unit that divides the ingredient image into regions, a counting unit that uses the explanation image corresponding to the ingredient image to count indices that quantify classification grounds for each divided region, and a divided image output unit that visualizes region-divided images that display the divided regions based on the counting results of the counting unit. The divided image output unit may also align and display the region-divided images of each region based on the counting results, or select and display the region-divided images of a specific region based on the counting results. By dividing the ingredient image into regions and counting indices that quantify classification grounds for each region, the contribution of each region to the classification into groups can be determined. This makes it possible to grasp the importance and differences in characteristics of each region. Visualizing the divided regions based on the counting results allows for preferential observation of specific regions or for more detailed evaluation of quality differences.

[0011] The explanatory image is, for example, an image based on the SHAP value, which makes it possible to visualize the classification basis for each pixel.

[0012] A second invention is a material analysis support device characterized by comprising: a classification model creation unit that creates a classification model for classifying a plurality of material images using group information assigned to each of the material images; and a classification model verification unit that verifies the classification results of the material images obtained by classifying the material images using the classification model using a chi-square test.

[0013] According to the second invention, a classification model for classifying ingredient images is created using multiple ingredient images and group information assigned to each. Classifying ingredient images using this classification model enables immediate analysis of the factors that cause quality differences between multiple groups. Furthermore, verifying the classification results of the classification model using a chi-square test allows objective evaluation of whether there are any differences between ingredients in the groups. This invention eliminates the need to develop dedicated image processing technology for each ingredient, enabling rapid quality control analysis and significantly improving the quality control process.

[0014] The third invention is a material analysis support method characterized in that a computer executes the steps of: creating a classification model for classifying a plurality of material images using group information assigned to each of the material images; creating an explanatory model for explaining the classification basis of the material images according to the classification model; creating an explanatory image for one or more of the material images using the explanatory model, quantifying the classification basis of the material images; and visualizing the explanatory image.

[0015] A fourth invention is a material analysis support method characterized in that a computer executes the steps of creating a classification model for classifying a plurality of material images using group information assigned to each of the material images, and verifying the classification results of the material images classified by the classification model using a chi-square test.

[0016] The fifth invention is a program for causing a computer to function as a classification model creation unit that uses a plurality of ingredient images and group information assigned to each of the ingredient images to create a classification model for classifying the ingredient images; an explanatory model creation unit that creates an explanatory model that explains the basis for classifying the ingredient images using the classification model; an explanatory image creation unit that uses the explanatory model to create an explanatory image for one or more of the ingredient images that quantifies the basis for classifying the ingredient images; and an explanatory image output unit that visualizes the explanatory image.

[0017] A sixth invention is a program that causes a computer to function as a classification model creation unit that creates a classification model for classifying a plurality of ingredient images using group information assigned to each of the ingredient images, and a classification model verification unit that verifies the classification results of the ingredient images using the classification model using a chi-square test. [Effects of the Invention]

[0018] According to the present invention, a material analysis support device and the like are provided for instantly analyzing the causes of quality differences between a plurality of groups. [Brief explanation of the drawings]

[0019] [Figure 1] FIG. 1 is a diagram illustrating an example of the hardware configuration of a computer used as a material analysis support device. [Figure 2] FIG. 2 is a block diagram showing an example of a functional configuration of the material analysis support device. [Figure 3] FIG. 10 is a diagram showing an example of data of an ingredient image to which group information has been assigned. [Figure 4] 10 is a flowchart showing an example of the flow of processing executed by the material analysis support device. [Figure 5] This is an example of a program that creates a classification model that classifies data into two groups. [Figure 6] 10 is an example of a verification screen that visualizes the verification results of a classification model. [Figure 7] This is an example of a program that creates an explanatory model. [Figure 8] FIG. 10 is a diagram illustrating a visualization example of an explanatory image. [Figure 9] FIG. 10 is a diagram illustrating a visualization example of an explanatory image. [Figure 10] FIG. 2 is a block diagram showing an example of a functional configuration of the material analysis support device. [Figure 11] FIG. 10 is a diagram illustrating an example of area division data. [Figure 12] FIG. 10 is a diagram illustrating an example of aggregated data. [Figure 13] 10 is a flowchart showing an example of the flow of processing executed by the material analysis support device. [Figure 14] FIG. 10 is a diagram illustrating an example of visualization of a region-divided image. [Figure 15] FIG. 15 is a diagram showing a material image that is the basis of the region division image of FIG. 14. [Figure 16] This is an example of a program that creates a classification model that classifies data into three groups. [Figure 17] 10 is an example of a verification screen that visualizes the verification results of a classification model. [Figure 18] FIG. 10 is a diagram illustrating a visualization example of an explanatory image. DETAILED DESCRIPTION OF THE INVENTION

[0020] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[0021] [First embodiment] Fig. 1 is a diagram showing an example of the hardware configuration of a materials analysis support device 10 according to an embodiment of the present invention. For example, when a general personal computer is used as the materials analysis support device 10, the materials analysis support device 10 includes a control unit 11, a storage unit 12, a communication unit 13, an input unit 14, a display unit 15, a peripheral device I / F unit 16, and the like, which are connected via a bus 17, as shown in Fig. 1. Note that the configuration shown in Fig. 1 is just an example, and the materials analysis support device 10 can have various configurations depending on the application and purpose.

[0022] The control unit 11 has a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), etc. The CPU loads a program stored in the storage unit 12 or a recording medium such as the ROM into a work memory area on the RAM and executes the program, and drives and controls each unit connected via a bus 17 to realize each process of the materials analysis support device 10, which will be described later.

[0023] The ROM is a non-volatile memory that permanently stores the computer's boot program, programs such as the BIOS, and data, etc. The RAM is a volatile memory that temporarily stores programs and data loaded from the storage unit 12 or a recording medium such as a ROM, and also has a work area that the control unit 11 uses to perform various processes.

[0024] The storage unit 12 is a hard disk drive (HDD). The storage device may be a hard disk drive (HDD), a solid state drive (SSD), a flash memory, or the like, and stores the program executed by the control unit 11, data required for program execution (data such as material image 1), an operating system (OS), and the like. The data required for program execution may be stored on a network such as a cloud server. In this case, the control unit 11 downloads the data from the cloud server or the like before or when executing the program.

[0025] The communication unit 13 includes a communication interface and a communication control circuit that mediate communication of the materials analysis support device 10, and controls communication via a network. The network may be a local area network (LAN), a wide area network (WAN), the Internet, or the like, and may be wired or wireless.

[0026] The input unit 14 includes input devices such as a keyboard, a mouse, or a touch panel, and various operation buttons, etc. The input unit 14 transmits input data and operation instructions to the control unit 11.

[0027] The display unit 15 includes a display such as a liquid crystal panel, and displays data such as images and text on the display in accordance with instructions from the control unit 11.

[0028] The peripheral device I / F unit 16 is a port for connecting peripheral devices, and includes short-range wireless communication such as USB and Bluetooth (registered trademark). The control unit 11 transmits and receives data to and from peripheral devices via the peripheral device I / F unit 16. A printer or the like is connected to the peripheral device I / F unit 16, for example.

[0029] Next, the functional configuration of the material analysis support device 10 will be described. Fig. 2 is a block diagram showing the functional configuration of a materials analysis support device 10 according to this embodiment. As shown in Fig. 2, the materials analysis support device 10 includes a classification unit 20 (classification model creation unit 21, classification model verification unit 23) and an explanation unit 30 (explanation model creation unit 31, explanation image creation unit 33, explanation image output unit 35). A control unit 11 of the materials analysis support device 10 reads out and executes programs stored in a storage unit 12 or a ROM, thereby realizing these functions. These functions allow the material analysis support device 10 to support rapid analysis in the quality control of materials. Each function will be explained below.

[0030] The classification unit 20 is a functional unit that creates and verifies a classification model 3 for classifying the ingredient images 1. As shown in FIG. 2, the classification unit 20 is made up of a classification model creation unit 21 and a classification model verification unit 23.

[0031] The classification model creation unit 21 uses a plurality of ingredient images 1 and group information 2 assigned to each of them to create a classification model 3 for classifying the ingredient images 1 into groups.

[0032] The material image 1 is an image of a material captured using an electron microscope, optical microscope, electron microanalyzer, or the like. In this embodiment, the material image 1 is an image of silicon carbide powder delivered from a vendor captured using an electron microscope and then trimmed into multiple small images. In this embodiment, the material image 1 is a grayscale image (brightness image) in which each pixel has a brightness value. However, the material image 1 is not limited to a grayscale image (brightness image). When capturing an image of powder using an electron microscope, it is desirable to intentionally scatter the powder and capture the image. This allows the material image 1 to be captured in such a way that the powder particles barely overlap each other.

[0033] Group information 2 is assigned to each material image 1. Group information 2 indicates attribute information related to the material, such as the production lot, production equipment, worker, vendor, etc. In this embodiment, a case will be described in which group information 2 is a production lot ("Lot A" and "Lot B"). "Lot A" and "Lot B" are, for example, lots of silicon carbide powder delivered from different vendors.

[0034] Fig. 3 is a diagram showing an example of data of ingredient images 1 to which group information 2 is assigned. In the example of Fig. 3, the ingredient images 1 "Image001," "Image002," "Image003," etc. are assigned group information 2 of "Lot A," and the ingredient images 1 "Image051," "Image052," "Image053," etc. are assigned group information 2 of "Lot B." The classification model 3 is trained using the ingredient images 1 to which group information 2 is assigned as training data.

[0035] The classification model creation unit 21 uses a plurality of ingredient images 1 assigned with group information 2 as training data to train a classification model 3 for classifying the ingredient images 1 into groups ("Lot A" and "Lot B"). The classification model 3 is a general supervised machine learning classification model, and in this embodiment, an example is shown in which a neural network is used as the machine learning method. However, it is not necessarily necessary to use a neural network, and other machine learning methods may also be used.

[0036] Creating a classification model 3 based on a neural network from scratch requires a large number of material images. However, when the purpose is rapid analysis in quality control, as in this embodiment, it is desirable to use transfer learning. Transfer learning is a widely used technique that partially utilizes a neural network (pre-trained model) that has been trained in advance for other image classification tasks (e.g., dogs, cats, cars, buildings, etc.). By using transfer learning, it is possible to create a classification model 3 even with a small number of material images.

[0037] The classification model verification unit 23 verifies the classification results obtained by classifying the ingredient images 1 using the created classification model 3 using a chi-square test. The classification performance of the classification model 3 is generally measured by creating a confusion matrix and using the accuracy rate, precision rate, recall rate, F-measure, etc. as indicators. However, since the purpose of this embodiment is analysis in quality control, the purpose is not necessarily to improve classification performance. Therefore, in this embodiment, from the perspective of analysis in quality control, the classification model 3 is verified using a chi-square test, which is one type of hypothesis test.

[0038] Specifically, the classification model verification unit 23 classifies the material images 1 using the classification model 3, creates a contingency table based on the classification results, and performs a chi-square test on the contingency table to determine the significance probability (see FIG. 6). This makes it possible to objectively evaluate whether there are any differences between materials in different lots.

[0039] The explanation unit 30 is a functional unit for explaining the classification basis of the ingredient image 1 according to the classification model 3. As shown in FIG. 2, the explanation unit 30 is composed of an explanation model creation unit 31, an explanation image creation unit 33, and an explanation image output unit 35.

[0040] The explanatory model creation unit 31 creates an explanatory model 4 for explaining the classification basis (basis for grouping) of the classification model 3. Specifically, the explanatory model creation unit 31 creates the explanatory model 4 using the classification model 3 and a plurality of material images 1. The purpose of the explanatory model 4 is to explain the classification results (prediction results) of the classification model 3. Therefore, when creating (learning) the explanatory model 4, it is desirable to use the same images as the material images 1 used when creating (learning) the classification model 3. However, it is sufficient if the classification results (prediction results) of the classification model 3 can be appropriately explained, and the images do not necessarily have to be exactly the same as the material images 1 used when creating the classification model 3.

[0041] The explanation model 4 is created using explainable artificial intelligence (AI) technology. Explainable AI mainly includes two technologies: one for simplifying machine learning models and one for visualizing machine learning models. In this embodiment, one of these technologies, visualization, is utilized. Known visualization technologies include LIME (Local Interpretable Model-agnostic Explanations), GradCam (Gradient-weighted Class Activation Mapping), and SHAP (SHapley Additive exPlanations). These technologies can visualize the basis for classification, such as which parts of an image a machine learning model focused on to make the classification. In particular, SHAP is widely known as an excellent example of explainable AI, and is capable of quantifying the basis for classification for each pixel. Specifically, it quantifies the impact on the classification result when a specific part of an image is hidden. This makes it possible to understand which parts of the image contribute to the classification result.

[0042] The explanatory image creation unit 33 uses the explanatory model 4 to create an explanatory image 5, which is an image that quantifies the classification basis for one or more ingredient images 1. In this embodiment, the explanatory image creation unit 33 uses the explanatory model 4 to calculate the SHAP value for each pixel of the ingredient image 1, and creates the explanatory image 5 based on the calculated SHAP value. The explanatory image 5 is an image in which the color and brightness of each pixel are changed according to the SHAP value.

[0043] Furthermore, in this embodiment, the explanatory image creation unit 33 creates explanatory images 5 for each of the two groups ("Lot A" and "Lot B"). This visualizes the basis for classification into each group ("Lot A" and "Lot B"), making it easy to visually understand the differences in the basis for classification between the two lots. An image quantifying the basis for classification into "Lot A" is referred to as explanatory image 51, and an image quantifying the basis for classification into "Lot B" is referred to as explanatory image 52. Note that it is also possible to create either one of the explanatory images 5 (51 or 52) depending on the purpose.

[0044] The explanatory image output unit 35 visualizes the explanatory image 5 created by the explanatory image creation unit 33. In this embodiment, the explanatory image output unit 35 displays, together with the material image 1, an explanatory image 51 that visualizes the basis for classification into "Lot A" and an explanatory image 52 that visualizes the basis for classification into "Lot B" on the display unit 15. Note that it is also possible to display either one of the explanatory images 5 (51 or 52) depending on the purpose.

[0045] Next, the flow of processing (material analysis support method) executed by material analysis support device 10 will be described with reference to the flowchart of Fig. 4. Each function of material analysis support device 10 is realized by control unit 11 of material analysis support device 10 executing the processing of each step in Fig. 4.

[0046] First, the control unit 11 of the material analysis support device 10 uses a plurality of material images 1 and group information 2 assigned to each of them (see FIG. 3) to create a classification model 3 for classifying the material images 1 into groups (step S1). In this embodiment, the material images 1 are images of silicon carbide powder delivered from a vendor, captured with an electron microscope and then cropped into multiple small sizes. The group information 2 is the production lot ("Lot A" and "Lot B"). The classification model 3 uses a neural network based on transfer learning.

[0047] Figure 5 is an example of program 81 that creates classification model 3, which classifies images into two groups ("Lot A" and "Lot B"). Program 81 is an example written in the programming language Python. The first line defines the use of the convolutional layer of a convolutional neural network called VGG16 for transfer learning. The second line and onward define classification model 3, which first inputs material image 1 and then runs the convolutional layer of VGG16, and then performs global average pooling on the results to reduce the features.

[0048] Next, the activation function ReLu is processed through the number of nodes previously stored in variable NODE1. After that, a process called dropout is performed to prevent overfitting, and the activation function ReLu is processed again through the number of nodes previously stored in variable NODE2. Finally, the output nodes are combined into one, and a value between 0 and 1 is output using a sigmoid function. If the output result is between 0 and 0.5, the classification result is "Lot A," and if it is between 0.5 and 1, the classification result is "Lot B."

[0049] Returning to the explanation of the flowchart in Fig. 4, the control unit 11 of the material analysis support device 10 verifies the classification result obtained by classifying the material image 1 using the classification model 3 created in step S1 using a chi-square test, and visualizes the verification result (step S2).

[0050] FIG. 6 is an example of a verification screen that visualizes the verification results of classification model 3. A contingency table is displayed on the upper side of the verification screen 91, and the results of the chi-square test are displayed on the lower side. This contingency table is the result of verifying classification model 3 by inputting 50 ingredient images 1 of "Lot A" and 50 ingredient images 1 of "Lot B," for a total of 100 ingredient images 1. From the contingency table, it can be seen that classification model 3 correctly classified 44 ingredient images 1 of "Lot A" as "Lot A," and incorrectly classified 6 as "Lot B." Similarly, it can be seen that classification model 3 correctly classified 14 ingredient images 1 of "Lot B" as "Lot B," and incorrectly classified 36 as "Lot A."

[0051] When a chi-square test, a type of hypothesis test, was performed on this contingency table, the significance probability was found to be 0.0431 (i.e., 4.31%) using the Likelihood Ratio method and 0.0455 (i.e., 4.55%) using the Pearson method. Although the performance of classification model 3 is not good, it can be seen that "Lot A" and "Lot B" are significantly different. In other words, from the perspective of quality control analysis, the multiple captured material images 1 contain information that "Lot A" and "Lot B" are different, and machine learning has detected this difference.

[0052] Returning to the explanation of the flowchart in Figure 4, the control unit 11 of the material analysis support device 10 creates an explanatory model 4 for explaining the classification basis (basis for grouping) of the classification model 3 (step S3). Specifically, the control unit 11 creates the explanatory model 4 using the classification model 3 and multiple material images 1. The explanatory model 4 is created using explainable AI technology.

[0053] FIG. 7 is an example of a program 82 for creating an explanatory model 4. The program 82 is an example written in the programming language Python. In the first line of the program 82, the file names of multiple material images 1 are read as a list, and in the second line, the program randomly samples the number of images previously stored in the variable N_DATA. In the third line, the sampled multiple material images 1 are read onto the RAM of the control unit 11, and in the fourth line, the classification model 3 (called model in the program) and the multiple material images 1 are input to create an explanatory model 4 based on SHAP. In the program 82, shap_explainer corresponds to the explanatory model 4.

[0054] Next, the control unit 11 uses the explanation model 4 to create an explanation image 5 that quantifies the classification basis for one or more ingredient images 1 (step S4). In this embodiment, the control unit 11 uses the explanation model 4 to calculate the SHAP value for each pixel of the ingredient image 1, and creates the explanation image 5 based on the calculated SHAP value. Also, in this embodiment, the control unit 11 creates an explanation image 5 (explanatory images 51, 52) that quantifies the classification basis for the ingredient images 1 for each group ("Lot A" and "Lot B").

[0055] Then, the control unit 11 visualizes the explanatory image 5 created in step S4 (step S5). In this embodiment, the control unit 11 displays, on the display unit 15, an explanatory image 51 that visualizes the basis for classification into "Lot A" and an explanatory image 52 that visualizes the basis for classification into "Lot B", together with the material image 1.

[0056] 8 and 9 show visualization examples of explanatory images 5. In the examples of FIGS. 8 and 9, two explanatory images 5 (51, 52) are displayed together with an ingredient image 1.

[0057] Figure 8 (illustration 1a) shows an example of a material image classified as "Lot A" by classification model 3. Material image 1a shows scattered silicon carbide powder. The light gray represents the powder, and the black or dark gray represents the background. Explanation image 51a for "Lot A" and explanation image 52a for "Lot B" were created by inputting material image 1a into explanation model 4. As shown in program 81, "Lot A" and "Lot B" are classified using a sigmoid function for one output node. Therefore, explanation image 51a for "Lot A" and explanation image 52a for "Lot B" differ only in that their shading is inverted. Therefore, it is sufficient to display only one of the explanatory images. The white portion of 51a indicates the basis for classification model 3's determination that material image 1a is "Lot A."

[0058] On the other hand, Figure 9, 1b, shows an example of an actual material image of "Lot B" that was classified as "Lot B" by classification model 3. Also, explanatory image 51b of "Lot A" and explanatory image 52b of "Lot B" were created by inputting material image 1b into explanatory model 4. Carefully examining explanatory images 51a, 52a, 51b, and 52b and comparing the white areas with large SHAP values, we notice that the white areas of 51a have smooth powder contours. Conversely, we notice that the white areas of 52b have sharp powder contours. From these results, we can see that "Lot A" contains a lot of smooth powder, while "Lot B" contains a lot of sharp powder, indicating that the two lots have different characteristics.

[0059] As described above, the material analysis support device 10 according to this embodiment uses multiple material images 1 and the group information 2 assigned to each of them to create a classification model 3 for classifying the material images 1. Furthermore, the classification results of the material images 1 created using the created classification model 3 are verified using a chi-square test. Furthermore, an explanatory model 4 for explaining the classification basis of the classification model 3 is used to visualize explanatory images 5 that quantify the classification basis of the material images 1. According to this embodiment, since multiple material images 1 can be classified using the classification model 3, it is possible to immediately analyze the factors that cause quality differences between groups. Furthermore, verifying the classification results of the classification model 3 using a chi-square test makes it possible to objectively evaluate whether there are any differences between the materials in the groups. Furthermore, by using the explanatory model 4 to create and visualize explanatory images 5 that quantify the classification basis of the material images 1, the classification basis of the material images 1 can be easily understood.

[0060] [Second embodiment] Next, a second embodiment of the present invention will be described. In the first embodiment, if the difference between two lots is very slight, it may not be possible to determine the difference simply by observing the explanatory image 5. Therefore, in the second embodiment, the material image 1 is displayed in divided regions, allowing the differences between the lots to be analyzed in detail.

[0061] Fig. 10 is a block diagram showing the functional configuration of a materials analysis support device 10a according to the second embodiment. In addition to the functional configuration of the materials analysis support device 10 (Fig. 2), the materials analysis support device 10a further includes a division unit 40. As shown in Fig. 10, the division unit 40 includes an area division unit 41, a counting unit 43, and a divided image output unit 45.

[0062] The region dividing unit 41 applies a region dividing process to the material image 1, dividing the material image 1 into regions (powders). When the material image 1 is imaged with an electron microscope with silicon carbide powder scattered throughout, the resulting image shows little overlap between the powder particles. In this case, the powder appears light gray in the material image 1, and the background is black or dark gray.

[0063] Next, the region dividing unit 41 performs a binarization process on the material image 1. For example, using threshold processing, pixels with a brightness value equal to or greater than a predetermined threshold are marked as white, and pixels with a brightness value less than the threshold are marked as black. This separates the background into black and the powder into white. Next, the region dividing unit 41 applies a labeling process to the resulting binary image to assign a label to each powder in the image. The labeling process assigns a different label to each region (each powder).

[0064] The region dividing unit 41 outputs region division data 6 as a result of the region division process. As shown in FIG. 11, the region division data 6 includes, for example, information on the material image 1 on which the region division process has been performed, information on labels 61 of each region (each powder) divided from the material image 1, boundary information 62 of each region (each powder), and a mask image 63. The boundary information 62 includes information such as the coordinates of the outline of each region (each powder) and the boundary box surrounding each region (the smallest rectangle surrounding each region). The mask image 63 is an image for distinguishing each divided region, and each pixel has a value of "1" if it belongs to the corresponding region and a value of "0" if it does not belong to the corresponding region. The region division process is not limited to the above-mentioned method, and other techniques such as instance segmentation may be used.

[0065] The counting unit 43 counts, for each divided region (powders), an index (SHAP value) that quantifies the classification basis using the explanatory image 5 (explanatory images 51 and 52) corresponding to the original ingredient image 1. Specifically, the counting unit 43 acquires the coordinates of the outline of the powder from the region division data 6 (boundary information 62, see FIG. 11 ) for each region (powders). The counting unit 43 then obtains the SHAP value corresponding to the coordinate of the outline of the powder from the explanatory image 5 (explanatory images 51 and 52) and calculates the sum of the SHAP values ​​for the outline of the powder. The counting unit 43 outputs counted data 7 as the SHAP value counting results. As shown in FIG. 12 , the counted data 7 includes, for example, information about the ingredient image 1 after region division processing, information about the label 61 of each region (each powder) divided from the ingredient image 1, and information about the sum 71 of the SHAP values ​​for each region (each powder) calculated based on the explanatory image 51 for "Lot A," and the sum 72 of the SHAP values ​​for each region (each powder) calculated based on the explanatory image 52 for "Lot B."

[0066] The divided image output unit 45 visualizes a divided image 9 that displays each divided area. The divided image 9 is created, for example, by combining the mask image 63 with the material image 1. Specifically, the divided image 9 is created by leaving pixels in the material image 1 that correspond to pixels with a value of "1" in the mask image 63 and setting an arbitrary background value (such as black) to pixels with a value of "0."

[0067] In this embodiment, the divided image output unit 45 visualizes the region-divided image 9 based on the counting result (counted data 7) of the counting unit 43. For example, the divided image output unit 45 targets a specific region to be visualized based on the counted data 7, and selects and displays the region-divided image 9 of that region. Specifically, by referring to the sum 71 or 72 of the SHAP values ​​in the counted data 7, the divided image output unit 45 displays the region-divided image 9 of the region (powder) with the largest SHAP value sum 71 or 72, or displays the region-divided image 9 of the region (powder) with the smallest SHAP value sum 71 or 72.

[0068] Furthermore, for example, the divided image output unit 45 sorts and displays the region divided images 9 of each region based on the aggregated data 7. Specifically, by referring to the total SHAP value 71 or 72 of the aggregated data 7, the region divided images 9 are sorted and displayed in descending order of the total SHAP value 71 or 72, or the region divided images 9 are sorted and displayed in descending order of the total SHAP value 71 or 72.

[0069] Other functional configurations (classification unit 20, explanation unit 30) of material analysis support device 10a are the same as the functional configuration of material analysis support device 10 (FIG. 2), and therefore description thereof will be omitted. Also, the hardware configuration of material analysis support device 10a is the same as the hardware configuration of material analysis support device 10 (FIG. 1).

[0070] Next, the flow of processing (materials analysis support method) executed by materials analysis support device 10a will be described with reference to the flowchart of Fig. 13. Each function of materials analysis support device 10a is realized by control unit 11 of materials analysis support device 10a executing the processing of each step in Fig. 13.

[0071] The processing of steps S1 to S5 in FIG. 13 is the same as steps S1 to S5 in FIG. 4. That is, the control unit 11 of the material analysis support device 10a uses a plurality of ingredient images 1 and group information 2 assigned to each of them (see FIG. 3) to create a classification model 3 for classifying the ingredient images 1 into groups (step S1). The control unit 11 also verifies the classification results of the ingredient images 1 classified by the created classification model 3 using a chi-square test (step S2). Next, the control unit 11 creates an explanatory model 4 for explaining the classification basis (basis for grouping) of the classification model 3 (step S3), and uses the explanatory model 4 to create an explanatory image 5 that quantifies the classification basis for one or more ingredient images 1 (step S4). The control unit 11 then visualizes the explanatory image 5 created in step S4 (step S5).

[0072] Next, in step S6, the control unit 11 applies region segmentation processing to the material image 1, dividing the material image 1 into regions (powders). For example, the control unit 11 first performs binarization processing on the material image 1, separating the background into black and the powder into white. The control unit 11 then applies labeling processing to the binary image, assigning labels to each region (powders) in the image. The labeling processing assigns a different label to each region (each powder). The control unit 11 outputs region segmentation data 6 (see FIG. 11) as a result of the region segmentation processing.

[0073] Next, the control unit 11 tally up an index (SHAP value) that quantifies the classification basis for each divided region (powders) using the explanatory image 5 (explanatory images 51, 52) that corresponds to the original ingredient image 1 (step S7). Specifically, the control unit 11 acquires the coordinates of the powder outline from the region division data 6 (boundary information 62) for each region (powders). The control unit 11 then obtains the SHAP value corresponding to the coordinates of the powder outline from the explanatory image 5 (explanatory images 51, 52) and calculates the sum of the SHAP values ​​for the powder outline. The control unit 11 outputs tally data 7 (see FIG. 12) as the tallying result of the SHAP values.

[0074] Then, the control unit 11 visualizes the region-divided image 9 based on the aggregation result (aggregated data 7) of the aggregation unit 43 (step S8). For example, the control unit 11 sets a specific region to be visualized based on the aggregated data 7, and selects and displays the region-divided image 9 of that region. Specifically, the control unit 11 refers to the sum 71 or 72 of the SHAP values ​​of the aggregated data 7, and displays the region-divided image 9 of the region (powder) with the largest SHAP value sum 71 or 72, or displays the region-divided image 9 of the region (powder) with the smallest SHAP value sum 71 or 72.

[0075] Furthermore, for example, the control unit 11 arranges and displays the area division images 9 of each area based on the aggregated data 7. Specifically, the control unit 11 refers to the total SHAP value 71 or 72 of the aggregated data 7, and arranges and displays the area division images 9 in descending order of the total SHAP value 71 or 72, or arranges and displays the area division images 9 in descending order of the total SHAP value 71 or 72.

[0076] FIG. 14 shows an example of visualization of the region-divided image 9. Figures 9a and 9b in Figure 14 show segmented images sorted in descending order of the sum of SHAP values ​​72 based on the explanatory image 52 for "Lot B." Segmented image 9a is the segmented image with the largest sum of SHAP values ​​72, i.e., the segmented image displaying the region (powder) most likely to represent "Lot B" among all the regions (powders) segmented from all the material images 1 that underwent segmentation processing. Segmented image 9b is the segmented image with the second largest sum of SHAP values ​​72, i.e., the region (powder) most likely to represent "Lot B" among all the regions (powders) segmented from all the material images 1 that underwent segmentation processing. Observing segmented images 9a and 9b reveals the presence of spiky powder, a characteristic of "Lot B." Figure 15 shows that segmented images 9a and 9b in Figure 14 are images segmented and extracted from material images 1c and 1d, respectively. Here, material images 1c and 1d are actually "Lot B," but are material images that classification model 3 mistakenly classified as "Lot A." In other words, even if classification model 3 classifies an incorrect lot, the characteristics of the actual lot can be captured by applying region segmentation processing and observing the details. For example, if the differences between different lots are slight, classification model 3 and explanation model 4 may not be able to accurately identify them. However, by observing each region (powders) in detail using region segmentation, it is possible to determine abnormalities or defects in specific regions (powders) of the material.

[0077] As described above, the material analysis support device 10a according to the second embodiment applies region segmentation processing to the material image 1, divides the material image 1 into regions (powders), and aggregates, for each divided region (powders), an index (SHAP value) that quantifies the classification basis using the explanatory images 5 (51, 52). Then, the region segmentation image 9 is visualized based on the aggregation results (aggregated data 7). For example, the region segmentation image 9 of a specific region may be selected and displayed based on the aggregation data 7, or the region segmentation images 9 of each region may be aligned and displayed based on the aggregation data 7. According to the second embodiment, by dividing the material image 1 into regions (powders) and aggregating the index (SHAP value) that quantifies the classification basis for each region (powders), the contribution of each region (each powder) to the classification into each group ("Lot A" or "Lot B") can be determined. This makes it possible to grasp the importance and differences in characteristics of each region (powders). Furthermore, by visualizing the region segmentation image 9 based on the aggregation results (aggregated data 7), it is possible to preferentially observe a specific region (powders) or evaluate quality differences in more detail. This is particularly effective when it is not possible to determine the differences between lots simply by observing explanatory image 5.

[0078] [Third embodiment] In the first and second embodiments, examples have been described in which the material images 1 are classified into two groups ("Lot A" and "Lot B"), but three or more groups may be used. In the third embodiment, an example will be described in which sintered magnets are classified into three groups ("Lot A", "Lot B", and "Lot C").

[0079] The functional configuration of the material analysis support device 10b according to the third embodiment is substantially the same as the functional configuration of the material analysis support device 10 according to the first embodiment (FIG. 2) or the functional configuration of the material analysis support device 10a according to the second embodiment (FIG. 10). The hardware configuration of the material analysis support device 10b is the same as the hardware configuration of the material analysis support devices 10 and 10a (FIG. 1).

[0080] In the third embodiment, the classification model creation unit 21 creates a classification model 3 for classifying the material images 1 into each group ("Lot A," "Lot B," "Lot C") using multiple material images 1 captured of sintered magnets and group information 2 ("Lot A," "Lot B," "Lot C") assigned to each material image 1. Furthermore, the classification model verification unit 23 verifies the classification results of the classification model 3 created by the classification model creation unit 21 using a chi-square test.

[0081] The explanatory model creation unit 31 creates an explanatory model 4 for explaining the classification basis (basis for the three groupings) of the classification model 3. Furthermore, the explanatory image creation unit 33 uses the explanatory model 4 to create explanatory images 5 that quantify the classification basis for one or more ingredient images 1. In the third embodiment, the explanatory image creation unit 33 creates explanatory images 5 that quantify the classification basis for the ingredient images 1 for each of three groups ("Lot A," "Lot B," and "Lot C"). The image that quantifies the classification basis for "Lot A" is denoted as explanatory image 51c, the image that quantifies the classification basis for "Lot B" is denoted as explanatory image 52c, and the image that quantifies the classification basis for "Lot C" is denoted as explanatory image 53c. Furthermore, the explanatory image output unit 35 visualizes the explanatory images 5 (51c, 52c, 53c) created by the explanatory image creation unit 33.

[0082] Next, the flow of processing (materials analysis support method) executed by materials analysis support device 10b will be described. The processing flow of materials analysis support device 10b is similar to the processing flow of the first embodiment (FIG. 4) or the processing flow of the second embodiment (FIG. 13), so the description will be made with reference to FIGS. 4 and 13.

[0083] First, the control unit 11 of the material analysis support device 10b uses multiple material images 1 taken of sintered magnets and the group information 2 assigned to each image ("Lot A," "Lot B," "Lot C") to create a classification model 3 for classifying the material images 1 into groups (step S1 in Figures 4 and 13).

[0084] Figure 16 is an example of program 83 that creates classification model 3, which classifies images into three groups ("Lot A," "Lot B," and "Lot C"). Program 83 is written in the programming language Python. In the first line of program 83, similar to program 81 (see Figure 5), it defines the use of the convolutional layer of a convolutional neural network called VGG16 for transfer learning. The second and subsequent lines define classification model 3, which first inputs material image 1 and then runs the convolutional layer of VGG16, and then reduces the features of the results using global average pooling processing.

[0085] Next, the activation function ReLu is processed through the number of nodes previously stored in the variable NODE1. After that, a process called dropout is performed to prevent overlearning, and the activation function ReLu is again processed through the number of nodes previously stored in the variable NODE2. Finally, the number of output nodes is set to three, and a softmax function is used so that the sum of the outputs of the three nodes is 1. The three output nodes are associated with "Lot A," "Lot B," and "Lot C," respectively, and the lot corresponding to the node with the largest output value among the three is the classification result. In the first and second embodiments, the sigmoid function was used to classify the data into two groups (see FIG. 5), but in the third embodiment, the softmax function is used to classify the data into three groups.

[0086] Returning to the explanation of the flowchart, the control unit 11 of the materials analysis support device 10b verifies the classification result obtained by classifying the material image 1 using the classification model 3 created in step S1 using a chi-square test, and visualizes the verification result (step S2 in FIGS. 4 and 13).

[0087] FIG. 17 is an example of a verification screen that visualizes the verification results of classification model 3. The verification screen 92 displays a contingency table on the upper side and the results of a chi-square test on the lower side. This contingency table is the result of verifying classification model 3 by inputting 80 ingredient images 1 from "Lot A," 80 ingredient images 1 from "Lot B," and 80 ingredient images 1 from "Lot C," for a total of 240 ingredient images 1. From the contingency table, it can be seen that classification model 3 correctly classified 80 ingredient images 1 from "Lot A" as "Lot A" and incorrectly classified none of the ingredient images 1 as "Lot B" or "Lot C." However, classification model 3 correctly classified none of the ingredient images 1 from "Lot B" as "Lot B," and incorrectly classified all 80 as "Lot A." This indicates that there is no difference between ingredient images 1 from "Lot A" and "Lot B," and that supervised machine learning was unable to classify ingredient images 1 from "Lot A" and "Lot B." On the other hand, classification model 3 correctly classified 51 images of "Lot C" material image 1 as "Lot C," and incorrectly classified 29 images as "Lot A."

[0088] When a chi-square test is performed on this contingency table, the significance probability is found to be less than 0.0001 (i.e., less than 0.01%) for both the Likelihood Ratio method and the Pearson method. In other words, it can be seen that "Lot A" and "Lot B" are similar and cannot be classified, but "Lot C" is significantly different.

[0089] Next, the control unit 11 creates an explanatory model 4 for explaining the classification basis of the classification model 3 (the basis for classification into three groups) (step S3 in FIGS. 4 and 13). Specifically, the control unit 11 creates the explanatory model 4 using the classification model 3 and multiple ingredient images 1. Next, the control unit 11 uses the explanatory model 4 to create explanatory images 5 (51c, 52c, 53c) that quantify the classification basis for one or more ingredient images 1 (step S4 in FIGS. 4 and 13). Then, the control unit 11 visualizes the explanatory images 5 (51c, 52c, 53c) created in step S4 (step S5 in FIGS. 4 and 13).

[0090] Thereafter, region division processing may be performed as necessary. That is, the control unit 11 applies region division processing to the ingredient image 1 to divide the ingredient image 1 into regions (powders) (step S6 in FIG. 13). Next, the control unit 11 counts, for each region (powders), an index (SHAP value) that quantifies the classification basis using the explanation images 5 (51c, 52c, 53c) corresponding to the original ingredient image 1 (step S7 in FIG. 13). Then, the control unit 11 visualizes the region-divided image 9 based on the counting result (counted data 7) of the counting unit 43 (step S8 in FIG. 13).

[0091] Figure 18 shows an example of a visualization of explanatory image 5. In the example shown in Figure 18, three explanatory images 5 (51c, 52c, and 53c) are displayed along with material image 1. 1e is an example of a material image in which classification model 3 classified "Lot C." The cross section of a manufactured sintered magnet was captured using an electron microscope, allowing observation of multiple magnet particles. Explanation models 4 created explanatory images 51c, 52c, and 53c by inputting material image 1e into explanation model 4. 51c is the explanatory image for "Lot A," 52c is the explanatory image for "Lot B," and 53c is the explanatory image for "Lot C." Observing explanatory image 53c, the outlines of the particles are depicted in white. In other words, the white portion of explanatory image 53c is the basis for classification model 3's identification of material image 1e as "Lot C." The white portion indicates a large SHAP value. Comparing the individual lots, only explanatory image 53c for "Lot C" had a slightly different particle shape. A detailed analysis was carried out based on the output results of the material analysis support device 10b, and it was confirmed that the sintering temperature of "Lot C" alone had fluctuated.

[0092] As described above, an example of application of the present invention when there are three groups has been described. According to the third embodiment, the same effects as those of the first and second embodiments can be obtained. Naturally, the present invention can also be applied when there are four or more groups.

[0093] While the preferred embodiments of the present invention have been described above with reference to the accompanying drawings, the present invention is not limited to these examples. It is clear that those skilled in the art can conceive of various modifications or alterations within the scope of the technical ideas disclosed herein, and it is understood that these modifications also fall within the technical scope of the present invention. [Explanation of symbols]

[0094] 1………………Material image 1a~1e……Material images 2. Group information 3. Classification model 4. Explanatory model 5………………Explanatory image 51, 51a, 51b, 51c... Lot A explanatory image 52, 52a, 52b, 52c... Lot B explanatory images 53c………………Explanatory image of Lot C 6. Area division data 7. Aggregated data 9, 9a, 9b...area segmented image 10, 10a, 10b...Material analysis support equipment 20………………Classification Department 21...Classification model creation section 23...Classification model validation section 30………………Explanation section 31...Explanatory model creation section 33...Explanatory image creation section 35...Explanatory image output section 40………………Divided part 41………………area division part 43………………Counting section 45: Divided image output section 61………………Area label 62………………Boundary information 63………………Mask image 71...................Total SHAP value based on the explanatory image of Lot A 72... Total SHAP value based on the explanatory image of Lot B 81, 82, 83...Program 91, 92...Classification model verification screen

Claims

1. a classification model creation unit that creates a classification model for classifying the ingredient images by using a plurality of ingredient images and group information assigned to each of the ingredient images; an explanatory model creation unit that creates an explanatory model for explaining a classification basis of the ingredient image according to the classification model; an explanatory image creation unit that uses the explanatory model to create an explanatory image for one or more of the ingredient images, the explanatory image quantifying the classification basis of the ingredient image; An explanatory image output unit that visualizes the explanatory image; A material analysis support device comprising:

2. the explanatory image creation unit creates an explanatory image that quantifies the classification grounds of the ingredient images for each group; The explanatory image output unit visualizes the explanatory image of each group that has been created.

2. The material analysis support device according to claim 1 .

3. A region dividing unit that divides the material image into regions; a counting unit that counts an index that quantifies a classification basis for each divided region by using the explanation image corresponding to the ingredient image; and a divided image output unit that visualizes a region divided image that displays the divided regions based on the counting result of the counting unit.

3. The material analysis support device according to claim 1 or 2.

4. The divided image output unit aligns and displays the region divided images of the respective regions based on the tallying result, or selects and displays the region divided image of a specific region based on the tallying result.

4. The material analysis support device according to claim 3.

5. The explanatory image is an image based on a SHAP value.

3. The material analysis support device according to claim 1 or 2.

6. a classification model creation unit that creates a classification model for classifying the ingredient images by using a plurality of ingredient images and group information assigned to each of the ingredient images; a classification model verification unit that verifies a classification result obtained by classifying the material images using the classification model by using a chi-square test; A material analysis support device comprising:

7. The computer creating a classification model for classifying the ingredient images using a plurality of ingredient images and group information assigned to each of the ingredient images; creating an explanation model for explaining a basis for classifying the material image by the classification model; creating an explanatory image for one or more of the ingredient images by using the explanatory model, the explanatory image quantifying the classification basis of the ingredient image; visualizing the explanatory image; A material analysis support method comprising the steps of:

8. The computer creating a classification model for classifying the ingredient images using a plurality of ingredient images and group information assigned to each of the ingredient images; A step of verifying a classification result obtained by classifying the material image by the classification model using a chi-square test; A material analysis support method comprising the steps of:

9. Computer, a classification model creation unit that creates a classification model for classifying the ingredient images by using a plurality of ingredient images and group information assigned to each of the ingredient images; an explanatory model creation unit that creates an explanatory model for explaining a basis for classifying the ingredient image by the classification model; an explanatory image creation unit that uses the explanatory model to create an explanatory image for one or more of the ingredient images, the explanatory image quantifying a classification basis of the ingredient image; an explanatory image output unit that visualizes the explanatory image; A program to function as a

10. Computer, a classification model creation unit that creates a classification model for classifying the ingredient images by using a plurality of ingredient images and group information assigned to each of the ingredient images; a classification model verification unit that verifies a classification result obtained by classifying the material images using the classification model by using a chi-square test; A program characterized by causing the program to function as a

Images