Material structure identification methods

Abstract

To provide a material structure identification method capable of identifying a plurality of kinds of structures with high accuracy in a structure observation image including the material structures.SOLUTION: A material structure identification method executes a first model creation step for creating a first model by machine learning with data of specifying an area occupied by an object structure as first teacher data in a structure observation image, a second model creation step for creating a second model by machine learning with data of specifying a boundary of dividing the object structure as second teacher data in the structure observation image, a first identification step for applying the first model to the structure observation image to identify an area occupied by the object structure, a second identification step for applying the second model to the structure observation image being the same as an object of the first identification step to identify the boundary, and an information integration step for integrating an identification result obtained in the first identification step and an identification result obtained in the second identification step to identify the area occupied by the object structure in the structure observation image.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present invention relates to a method for identifying a material structure, and more particularly, to a method for identifying a material structure in an observation image of a structure including a plurality of material structures.

Background Art

[0002] In an observation image of a structure including a plurality of types of material structures, it is important to identify and detect each material structure in order to know the state of the material. For example, the microstructure of a metal greatly affects the material properties, and an observation image obtained by a microscope such as a scanning electron microscope (SEM) is used to evaluate the structure. When a plurality of material structures such as a plurality of crystal phases are observed in those tissue observation images, the plurality of structures can be identified based on the characteristics of each material structure appearing in the image. As a method for identifying a plurality of types of material structures in an image, image processing technology has been developed. As the simplest image processing method, a method of identifying two types of material structures by binarizing a tissue observation image such as an SEM image using a luminance threshold can be mentioned.

[0003] However, when a plurality of mixed material structures do not give a significant difference as a feature on the image, such as showing adjacent luminances, or when they are mixed in a complex spatial pattern, it is often difficult to accurately identify a plurality of types of material structures using image processing techniques such as binarization. Therefore, in recent years, image analysis using machine learning has also been performed as a method that can identify material structures with higher accuracy than when using image processing techniques. For example, Patent Document 1 discloses a method of identifying a granular region and a non-granular region of a tissue image using machine learning. In the case of image analysis using machine learning, not only the luminance and color in the image but also complex information such as the shape and texture (texture) of the tissue contained in the image can be used for the analysis of the material structure.

Prior Art Documents

Patent Documents

[0004] [Patent Document 1] Japanese Patent Publication No. 2021-18752 [Overview of the project] [Problems that the invention aims to solve]

[0005] As described above, when identifying multiple material structures in microstructure observation images such as SEM images of metal structures, using machine learning-based image analysis can achieve higher accuracy in identifying material structures than using image processing based on a fixed algorithm, such as binarization. However, even when using machine learning-based image analysis, there are limitations to improving the accuracy of material structure identification. In particular, when there is a high degree of similarity in the images produced by different material structures in the microstructure observation images, or when structures that do not originate from the essence of the material structure, such as shadows that occur during image acquisition, are present in the observation images, it may not be possible to accurately identify the material structures even when applying machine learning-based image analysis.

[0006] For example, Figure 3(a) shows an SEM image of a Ni-based alloy containing γ and γ' phases. In a similar SEM image, Figure 1(a), the γ and γ' phases are identified in Image I, but the γ phase is observed to be slightly brighter than the γ' phase. However, the difference in brightness between the γ' and γ phases in the SEM image is small. Based on training data that identifies the region corresponding to the γ' phase, a machine learning model was created, and the results of image analysis performed on the SEM image in Figure 3(a) are shown in Figure 3(d). Here, the region determined to be the γ' phase is shown in white. In the areas indicated by arrows, although there are slightly brighter regions that are recognized as the γ phase in the SEM image, they are displayed in white in the image analysis results and determined to be the γ' phase. Thus, over-detection of the γ' phase is occurring.

[0007] The distribution of crystalline phases in a metal microstructure is closely related to material properties, and obtaining highly accurate information on the distribution of crystalline phases is desirable as a basis for material evaluation and development. Furthermore, when analyzing numerous observation images, it is important to suppress variations in judgment results due to non-essential factors such as the skill of the analyst and the condition of the observation images. Therefore, there is a need for the development of image analysis methods that can identify material structures with high accuracy in microstructure observation images, including those of metals and those containing multiple types of material structures.

[0008] The problem that this invention aims to solve is to provide a material structure identification method that can identify multiple types of material structures with high accuracy in a microstructure observation image that includes those structures. [Means for solving the problem]

[0009] To solve the above problems, the present invention provides a material structure identification method for identifying multiple types of material structures in a tissue observation image containing multiple types of material structures, comprising: a first model creation step of creating a first model by machine learning using data that identifies the region occupied by a target structure selected as at least one of the multiple types of material structures in the tissue observation image, distinguishing it from the region occupied by other material structures, as first training data; a second model creation step of creating a second model by machine learning using data that identifies the boundary separating the target structure from other material structures in the tissue observation image, as second training data; a first identification step of applying the first model to the tissue observation image to identify the region occupied by the target structure in the tissue observation image; a second identification step of applying the second model to the same tissue observation image that was the target of the first identification step to identify the boundary in the tissue observation image; and an information integration step of integrating the identification result obtained in the first identification step and the identification result obtained in the second identification step to identify the region occupied by the target structure in the tissue observation image.

[0010] In the information integration step, the agreement rate between the outer edge of the area identified as occupied by the target organization by the first identification step and the boundary identified by the second identification step is evaluated, and areas where the agreement rate is equal to or greater than a predetermined threshold are determined to be areas occupied by the target organization.

[0011] The microstructure observation image is preferably an electron microscope image of a metallic structure containing multiple crystalline phases. In this case, the microstructure observation image is preferably a scanning electron microscope image of a Ni-based alloy containing γ and γ' phases.

[0012] The first model and the second model may be created using a neural network. [Effects of the Invention]

[0013] In the material structure identification method according to the above invention, material structures can be identified by applying image analysis using machine learning to a tissue observation image that includes multiple types of material structures. By going through the first model creation step and the first identification step, it is possible to identify whether or not each region in the tissue observation image is the target tissue of interest. However, if this method is applied alone, there is a possibility that the identification of the target tissue may not be performed with high accuracy, especially when the target tissue and other types of tissues give similar images in the tissue observation image. However, by performing the second model creation step and the second identification step and identifying the boundary that separates the target tissue from other material structures, it is possible to obtain information on whether or not the target tissue separated by the boundary exists in each region of the tissue observation image. Therefore, in the information integration step, by integrating the information on the type of material structure obtained in the first identification step and the information on the boundary obtained in the second identification step, it becomes possible to determine whether or not the target tissue exists in each region of the tissue observation image with higher accuracy than when the information obtained in the first identification step is used alone.

[0014] In this process, the information integration step evaluates the agreement rate between the outer edge of the area identified as occupied by the target tissue in the first identification step and the boundary identified in the second identification step. If the agreement rate is above a predetermined threshold, the area is determined to be occupied by the target tissue. In this case, even if the first identification step mistakenly identifies an area that is not actually occupied by the target tissue as being occupied by the target tissue, if the area is not sufficiently clearly demarcated by the boundary identified in the second identification step, the information integration step will determine that the area is not occupied by the target tissue. Therefore, overdetection, where the target tissue is detected more than it actually is, can be effectively suppressed within the tissue observation image, and the target tissue can be identified with high accuracy.

[0015] When the microstructure observation image is an electron microscope image of a metal microstructure containing multiple types of crystalline phases, it is possible to identify the multiple crystalline phases that coexist in minute regions with high accuracy and use this as basic information for material evaluation and material development. In micrographs of metal microstructures, it may be difficult to distinguish multiple crystalline phases based on image characteristics such as brightness, texture, and shape, but phase interfaces are often clearly observed. Therefore, by performing a first identification step to identify the phases constituting each region and a second identification step to identify the phase interfaces, and then integrating the identification results in an information integration step, it is possible to identify the crystalline phases with high accuracy.

[0016] In this case, if the microstructure observation image is a scanning electron microscope image of a Ni-based alloy containing γ and γ' phases, it is difficult to distinguish the crystalline phase from information such as brightness, texture, and shape among electron microscope images of various metal structures. Therefore, the effect of improving identification accuracy by performing a second identification step in conjunction with the first identification step and integrating the identification results in the information integration step is particularly high.

[0017] When the first model and the second model are created using a neural network, it is easy to accurately perform the creation of the first model and the second model in the first model creation step and the second model creation step, the identification of the target tissue in the first identification step, and the identification of the boundary in the second identification step.

Brief Description of the Drawings

[0018] [Figure 1] It is a diagram for explaining a material tissue identification method according to an embodiment of the present invention, showing (a) the first model creation step and (b) the second model creation step. For each step, an organization observation image is shown as I, and a correct answer image is shown as II. [Figure 2] It is a diagram for explaining a material tissue identification method according to an embodiment of the present invention, showing (a) an organization observation image to be analyzed, (b) a first identification image obtained by the first identification step, (c) a second identification image obtained by the second identification step, (d) an integrated image obtained in the information integration step, and (e) a finally obtained tissue identification result. [Figure 3] It is a diagram showing each step of image analysis for Example 1. [Figure 4] It is a diagram showing each step of image analysis for Example 2.

Mode for Carrying Out the Invention

[0019] Hereinafter, a material tissue identification method according to an embodiment of the present invention will be described with reference to the drawings.

[0020] [Outline of the Material Tissue Identification Method] In the material tissue identification method according to the present embodiment, an organization observation image including a plurality of types of material tissues is taken as an analysis target, and a plurality of types of material tissues are identified in the organization observation image. That is, attention is paid to a target tissue selected as at least one of a plurality of types of material tissues, and the region occupied by the target tissue in the organization observation image is identified. For the identification of the tissue, image analysis using machine learning is used.

[0021] The tissue observation image to be used for tissue identification is not particularly limited as long as it is an image including regions corresponding to a plurality of types of material tissues. As the plurality of types of material tissues, it is only necessary that they have differences giving distinguishable features on the tissue observation image, and such differences include differences in material composition, differences in phases constituting the material such as crystal phases, or combinations of those differences. Further, as the tissue observation image, any type of observation image may be used as long as the differences in the material tissues can be identified, and examples include photographed images, various microscope observation images, and the like. The number of coexisting material tissues is also not particularly limited as long are there are two or more types. Hereinafter, as the tissue observation image, an electron microscope image of a metal tissue including a plurality of crystal phases, particularly a scanning electron microscope (SEM) image of a Ni-based alloy including γ phase and γ' phase, will be treated as an example.

[0022] In the method for material tissues according to the present embodiment, a first model creation step and a second model creation step of creating two types of machine learning trained models are performed using a tissue observation image with a known distribution of material tissues. Then, as steps of performing image analysis on a tissue observation image with an unknown distribution of material tissues using the two types of trained models obtained in those model creation steps, a first identification step and a second identification step are performed. Further, an information integration step of integrating the identification result of the first identification step and the identification result of the second identification step to finally identify the region where the target tissue is distributed in the tissue observation image is performed.

[0023] [Each step of the material tissue identification method] As described above, the material tissue identification method according to the present embodiment includes (1) a first model creation step, (2) a second model creation step, (3) a first identification step, (4) a second identification step, and (5) an information integration step. Hereinafter, each step will be described with examples.

[0024] First, when explaining each step of the material microstructure identification method according to this embodiment, we will describe the SEM image of a Ni-based alloy used as an example of a microstructure observation image. Figure 1(a)I shows the SEM image used as an example. This is an SEM image obtained by chemically mechanically polishing a Ni-based alloy and under conditions that emphasize the phase interface.

[0025] As illustrated in the image, the γ phase and γ' phase coexist in the SEM image. More specifically, the γ' phase precipitates inside the crystal grains of the γ phase. The SEM image displays the internal state of the crystal grains of the γ phase, and the γ' phase formed as precipitates within the crystal grains of the γ phase is observed as a relatively low-brightness region with a shape close to a square. On the other hand, the γ phase is observed as a high-brightness region other than where the γ' phase is formed, and includes a network-like region that occupies the space between adjacent γ' phases, and a region that extends and occupies a certain area, as indicated by the arrows in the image. The material microstructure identification method according to this embodiment identifies the region occupied by the γ' phase in the SEM image, with the γ' phase as the target microstructure.

[0026] (1) First model creation process In the first model creation process, a first model is created using machine learning to identify the region occupied by the target tissue in the tissue observation image. In other words, data identifying the region occupied by the target tissue (γ' phase) in the tissue observation image, distinguishing it from the region occupied by other material tissues (γ phase), is used as the first training data to perform machine learning and create the first model.

[0027] Figure 1(a) illustrates the first model creation process. In the tissue observation image (SEM image) I, the γ' phase can be visually distinguished from the γ phase and the phase interface. By visually identifying the regions where the γ' phase is formed in this way, training data for machine learning is created. II shows the ground truth images that constitute the training data. In the ground truth images, the regions occupied by the γ' phase are shown in white, and the regions that the analyst judges to be occupied by the γ' phase are identified as the ground truth in machine learning. Regions other than those occupied by the γ' phase, i.e., regions occupied by the γ phase, are shown in black. In this way, ground truth images in which the γ' phase is identified based on the tissue observation images are created for multiple tissue observation images. Then, the set of data that associates these multiple tissue observation images with the ground truth images, that is, the data in which the regions occupied by the γ' phase are identified in the tissue observation images, is used as the first training data.

[0028] Next, the first training data obtained above is used as training data to perform machine learning, and a trained model (semantic segmentation model) is obtained as the first model. The method used to create the first model by machine learning is not particularly limited as long as it can perform supervised learning, and publicly known models can be used, but it is preferable to use a neural network, especially a convolutional neural network (CNN), which shows high ability in image recognition and can be suitably used for analyzing two-dimensional information. The first model using a CNN is created with a tissue observation image as the input layer and an image in which the region occupied by the γ' phase in the tissue observation image is identified (an image in which the γ' phase is whited out) as the output layer.

[0029] (2) Second model creation process In the second model creation process, a second model is created using machine learning to identify the boundary that demarcates the target tissue (γ' phase) in the tissue observation image. In other words, data identifying the boundary that demarcates the target tissue (γ' phase) from other material tissues (γ phase) in the tissue observation image is used as second training data to perform machine learning and create the second model. The second model creation process is carried out independently of the first model creation process.

[0030] Figure 1(b) illustrates the second model creation process. The tissue observation image I is the same as that shown in Figure 1(a) used in the first model creation process. In the tissue observation image, the phase interface between the γ phase and the γ' phase can be recognized. By identifying the locations where these phase interfaces exist, training data for machine learning is created. Specifically, training data can be created by extracting only the contour of the target tissue from the first training data used in the first model creation process. Figure 1(b) shows the ground truth images that constitute the training data as II. In the ground truth images, the phase interfaces are shown with white lines, and areas other than the phase interfaces are shown in black. In this way, ground truth images with identified phase interfaces based on the tissue observation images are created for multiple tissue observation images. Then, the set of data that associates these multiple tissue observation images with the ground truth images, that is, the data that identifies the locations where phase interfaces exist in the tissue observation images, is used as the second training data. It is preferable to use the same tissue observation images used to create the first training data for creating the second training data. In this embodiment, the second training data is created by extracting only the contour of the target tissue from the first training data used in the first model creation process described above. Alternatively, the phase interface may be visually identified in the tissue observation image and used as the second training data.

[0031] Next, the second training data obtained above is used as training data to perform machine learning and obtain a trained model (semantic segmentation model) as the second model. The method used to create the second model by machine learning is the same as the method described above for creating the first model, and it is preferable to use a neural network, especially a CNN. The second model using a CNN is created with a tissue observation image as the input layer and an image in which the locations of phase interfaces in the tissue observation image are identified (an image in which phase interfaces are shown as lines) as the output layer.

[0032] In the example of the Ni-based alloy used in this embodiment, the only phase interface that exists is one that demarcates the target microstructure, the γ' phase, and the other type of microstructure, the γ phase. However, phase interfaces may also be formed between multiple regions formed by the target microstructure, between regions formed by other types of microstructures, and, if multiple types of other microstructures exist, between regions formed by those multiple types of other microstructures. In such cases, when creating the second training data, it is possible to identify the phase interfaces formed at each location without distinguishing between them, that is, regardless of the type of microstructure demarcated by each phase interface, or to identify only the phase interfaces that demarcate the target region and the other type of microstructure. In the former case, even if information on phase interfaces other than those demarcating the target region and the other type of microstructure is included in the second training data, it will simply not be adopted as a phase interface demarcating the target region in the information integration process described later, and will not hinder the second identification process or the information integration process.

[0033] (3) First identification step In the first identification step, the first model obtained in the first model creation step is applied to the tissue observation image to identify the region occupied by the target tissue in the tissue observation image. In other words, a tissue observation image in which the material tissue is not identified is input to the first model obtained based on a tissue observation image in which the material tissue has been identified, and as output, an image is obtained in which the region occupied by the target tissue (γ' phase) in that tissue observation image is identified.

[0034] In the example shown in Figure 2, the tissue observation image in Figure 2(a), which is a different SEM image from the one used to create the first and second models, is used for analysis. The first model is applied to this tissue observation image. As a result, the first identification image is obtained, as shown in Figure 2(b), in which the region identified as the γ' phase by the first model is colored white, and the other regions are colored black.

[0035] (4) Second identification step In the second identification step, the second model obtained in the second model creation step is applied to the same tissue observation image that was the target of the first identification step, to identify the boundaries (phase interfaces) that demarcate the target tissue in the tissue observation image. In other words, the same tissue observation image that was the target of identification in the first identification step, but where the boundaries are not identified, is input to the second model obtained based on the tissue observation image in which the boundaries have been identified. As output, an image in which the boundaries have been identified in that tissue observation image is obtained. The second identification step is performed independently of the first identification step.

[0036] In the example shown in Figure 2, the second model is applied to the tissue observation image in Figure 2(a). As a result, the output is a second identification image, as shown in Figure 2(c), in which white lines are displayed at the locations identified by the second model as having boundaries, and the other areas are colored black.

[0037] (5) Information integration process In the information integration process, the identification result obtained as the first identification image in the first identification process and the identification result obtained as the second identification image in the second identification process are integrated to identify the region occupied by the target tissue in the tissue observation image. In this embodiment, in the information integration process, the agreement rate between the outer edge of the region identified as occupied by the target tissue (γ' phase) in the first identification image and the boundary (phase interface) identified in the second identification image is evaluated, and the region in which the agreement rate is above a predetermined threshold, that is, the region surrounded by a boundary showing an agreement rate above the threshold, is determined to be the region occupied by the target tissue.

[0038] Figure 2(d) shows an integrated image obtained by superimposing the first identification image from Figure 2(b) and the second identification image from Figure 2(c). In the integrated image, the phase interface, which was shown in white in the second identification image, is shown in gray (shown in green in the color diagram). In the integrated image of Figure 2(d), the outer edge of the region identified as the γ' phase, shown in white, is generally demarcated by the gray phase interface. However, a closer look at the integrated image reveals that there are differences from region to region in whether the region identified as the γ' phase is clearly demarcated by the phase interface. For example, in region B shown in magnified image d2, a clear phase interface exists that completely encircles the outer edge of the region identified as the γ' phase, which is shown in white. In other words, the agreement rate between the outer edge of region B, which is identified as being occupied by the γ' phase, and the phase interface is high. The agreement rate in region B is almost 100%. On the other hand, in region A shown in magnified image d1, a gray phase interface is superimposed on part of the outer edge of the region identified as the γ' phase, which is shown in white. However, there are also many areas on the outer edge of region A where no phase interface exists. In other words, the agreement rate between the outer edge of region A, which is identified as being occupied by the γ' phase, and the phase interface is clearly lower compared to the case of region B in magnified image d2. The agreement rate is approximately 11%.

[0039] Regarding the above agreement rate, a threshold is set between the value in region A and the value in region B (for example, 50%). Then, in the integrated image, regions where the agreement rate is equal to or greater than that threshold are determined to be regions occupied by the γ' phase. In magnified image d1, region A has an agreement rate below the threshold, so it is determined that region A is not occupied by the γ' phase. Magnified image d1' is obtained by repainting region A in black. On the other hand, in magnified image d2, region B has an agreement rate equal to or greater than the threshold, so it is determined that region B is occupied by the γ' phase. Magnified image d2' is obtained by keeping region B as it is, indicating the γ' phase, in white. In this way, for regions identified as occupied by the γ' phase in the first discriminant image, a region is determined to be the γ' phase only if the agreement rate between the outer edge of that region and the phase interface identified in the second discriminant image is higher than the threshold. Regions with a low agreement rate, even if identified as being occupied by the γ' phase in the first discriminant image, will be re-determined as being occupied by the γ phase, meaning they are not occupied by the γ' phase.

[0040] Looking at the SEM image in Figure 2(a), we can see that the area corresponding to region B is the γ' phase, which is observed at a lower brightness than the γ phase, while the area corresponding to region A is the γ phase, although its brightness is slightly lower than the γ phase in other areas. In the case of region A, even though the γ' phase is not actually formed, the first identification step incorrectly determined that the γ' phase was formed. However, by integrating the identification result of the first identification step with the second identification result, which identifies the phase interface, and combining the information about the phase interface, a correct determination result can be obtained that region A is not occupied by the γ' phase.

[0041] Similar to what was done for regions A and B in the magnified image, the entire integrated image in Figure 2(d) was processed as follows: regions identified as the γ' phase that had a matching rate between their outer edge and the identified boundary (phase interface) above a threshold were kept white, while regions with a matching rate below the threshold were changed to black. This is the tissue identification result in Figure 2(e). In this tissue identification result image, the white regions are ultimately identified as regions determined to be occupied by the γ' phase, and the black regions are ultimately identified as regions not occupied by the γ' phase, i.e., regions occupied by the γ phase. Most of the regions that were white in the first identification image in Figure 2(b) remain white in Figure 2(e), but some of those regions have been changed to black in Figure 2(e).

[0042] [Results of material microstructure identification] In the material microstructure identification method according to this embodiment, a first identification step of identifying whether each region in the microstructure observation image is the target microstructure (γ' phase) and a second identification step of identifying the location of the boundary (phase interface) in the microstructure observation image are performed independently on the same microstructure observation image. The identification results obtained in both identification steps are then integrated in an information integration step to finally identify the region occupied by the target microstructure in the microstructure observation image. In this way, by using machine learning to integrate two different types of information obtained from the same microstructure observation image in the first and second identification steps, the accuracy of material microstructure identification can be improved compared to using only one of the pieces of information. In the example described above, as shown for region A in Figure 2(d), a region that was incorrectly identified as being occupied by the γ' phase in the first identification step, even though it is actually occupied by the γ phase, is correctly identified as a region not occupied by the γ' phase by integrating the identification results of the second identification step, thus improving the accuracy of microstructure identification.

[0043] In the first identification step, for each region in the tissue observation image, it is determined whether or not each region is occupied by the target tissue based on the image characteristics of that region, such as brightness, texture, and shape. Therefore, in cases where coexisting different material tissues, such as the γ phase and γ' phase in the example described above, give similar characteristics on the image, such as close brightness, or when non-essential structures such as shadows generated during image acquisition are superimposed on the image, or when the appearance of the image changes significantly due to changes in image acquisition conditions, it may be difficult to correctly identify the type of tissue in each region of the image, and thus the mutual identification of multiple tissues based on training data becomes difficult. Therefore, if the identification of crystalline tissue is performed based solely on the results of the first identification step, misidentification is likely to occur.

[0044] On the other hand, the second identification step identifies the boundaries between different material structures, allowing for highly accurate determination of the boundary locations. In many cases, the phase interfaces of different materials can be clearly identified, and even if non-essential structures such as shadows are superimposed on the microstructure observation image, or if the image acquisition conditions have an effect, these non-essential events rarely affect the location of the boundary itself as seen in the image. Therefore, the determination of the boundary location can be performed with higher accuracy than the identification of the type of structure occupying each region, as performed in the first identification step. In particular, in SEM images of metal structures, phase interfaces are easily captured clearly, and furthermore, the phase interfaces can be made particularly prominent depending on the pretreatment conditions such as chemical mechanical polishing and the SEM observation conditions, enabling highly accurate identification of phase interfaces. Thus, by integrating the information on material structure boundaries, which has fewer misidentifications, obtained in the second identification step, with the information on the type of material structure obtained in the first identification step, the information in the first identification step is complemented by the information in the second identification step. As a result, the accuracy of microstructure identification can be improved compared to simply adopting the identification results of the first identification step. While the information regarding the boundaries of the material structure obtained through the second identification process is highly accurate, it is difficult to identify the type of tissue occupying each region in the tissue observation image using this information alone. Although it is conceivable to identify the type of tissue based on the shape of the regions demarcated by the boundaries, even in that case, it is difficult to identify the tissue unless the boundaries demarcate completely closed regions.

[0045] In the analysis of SEM images of Ni-based alloys containing the γ and γ' phases described above, as an information integration step, the agreement rate between the outer edge of the region identified as occupied by the γ' phase in the first identification step and the phase interface identified in the second identification step was evaluated, and regions where the agreement rate was above a predetermined threshold were determined to be actually occupied by the γ' phase. In other words, only regions where both the results of the first and second identification steps indicate that it is the γ' phase were determined to be the γ' phase, and regions where only one result indicates that it is the γ' phase were not adopted as the γ' phase. By implementing the information integration step in this manner, it is possible to effectively suppress the over-detection of the γ' phase, that is, the phenomenon of detecting regions that are not actually the γ' phase as the γ' phase, which is prone to occurring as a false detection in the first identification step.

[0046] However, the information integration process is not limited to this form. The form of integration is not particularly limited as long as it integrates the identification results obtained in the first identification process with the identification results obtained in the second identification process to improve the accuracy of material structure identification. Depending on the type of tissue observation image to be targeted and the image characteristics of the material structures to be distinguished from each other, the form of information integration should be selected in a way that effectively suppresses false detection of the target tissue of interest. For example, if under-detection of the target tissue, that is, the phenomenon of not detecting an area that is actually the target tissue, is likely to occur in the first identification process, the information integration process should be carried out in such a way that all areas in which the results of at least one of the first and second identification processes suggest that the tissue is the target tissue should be adopted as the target tissue. [Examples]

[0047] The present invention will be described in more detail below using examples. Here, we confirmed whether the accuracy of material structure identification can be improved by using the material structure identification method according to the embodiment of the present invention described above.

[0048] [Test Method] After chemical mechanical polishing of the Ni-based alloy, SEM observation was performed to obtain microstructure images including the γ and γ' phases. Here, multiple microstructure images, including two types, from Example 1 and Example 2, were obtained using different sample specimens. Note that the microstructure image from Example 2 is the same as the one used in Figure 2 as an example to explain the material microstructure identification method.

[0049] As described above, using multiple tissue observation images, (1) the first model creation step and (2) the second model creation step were performed, and then image analysis was performed on each of the tissue observation images of Examples 1 and 2 using the material tissue identification method according to an embodiment of the present invention, which includes (3) the first identification step, (4) the second identification step, and (5) the information integration step. In addition, for comparison, image processing by binarization was performed on each tissue observation image.

[0050] [Test Results] Figures 3 and 4 show the images obtained at each stage of image analysis for Examples 1 and 2, respectively. (a) is the tissue observation image (SEM image). (b) is the tissue observation image in which the constituent phases were visually identified, and the areas occupied by the γ' phase are shown in white, while the other areas are shown in black. This corresponds to the first training data used in the first model creation stage and is the ground truth image that shows the correct tissue identification. (c) is the result of binarization performed for comparison. Areas observed to be lower in brightness than the threshold in the SEM image (corresponding to the γ' phase) are shown in white. (d) is the first identification image obtained by the first identification stage. (e) is the second identification image obtained by the second identification stage. (f) is the final identification image obtained by integrating the identification results of the first and second identification stages in the information integration stage. In Figures 3 and 4, for each of (b), (c), (d), and (f), the γ' area ratio obtained by the white-filled region determined to be occupied by the γ' phase is displayed below the image. Furthermore, for (c), (d), and (f), the Dice coefficient, which indicates the degree of agreement with the ground truth image in (b), is also shown.

[0051] In both Figures 3 and 4, when using binarization (c), the area fraction of the γ' phase is larger compared to the ground truth image (a). The Dice coefficient is also a small value of 0.90 or less. In other words, the γ phase cannot be accurately identified using binarization. Changing the binarization threshold did not improve the issue of the excessively large area fraction of the γ' phase.

[0052] The first discriminated image in Figure (d) is the result of machine learning-based phase type identification only. Compared to the case using binarization in Figure (c), the area ratio of the γ' phase is closer to the value in the ground truth image, and the Dice coefficient is also larger. However, the area ratio of the γ' phase is still larger than the value in the ground truth image. Looking at the image itself, there are multiple areas that do not match between the first discriminated image in Figure (d) and the ground truth image in Figure (b). Specifically, as illustrated by the arrows in Figure (d), areas that are blacked out in the ground truth image because they are judged not to be occupied by the γ' phase are whited out in the first discriminated image because they are judged to be occupied by the γ' phase. In other words, over-detection of the γ' phase is occurring, and the area ratio of the γ' phase is overestimated. Thus, while the accuracy of phase identification is improved by using machine learning-based phase identification compared to the case of binarization, false detections still occur.

[0053] The final discriminant image in Figure (f) is obtained by integrating the phase discrimination results of the first discriminant image in Figure (d) with the phase interface information of the second discriminant image in Figure (e). Comparing this final discriminant image with the ground truth image in Figure (b), the two are very similar. The region where overdetection of the γ' phase occurred, indicated by the arrow in the first discriminant image in Figure (d), is shown as a blacked-out region not occupied by the γ' phase in the final discriminant image in Figure (f), matching the ground truth image in Figure (b). The area ratio value of the γ' phase is also close to that of the ground truth image. In particular, in Example 1 in Figure 3, the area ratio value matches that of the ground truth image. Large Dice coefficients of 0.99 or 0.98 were also obtained, indicating a high degree of agreement between the final discriminant image and the ground truth image. Based on the above, it is confirmed that high accuracy can be obtained in identifying material structures by integrating the results obtained independently using machine learning—specifically, the results of identifying the material structure of each region in the first identification step and the results obtained by identifying the boundaries in the second identification step—to ultimately determine the type of material structure occupying each region in the microstructure observation image.

[0054] Although embodiments of the present invention have been described in detail above, the present invention is not limited to the above embodiments, and various modifications are possible without departing from the spirit of the invention. For example, in the above, only one type of target structure to be identified was specified (only the γ' phase in the illustrated embodiment), but in cases where three or more types of material structures coexist, two or more material structures may be specified as the target structure.

Claims

1. A method for identifying material structures in a microstructure observation image that includes multiple types of material structures, A first model creation step involves creating a first model using machine learning, with data obtained by distinguishing the region occupied by the target tissue selected as at least one of the multiple types of material tissues in the aforementioned tissue observation image from the regions occupied by other material tissues, as first training data. The process involves creating a second model using machine learning, with data identifying the boundary separating the target tissue from other material tissues in the aforementioned tissue observation image as second training data. A first identification step involves applying the first model to the tissue observation image to identify the region occupied by the target tissue in the tissue observation image, A second identification step is performed by applying the second model to the same tissue observation image that was targeted in the first identification step, thereby identifying the boundary in the tissue observation image. An information integration step is performed to integrate the identification result obtained in the first identification step and the identification result obtained in the second identification step, and to identify the region occupied by the target tissue in the tissue observation image. A material structure identification method comprising the information integration step, in which the agreement rate between the outer edge of the region identified as being occupied by the target structure by the first identification step and the boundary identified by the second identification step is evaluated, and a region in which the agreement rate is equal to or greater than a predetermined threshold is determined to be a region occupied by the target structure.

2. The method for identifying a material microstructure according to claim 1, wherein the microstructure observation image is an electron microscope image of a metal microstructure containing multiple types of crystalline phases.

3. The method for identifying the material structure according to claim 2, wherein the microstructure observation image is a scanning electron microscope image of a Ni-based alloy containing a γ phase and a γ' phase.

4. The first model and the second model are created using a neural network, the method for identifying material structures according to any one of claims 1 to 3.

Images