Material structure analysis method and analysis system

Abstract

To provide an analysis method and analysis system of a material organization which reduces the number of manual corrections in a prediction image output from machine learning.SOLUTION: An analysis method of a material organization comprises the steps of: creating an input image set by performing image pre-processing on an observation image set of the material organization; creating a teacher image set obtained by dividing each input image of the input image set into a plurality of regions; creating the first image set and the second image set different from the first image set from the input image set and the teacher image set; creating a machine learning model by using the first image set; creating a prediction image set by inputting the second image set to the machine learning model; and a prediction image evaluation step of executing evaluation with the confusion matrix between the image applied with morphology processing and the image not applied with morphology processing by creating the image applied with the morphology processing for expanding or contracting the region included in one of the prediction image and the teacher image of the second image set.SELECTED DRAWING: Figure 3A

Description

[Technical field]

[0001] The present disclosure relates to a technique for analyzing structural characteristics of a material from a cross-sectional image of the material. [Background technology]

[0002] In the development of metal and ceramic materials, research and development is being conducted to reduce material development costs by analyzing the relationship between process conditions, structural characteristics, and bulk properties. Metal and ceramic materials are assumed to be high-strength materials, heat-resistant materials, magnetic materials, dielectric materials, etc. Taking sintered magnets, which are a type of magnetic material, as an example, many sintered magnets are formed from magnetic crystal grains called the main phase and a grain boundary phase sandwiched between the main phase and the main phase. It is known that in magnet structures, the structural shape, such as the shape of the main phase and the width (thickness) of the grain boundary phase, as well as the crystal structure and composition of the main phase and the grain boundary phase, greatly influences the bulk magnetic properties. To measure the structural shape, for example, a process is required to divide an image of a magnet cross section obtained by a microscope into the main phase and the grain boundary phase. Therefore, Patent Document 1 and Patent Document 2 are known as technologies for dividing an image into the main phase and the grain boundary phase and measuring the structural shape of the main phase and the grain boundary phase.

[0003] Patent Document 1 discloses an image input process using an information processing device equipped with a processing device, a storage device, an input device, and an output device to input an image of a cross section of a sample including a main phase, which is a crystal grain, and a grain boundary phase sandwiched between the main phases; an image processing process generating from the input image a main phase image showing the main phase as an area, a grain boundary phase image showing the grain boundary phase as a line, and a triple point grain boundary phase image showing the triple point grain boundary phase, which is a part of the grain boundary phase, as an area; a main phase analysis process labeling each main phase in the main phase image, analyzing the structure and shape of each main phase, and generating a main phase database in a format corresponding to the labels of the main phase; the grain boundary phase database in a format corresponding to the label of the triple point grain boundary phase; a triple point grain boundary phase analysis step of labeling each triple point grain boundary phase in the triple point grain boundary phase image, analyzing the microstructural shape of each triple point grain boundary phase, and generating a triple point grain boundary phase database in a format corresponding to the label of the triple point grain boundary phase; an adjacency relationship analysis step of generating an adjacency relationship database indicating the positional relationships between a main phase, a grain boundary phase, and a triple point grain boundary phase; and a database construction step of constructing the main phase database, the grain boundary phase database, the triple point grain boundary phase database, and the adjacency relationship database as mutually referable databases.

[0004] Patent Document 2 describes a method of correcting the teacher label image by comparing the discrimination label image 14 with the teacher label image 12, assuming that the discrimination label image 14 has less variation in the boundary than the teacher label image 12. As a method of comparing the discrimination label image 14 with the teacher label image 12, a confusion matrix for two classes (positive, negative) classification shown in Fig. 11 is exemplified. It describes that FN (False Negative) and FP (False Positive) are expressed by the total number of pixels with different labels. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Patent Publication No. 2019-215788 [Patent Document 2] WO2020 / 031243A1 Summary of the Invention [Problem to be solved by the invention]

[0006] To analyze the relationship between process conditions, microstructure, and bulk properties from images of magnet cross sections taken with a microscope, it is necessary to use images taken with a wide field of view. Since the width of the grain boundary phase is non-uniform, narrow grain boundary phases are not visualized in images taken with a wide field of view, and the grain boundary phase is interrupted in the image. Therefore, the interruption of the grain boundary phase is manually corrected using paint software before measuring the shapes of the main phase and grain boundary phase. This manual correction significantly reduces the TAT (Turn Around Time) of microstructure measurement, so there is a demand to reduce the number of manual corrections. In particular, when introducing machine learning to analyze microstructure, it is necessary to process many images, so it is important to reduce the number of manual corrections.

[0007] Patent Document 1 describes a method of manually correcting grain boundary phases that could not be extracted by image processing, and a method of creating a main phase image 305, a grain boundary phase image 304, and a triple point grain boundary phase image 306 from an SEM image 301 by machine learning.

[0008] However, Patent Document 1 does not describe reducing the number of manual corrections in a predicted image output from machine learning. Patent Document 2 also does not describe reducing the number of manual corrections in a predicted image output from machine learning.

[0009] The objective of the present disclosure is to analyze material structures while reducing the number of manual corrections in predicted images output from machine learning. [Means for solving the problem]

[0010] A preferred aspect of the present disclosure is a method for analyzing a material structure using an information processing device including a processing device, a storage device, an input device, and an output device, the method comprising: an input image set creation step of creating an input image set by performing image preprocessing on the material structure observation image set; a teacher image set creation step of creating a teacher image set by dividing each input image of the input image set into a plurality of regions; creating a first image set and a second image set different from the first image set from the input image set and the teacher image set; A machine learning model creation step of creating a machine learning model using the first image set; a predicted image set generation step of inputting the second image set into the machine learning model to generate a predicted image set; The method for analyzing material structure includes a predicted image evaluation step of creating a morphologically processed image that expands or contracts an area contained in one of the predicted images or teacher images of the second image set, and performing an evaluation using a confusion matrix between the morphologically processed image and a non-morphological image. Effect of the Invention

[0011] According to the present disclosure, it is possible to analyze material structure by reducing the number of manual corrections in predicted images output from machine learning. [Brief description of the drawings]

[0012] [Figure 1A] FIG. [Figure 1B] FIG. [Figure 1C] FIG. 13 is a diagram showing a false negative image created from a training image and a predicted image. [Figure 1D] FIG. 13 shows a false positive image created from a training image and a predicted image. [Figure 1E] FIG. 13 is a diagram showing an expanded predicted image obtained by expanding the grain boundary phase of the predicted image. [Figure 1F] FIG. 13 is a diagram showing an expanded teacher image obtained by expanding the grain boundary phase of the teacher image. [Figure 1G] FIG. 13 is a diagram showing a false negative image created from a teacher image and a dilation predicted image. [Figure 1H]A figure showing a false positive image created from a predicted image and an expanded teacher image. [Diagram 2] Illustrative diagram of a confusion matrix. [Figure 3A] Flowchart for creating a machine learning model. [Figure 3B] Flowchart for measuring tissue shape using a machine learning model. [Figure 4A] 13 is a flowchart of an input image set creation process. [Figure 4B] 13 is a flowchart of a teacher image set creation process. [Figure 4C] Flowchart of the machine learning model creation process. [Figure 4D] 13 is a flowchart of a predicted image set creation process. [Figure 4E] 13 is a flowchart of a predicted image evaluation step. [Figure 4F] 13 is a flowchart of a process for creating an image set for measurement. [Figure 5A] FIG. 1 shows an observation image taken by SEM. [Figure 5B] FIG. 4 is a diagram showing an input image obtained by performing a first image pre-processing on an observed image. [Figure 6A] Image intensity histogram of the observed image. [Figure 6B] Image intensity histogram of the observed image. [Figure 7A] FIG. [Figure 7B] FIG. 13 is a diagram showing a binary image obtained by dividing an input image into regions using a threshold method. [Figure 7C] FIG. 13 is a diagram showing an expanded binary image obtained by expanding a binary image. [Figure 7D] FIG. 13 is a diagram showing a corrected image obtained by correcting a dilated binary image. [Figure 7E] FIG. 13 is a diagram showing a thin-line image obtained by thinning a corrected image. [Figure 7F] FIG. 13 is a diagram showing a logical sum image of a binary image and a thin line image. [Figure 7G] FIG. 13 is a diagram showing a thick line image obtained by dilating a thin line image. [Figure 7H] FIG. 13 is a diagram showing a logical sum image of a binary image and a thick line image. [Figure 7I]FIG. 13 shows a corrected image in which isolated grain boundary phases have been removed. [Figure 8A] 13 is a first example of a display screen for setting preprocessing conditions for evaluation. [Figure 8B] 2 shows a second example of a display screen for setting preprocessing conditions for evaluation. [Figure 9A] Conditions for creating input images and teacher images. [Figure 9B] Evaluation results of predicted images under each condition. [Figure 10A] FIG. [Figure 10B] FIG. [Figure 10C] FIG. 13 is a diagram showing a corrected image obtained by correcting a predicted image. [Figure 10D] FIG. 13 is a diagram showing a thin-line image obtained by thinning a corrected image. [Figure 10E] FIG. 13 is a diagram showing a binary image obtained by dividing an input image into regions using a threshold method. [Figure 10F] FIG. 13 is a diagram showing a measurement image created by the logical sum of a binary image and a thin line image. [Figure 11] FIG. 1 is a block diagram showing an overall configuration of an information processing apparatus according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] The following examples are described in detail with reference to the drawings. However, the present disclosure is not to be interpreted as being limited to the description of the following examples. It is easily understood by those skilled in the art that the specific configurations can be changed without departing from the concept or spirit of the present disclosure.

[0014] In addition, in the configuration of the invention described below, the same reference numerals are used in common between different drawings for the same parts or parts having similar functions, and duplicated descriptions may be omitted. The notations "first", "second", "third" and the like in this specification are given to identify components, and do not necessarily limit the number, order, or contents. In addition, numbers for identifying components are used in each context, and a number used in one context does not necessarily indicate the same configuration in another context. In addition, a component identified by a certain number does not prevent the component from also having the function of a component identified by another number. The position, size, shape, range, etc. of each component shown in the drawings, etc. may not represent the actual position, size, shape, range, etc. in order to facilitate understanding of the invention. For this reason, the present disclosure is not necessarily limited to the position, size, shape, range, etc. disclosed in the drawings, etc. EXAMPLES

[0015] In Example 1, an information processing device including a processing device, a storage device, an input device, and an output device was used to perform scanning electron microscopy ( S canning E lectron M We will explain the details of the method for measuring tissue shape from CT images. We are investigating the use of machine learning for segmentation to reduce the number of manual corrections.

[0016] First, a flowchart in this embodiment when machine learning is introduced into region segmentation is shown in FIG. 3. FIG. 3A is a flowchart for creating a machine learning model. An input image set is created from a set of observed images of a cross section (S1), and a set of teacher images is created by region segmenting the input image (S2). A first image set for creating a machine learning model and a second image set for evaluating the machine learning model are created from the input image set and the teacher image set (S3). A machine learning model is created using the input image set and teacher image set of the first image set (S4). An input image of the second image set is input to the created machine learning model, and a predicted image is created (S5). The predicted image is evaluated from the predicted image and the teacher image (S6).

[0017] Figure 3B is a flowchart for measuring the microstructural shape using a machine learning model. A set of input images for measuring the microstructural shape is created (S1), and the input images are input into the machine learning model to create a predicted image set (S5). Compared to an image segmented using general-purpose image processing, the predicted image contains significantly less discontinuity in the grain boundary phase, but still contains some areas that require manual correction. A measurement image set is created by manually correcting these images (S7), and the microstructural shape is measured (S8).

[0018] The evaluation of a predicted image is generally performed using a confusion matrix as shown in Fig. 2. The problems that arise when evaluating a predicted image based on a confusion matrix will be explained with reference to Fig. 1.

[0019] Fig. 1A is a diagram showing a teacher image, in which the main phase (10) is shown as negative and the grain boundary phase (20) is shown as positive, and Fig. 1B is a diagram showing a predicted image.

[0020] When the predicted image is evaluated using the confusion matrix, pixels that are grain boundary phase in the teacher image and grain boundary phase in the predicted image are classified as true positives (33), pixels that are main phase in the teacher image and main phase in the predicted image are classified as true negatives (34), pixels that are grain boundary phase in the teacher image and main phase in the predicted image are classified as false negatives (31), and pixels that are main phase in the teacher image and grain boundary phase in the predicted image are classified as false positives (32).

[0021] A false negative image created using the conventional method is shown in Figure 1C, and a false positive image is shown in Figure 1D. All false negatives (35) and false positives (36) caused by a misalignment of about one pixel at the boundary between the main phase and grain boundary phase are detected. However, since the shape of the main phase and the width of the grain boundary phase are measured in the structure shape measurement process (S8), only interrupted areas of the grain boundary phase (31) caused by false negatives and grain boundary phases that have occurred within the main phase due to false positives (32) are corrected in the measurement image set creation process (S7).

[0022] A boundary misalignment of about one pixel has little effect on the particle shape measurement results, and there is no need to correct false negatives (35) and false positives (36) caused by boundary misalignment. In a sample with a narrow grain boundary phase, the number of false negative (35) and false positive (36) pixels that do not require correction due to boundary misalignment will be greater than the number of false negative (31) and false positive (32) pixels that must be corrected.

[0023] In other words, without using this embodiment, it is impossible to evaluate the increase or decrease in false negatives (31) and false positives (32) that require correction. Therefore, even if measures to reduce manual corrections are implemented, the effectiveness of the measures cannot be quantitatively evaluated, and the most effective measures cannot be selected.

[0024] The details of the predicted image evaluation step (S6) will be described with reference to FIG. 1. Since the main phase (10) is shown as negative and the grain boundary phase (20) is shown as positive, the predicted image is expanded to create an expanded predicted image (FIG. 1E) in which the grain boundary phase region is expanded. FIG. 1G shows a false negative image created from the teacher image (FIG. 1A) and the expanded predicted image (FIG. 1E). False negatives (35) that do not require correction due to boundary misalignment are not extracted, and false negatives (31) that should be corrected are extracted. FIG. 1F shows an expanded teacher image in which the teacher image is expanded to expand the grain boundary phase region. FIG. 1H shows a false positive image created from the predicted image (FIG. 1B) and the expanded teacher image (FIG. 1F). False positives (36) that do not require correction are not extracted, and false positives (32) that should be corrected are extracted. The present disclosure makes it possible to quantify false negatives (31) and false positives (32) that require correction.

[0025] 11 is a block diagram showing an overall configuration of an information processing apparatus 1100 according to this embodiment. The information processing apparatus 1100 includes a processing device 1101, a storage device 1102, an input device 1103, and an output device 1104.

[0026] The processing device 1101 has an input image set creation unit, a teacher image set creation unit, a first image set creation unit, a second image set creation unit, a machine learning model creation unit, a predicted image set creation unit, a predicted image evaluation unit, a measurement image set creation unit, and a tissue shape measurement unit, which will be described later.

[0027] For example, the information processing apparatus 1100 uses a server as a processing device 1101 having a processor. In this embodiment, functions such as calculation and control are realized by the processing device 1101 reading a program stored in the storage device 1102 and executing the functions of the program, and collaborating with other hardware to perform the defined processing. A program executed by a computer or the like, its function, or means for realizing the function may be referred to as "function", "means", "section", "unit", "module", etc. For convenience, these functions are shown as functional blocks in the processing device 1101 in FIG. 11.

[0028] The above configuration may be composed of a single computer, or any part of the input device 1103, output device 1104, processing device 1101, and storage device 1102 may be composed of other computers connected by a network. The hardware configuration of the server and the like is well known and will not be particularly illustrated here.

[0029] According to this embodiment, the T.A.T. of the measurement image set creation step (S7) for removing false negatives and false positives included in the prediction image by manual correction, which is performed before measuring the tissue shape (S8), can be improved. Therefore, this embodiment can reduce the number of false negatives and false positives to be corrected. False negatives and false positives included in the prediction image change depending on the input image set creation step (S1), teacher image set creation step (S2), and machine learning model creation step (S4). By using the prediction image evaluation step (S6) according to this embodiment, false negatives and false positives that require correction can be quantified. It becomes possible to compare and examine which step was performed and which countermeasure is effective, and to select an optimal countermeasure. The details of each step will be described below.

[0030] <S1. Input Image Set Creation Step> The input image set creation step (S1) is assumed to be executed by the input image set creation unit of the processing device 1101 in the information processing apparatus 1100.

[0031] A flowchart of the input image set creation step (S1) is shown in Fig. 4A. It includes a step of taking an observation image set (S11), an image pre-processing step (S12), and a step of outputting the input image set (S13). The step of outputting the input image set (S13) includes a step of storing the input image set in the storage device 1102 for the next step.

[0032] The details of the step (S11) of taking a set of observation images are shown below. First, the cross section of the sample is exposed in order to take an SEM image. Since functional materials often have anisotropic sample structures, it is better to process the sample so that the surface orientation of the sample can be confirmed. For example, in anisotropic Nd-Fe-B magnets, the axis of easy magnetization is often called the c-axis. In this case, we will be taking an image of the surface perpendicular to the c-axis (often called the ab-plane), so we cut the sample into a rectangular parallelepiped with the ab-plane facing up.

[0033] Furthermore, if the cross section being photographed is uneven, image contrast dependent on the unevenness will be observed in the SEM image, so it is better to polish the surface to be photographed flat before photographing. A focused ion beam processing device or cross-section polishing device can be used for flattening. Insert the processed sample into the SEM, set the accelerating voltage, lens mode, and detection signal, and adjust the detector amplifier, axial misalignment, astigmatism, and focus of the electron optical system according to the adjustment procedure manual for the device. If the SEM is equipped with an automatic adjustment function for the detector and electron optical system, you can also use this automatic adjustment function.

[0034] When automatically photographing multiple fields of view, it is better to align the sample stage. After adjusting the SEM, take the SEM images required for analysis and save them as observation images on a recording medium.

[0035] The step (S11) of photographing an observation image set is not limited to being executed by the processing device 1101 in the information processing device 1100, but may be executed by a device other than the information processing device 1100. In that case, data of the observation image is input from an input device 1103 in the information processing device 1100 and stored in the storage device 1102. Then, the stored data of the observation image is read out and proceeds to an image pre-processing step (S12).

[0036] Details of the image pre-processing step (S12) are given below. Many images are processed in the analysis of material structure. Differences occur between observed images due to the sample processing method, the SEM used for shooting, shooting magnification, selection of detection signal, adjustment of detector amplifier, etc. Input images are created in which these differences are reduced by image pre-processing.

[0037] First, let us consider the difference in magnification between observed images. The shooting magnification of an SEM is a value calibrated using a standard sample, and there may be differences in the magnification settings between instruments. It is best to shoot an image set with the same SEM. Even if images are shot with different SEMs, the magnification within the image set can be unified by shooting the same field of view with each instrument, measuring the magnification error between the instruments, and enlarging or reducing the observed images.

[0038] Next, we will discuss the difference in image contrast between observed images. Image contrast changes depending on the selection of detection signal. In this case, we divided the region into the main phase and the grain boundary phase, so we selected backscattered electrons, which increase the difference in image intensity between the main phase and the grain boundary phase.

[0039] An observation image formed by backscattered electrons and taken by SEM is shown in Figure 5A. In the grain boundary phase, a two-particle grain boundary phase 21 sandwiched between two grains and a triple-junction grain boundary phase (22, 23) surrounded by multiple main phases are observed. It can be seen that the width of the two-particle grain boundary phase 21 is narrow, and there is a two-particle grain boundary phase that cannot be visualized at the imaging magnification.

[0040] The image contrast of triple junction grain boundary phases (22, 23) reveals that there are multiple types of triple junction grain boundary phases. Since the type of triple junction grain boundary phase is structural information that affects process conditions and bulk properties, images are usually taken with a SEM formed by backscattered electrons, which allows the type of triple junction grain boundary phase to be identified.

[0041] Conventionally, the image in Fig. 5A was used as the input image, and the machine learning model creation step (S4) and predicted image set creation step (S5) described below were carried out. This time, after considering various image preprocessing methods, the image in Fig. 5B, which was created by enhancing the image contrast of the observed image as image preprocessing in Fig. 5A, was used as the input image. It was found that using the image in Fig. 5B as the input image reduced false negatives and false positives in the predicted image evaluation step (S6) described below.

[0042] A method for enhancing image contrast will be described with reference to FIG. 6A and FIG. 6B. FIG. 6A is an image intensity histogram of an observed image. The most frequent value M1 of the image intensity of the main phase and the most frequent value M2 of the image intensity of the grain boundary phase are measured from the image intensity histogram. It is also possible to automate the measurement of M1 and M2 by writing a script that specifies the range of the image intensity of the main phase as a first image intensity range (51), extracts the most frequent value M1 within the first image intensity range 51, specifies the range of the image intensity of the grain boundary phase as a second image intensity range (52), and extracts the most frequent value M2 within the second image intensity range 52. The image intensity is normalized so that the values ​​of M1 and M2 are the specified values. For example, if M1=60 and M2=130 are measured in an observed image captured in 256 gradations, the values ​​are normalized to M1=120 and M2=1024, and then the intensities below 0 are converted to 0 and the intensities above 255 are converted to 255.

[0043] Since there are multiple types of grain boundary phases, it is necessary to align the types of grain boundary phases used for normalization within the image set. Depending on the image, the ratio of the grain boundary phase with the image intensity of M2 may be low, and a peak may not be formed in the image intensity histogram (Fig. 6B). In that case, the image intensity range of the main phase is specified as the first image intensity range (51), Gaussian function fitting is performed within the first specified range 51, and the average A and the half-width σ of the image intensity of the main phase are measured. For example, when measured as A = 60 and σ = 6 in an observation image taken with 256 gradations, after normalizing the image intensity so that A = 127 and σ = 200, intensities of 0 or less are converted to 0, and intensities of 255 or more are converted to 255. Although the half-width was used for the explanation, the full-width may also be used. Not limited to the half-width, a value indicating the relationship between the peak value and the width of the waveform in Fig. 6B can be used, and here, the width of a specific location of the waveform will be referred to as the variance σ.

[0044] Note that the conditions for image contrast enhancement are not limited to the above conditions. Also, when white noise is enhanced by enhancing the image contrast, it may be smoothed with a Gaussian filter having a half-width of about 2 after the image contrast is enhanced. The type of filter used for smoothing is not limited to the above. Apply various image contrast enhancement conditions and smoothing conditions, and adopt the conditions under which false negatives and false positives decrease in the prediction image evaluation step (S6) described later.

[0045] Observing Fig. 5B, it can be seen that the curtaining 4 is visualized by enhancing the image contrast. In ion beam processing, when a processing beam is irradiated from one direction, grooves substantially parallel to the processing beam are formed due to the ion beam curtain effect. This groove is called curtaining. Measures were studied to suppress the misdetection of the curtaining 4 as a two-particle grain boundary phase. As a result, it was found in the prediction image evaluation step (S6) described later that a countermeasure including a step of rotating each observation image so that the directions of the grooves formed by the ion beam curtain effect during the production of the observation surface using the ion beam are aligned among the input image sets is effective.

[0046] <S2. Teacher Image Set Creation Step> The teacher image set creating step (S2) is executed by a teacher image set creating unit of the processing device 1101 in the information processing device 1100.

[0047] A flowchart of the teacher image set creation step (S2) is shown in Fig. 4B. This step includes a step (S21) of inputting the first input image set from the step (S13) of outputting the input image set, a step (S22) of dividing the area using a threshold method, a first correction processing step (S23), and a step (S24) of outputting the teacher image set.

[0048] The teacher image set creation step (S2) will be described in detail using an input image including a main phase 10 and a grain boundary phase 20 shown in Fig. 7A as an example. First, a binary image (Fig. 7B) is created (S22) in which the regions are divided into the grain boundary phase (positive) and the main phase (negative) using general-purpose image processing such as a threshold method. With general-purpose image processing, a partially interrupted grain boundary phase (24) and an undetectable grain boundary phase (25) are observed.

[0049] First, the grain boundary phase in the binary image (Figure 7B) is expanded by a few pixels to create an expanded binary image (Figure 7C), and the gaps of a few pixels are joined. Next, the grain boundary phase in the expanded binary image that could not be detected is manually corrected using paint software to create a corrected image (Figure 7D).

[0050] In manual correction, a layer display in which the input image and the binary image are superimposed is used. In manual correction, correction is performed so that the minimum value of the grain boundary phase width is a few pixels. The corrected image (FIG. 7D) corrected by the user is input from the input device 1103 of the information processing device 1100 and stored in the storage device 1102.

[0051] Figure 7E is a thin-line image obtained by thinning the corrected image (Figure 7D). The logical sum image of the binary image (Figure 7B) and the thin-line image (Figure 7E) is shown in Figure 7F. In Figure 7F, the minimum width of the grain boundary phase is one pixel.

[0052] The thick-line image, which is a result of expanding the thin-line image (FIG. 7E) by a few pixels, is shown in FIG. 7G. The logical sum image of the thick-line image (FIG. 7G) and the binary image (FIG. 7B) is shown in FIG. 7H.

[0053] In FIG. 7H, the minimum value of the width of the grain boundary phase is several pixels. A teacher image set created under the first condition where the minimum value of the width of the grain boundary phase is 1 pixel and a teacher image set created under the second condition where the minimum value of the width of the grain boundary phase is several pixels were created, a machine learning model was created for each, and as a result of performing the prediction image evaluation step (S6) described later, since false negatives and false positives decreased when creating the teacher image under the second condition, the second condition was adopted as the creation condition of the teacher image.

[0054] Also, whether or not to classify the grain boundary phase 26 isolated within the main phase in FIG. 7D as a grain boundary phase depends on the content of the microstructure shape measurement (S8) described later. When the grain boundary phase 26 isolated within the main phase is not classified as a grain boundary phase, a corrected image (FIG. 7I) with the isolated grain boundary phase removed is created. The corrected image (FIG. 7I) with the isolated grain boundary phase removed may be manually corrected by the user or may be automatically corrected to remove the isolated grain boundary phase.

[0055] Create a logical product image of the logical sum image (FIG. 7H) of the binary image and the thick line (thin line) image and the corrected image (FIG. 7I), and use it as the teacher image.

[0056] By adding the step of calculating the logical sum image with the corrected image, it becomes possible to adjust the minimum value of the grain boundary phase width even for the teacher image with the grain boundary phase 26 isolated within the main phase removed. Note that the creation conditions of the teacher image are not limited to the above. Apply various creation conditions and adopt the conditions under which false negatives and false positives decrease in the prediction image evaluation step (S6) described later.

[0057] <S3. Step of creating two image sets from the input image set and the teacher image set> The image set used for creating the machine learning model and the image set used for evaluating the machine learning model need to be separated as different image sets from each other. Therefore, a step of creating a first image set for creating a machine learning model from the created input image set and teacher image set is executed. Also, a step of creating a second image set for evaluating the machine learning model from the created input image set and teacher image set is executed. In this step, the created input image set and teacher image set will be separated into a first image set for creating a machine learning model and a second image set for evaluating the machine learning model used in the predicted image evaluation step.

[0058] The first image set creation step and the second image set creation step shall be executed by the first image set creation unit and the second image set creation unit of the processing device 1101 in the information processing device 1100, respectively.

[0059] <S4. Machine Learning Model Creation Step> The machine learning model creation step (S4) shall be executed by the machine learning model creation unit of the processing device 1101 in the information processing device 1100.

[0060] The flowchart of the machine learning model creation step (S4) is shown in FIG. 4C. The machine learning model creation step (S4) includes a step (S41) of inputting the input image set and teacher image set of the first image set, a step (S42) of creating a machine learning model, and a step (S43) of outputting and creating a machine learning model.

[0061] As a machine learning model suitable for region segmentation, a fully convolutional neural network is known. In the input layer, i.e., the first half of the network, global features are captured while repeatedly performing feature extraction by convolution and image reduction by pooling from a wide area to local features. In the second half of the network, i.e., the output layer, the feature space is gradually expanded (restored) from local features to the original size, and finally, the discrimination result for each pixel unit in the original image is output. Various algorithms have been proposed by changing the total number and types of layers of the neural network. In this embodiment, U-NET is adopted, but other neural networks may also be used.

[0062] An input image set and a teacher image set are associated and input to the machine learning model, and parameter adjustment within the machine learning model is executed so that the predicted image output from the machine learning model matches the teacher image set. Eight sets of images were used to create the machine learning model. Although the number of pixels in the input image and the teacher image is 1280×1280 pixels, using an image with a large pixel size as it is for creating the machine learning model would result in insufficient memory on the computer. Therefore, each image was cut into 256×256 pixel images for use.

[0063] <S5. Prediction Image Set Creation Step> The prediction image set creation step (S5) shall be executed by the prediction image set creation unit of the processing device 1101 in the information processing device 1100.

[0064] The flowchart of the prediction image set creation step (S5) is shown in FIG. 4D. It includes a step (S51) of inputting the input image set of the second image set into the machine learning model, a region segmentation step (S52) by the machine learning model, and a step (S53) of outputting the prediction image set.

[0065] In order to input the input image set into the machine learning model, the input images are segmented. When segmenting the input images, it is advisable to save the input images with a suffix such as _X1Y1 in the file name to identify the position of the image in the original image. When combining the predicted images output from the machine learning model and returning them to the original image size, the images are combined by referring to the suffix attached to the file name.

[0066] <S6. Predicted Image Evaluation Step> The predicted image evaluation step (S6) shall be executed by the predicted image evaluation unit of the processing device 1101 in the information processing device 1100.

[0067] The predicted image evaluation step (S6) is a step devised to quantify false negatives (31) and false positives (32) that require manual correction in the measurement image set creation step (S7) described later. In the conventional method, that is, when creating false negative images (Fig. 1C) and false positive images (Fig. 1D) using the teacher image (Fig. 1A) and the predicted image (Fig. 1B), all false negatives (35) and false positives (36) caused by a positional shift of about one pixel at the boundary between the matrix phase and the grain boundary phase are also detected.

[0068] In order not to detect false negatives (35) and false positives (36) that do not require correction, first, the false negative image and the false negative image are interpreted as follows. It is determined in the predicted image that the grain boundary phase in the teacher image becomes the grain boundary phase in the predicted image, and a false negative image is created. In order not to detect false negatives due to the positional shift of the boundary, the predicted image (Fig. 1B) used for the determination is changed to an expanded predicted image (Fig. 1E) in which the grain boundary phase of the predicted image (Fig. 1B) is expanded. By this change, a false negative image in which false negatives (31) that require correction are extracted is created (Fig. 1G). Similarly, it is determined in the teacher image that the grain boundary phase in the predicted image becomes the grain boundary phase in the teacher image, and a false positive image is created. In order not to detect false positives due to the positional shift of the boundary, the teacher image (Fig. 1A) used for the determination is changed to an expanded teacher image (Fig. 1F) in which the grain boundary phase is expanded. By this change, a false positive image in which false positives (32) that require correction are extracted is created (Fig. 1H).

[0069] A flowchart of the predicted image evaluation step (S6) is shown in Fig. 4E. It includes a step of inputting the teacher image set and the predicted image set of the second image set (S61), a preprocessing step for evaluation (S62), a step of creating false negative images and false positive images (S63), a step of quantifying the false negatives and false positives (S64), and a step of outputting the quantitative values ​​of the false negatives and false positives (S65).

[0070] A first example display screen showing images processed in each process is shown in Fig. 8A. The input teacher image and predicted image are confirmed (S61). Since the main phase (10) is shown as negative and the grain boundary phase (20) is shown as positive, an expanded predicted image is created by expanding the grain boundary phase of the predicted image, and an expanded teacher image is created by expanding the grain boundary phase of the teacher image, and these are confirmed on the display screen (S62). The number of pixels to be expanded is adjusted by the user depending on the tolerance for the deviation of the boundary between the main phase and the grain boundary phase.

[0071] A false negative image is created using the teacher image and the expanded predicted image, and a false positive image is created using the predicted image and the expanded teacher image, and these are displayed on the screen (S63). A false negative image is created by performing an evaluation operation according to a confusion matrix between the teacher image and the expanded predicted image on a pixel-by-pixel basis. A false positive image is created by performing an evaluation operation according to a confusion matrix between the predicted image and the expanded teacher image on a pixel-by-pixel basis.

[0072] It is confirmed that false negatives (31) and false positives (32) that require correction have been extracted from the false negative and false positive images. The area (number of pixels) and number of false negative and false positive regions are measured and displayed on a display device, which is the output device 1104 of the information processing device 1100 (S64). When evaluating a predicted image using a confusion matrix, evaluation is generally based on area, but in order to estimate manual work time, it is better to evaluate based on the number of regions.

[0073] FIG. 8B shows a display screen of a second example when the same predicted image evaluation process is performed with a dilation pixel number of 1. When the dilation pixel number is 1, it includes a false negative region (35) and a false positive region (36) that do not require correction in the false negative image and the false positive image, and it can be seen that the dilation pixel number needs to be increased.

[0074] The predicted image evaluation results change depending on the input image set creation step (S1), the teacher image set creation step (S2), and the machine learning model creation step (S4). In this embodiment, by quantifying false negatives and false positives that require correction, it becomes possible to compare and examine which step was performed and which countermeasures are effective, and it becomes possible to select an optimal countermeasure. When manual correction is reduced by countermeasures, the tissue shape analysis T.A.T. is improved, and the efficiency of analyzing the relationship between process conditions, tissue shape, and bulk characteristics is improved.

[0075] An example of a countermeasure for reducing the number of manual corrections is shown. FIG. 9A shows four conditions in which the minimum grain boundary width of the teacher image set, the image contrast enhancement condition of the input image set, and the presence or absence of direction unification of curtaining in the input image set and the teacher image set are changed. FIG. 9B shows the total number of false negatives and false positives that require correction included in the predicted image set created under each condition. For condition 1 which is without countermeasures, the total number of false negatives + false positives decreased to about 1 / 10 in condition 4 where all countermeasures were implemented.

[0076] <S7. Measurement Image Creation Step> The measurement image set creation step (S7) shall be executed by the measurement image set creation unit of the processing device 1101 in the information processing device 1100.

[0077] The flowchart of the measurement image set creation process (S7) is shown in FIG. 4F. It includes a step of inputting a predicted image set (S71), a second correction process step (S72), and a step of outputting a measurement image set (S73). As described in the input image set creation process (S1), the input image is created under the condition that false negatives and false positives decrease in the predicted image evaluation process (S6). Also, in the teacher image set creation process (S2), the teacher image is created under the condition that false negatives and false positives decrease in the predicted image evaluation process (S6). The input image set and the teacher image set created under the condition that false negatives and false positives decrease are input to create a machine learning model, and a predicted image set is created under the condition that false negatives and false positives decrease. Such predicted images created under the condition that false negatives and false positives decrease will be input to the measurement image set creation process (S7).

[0078] Taking the input image shown in FIG. 10A as an example, the details of the second correction process step (S72) will be described. The predicted image (FIG. 10B) created by inputting the input image (FIG. 10A) into the machine learning model contains false negatives (31) and false positives (32) to be corrected. A layer display is created by superimposing the input image (FIG. 10A) and the predicted image (FIG. 10B), and a corrected image is created by correcting false negatives and false positives using painting software (FIG. 10C). The user corrects by connecting the broken lines of the false negatives 31 included in the predicted image (FIG. 10B) and corrects by deleting the lines of the false positives 32 included in the predicted image (FIG. 10B). The corrected image (FIG. 10C) corrected by the user is input from the input device 1103 of the information processing apparatus 1100 and stored in the storage device 1102.

[0079] The thin line image (FIG. 10D) is created by thinning FIG. 10C. A binary image (FIG. 10E) is created by regionally dividing the input image (FIG. 10A) by the threshold method. The logical sum image of the binary image (FIG. 10E) and the thin line image (FIG. 10D) is used as the shape measurement image (FIG. 10F). Note that the creation conditions of the measurement image are not limited to the above. It may be changed according to the content of the tissue shape measurement (S8) described later.

[0080] <S8. Tissue Shape Measurement Process> The tissue shape measuring step (S 8 ) is executed by a tissue shape measuring unit of the processing device 1101 in the information processing device 1100 .

[0081] In the structure shape measurement step (S8), the characteristics of the material structure are measured using the measurement image set. For example, the items described in Patent Document 1 are measured. Details are omitted since they are described in the main phase analysis section and grain boundary phase analysis section in Patent Document 1.

[0082] In this embodiment, the process involves expanding the grain boundary phase, which is a divided region included in the predicted image or the teacher image. However, the process is not limited to this, and in the case where the structural shape of the object to be analyzed has a main phase and grain boundary phase that are approximately the same size, the region to be expanded may be the main phase.

[0083] In this embodiment, the main phase (10) is shown as negative and the grain boundary phase (20) as positive, but the main phase (10) can also be shown as positive and the grain boundary phase (20) as negative for evaluation. In this case, the morphology process that expands the area of ​​the grain boundary phase becomes a process that reduces the area of ​​the main phase, which is a divided area included in the predicted image or the teacher image. Also, the false positive area and the false positive area are switched, but the total number of manual corrections will be the same. Also, depending on the structure shape of the analysis target, the area to be reduced may be the grain boundary phase.

[0084] Furthermore, this embodiment is described on the premise of the confusion matrix for two classes shown in FIG. 2, but it is also applicable to region division for three or more classes.

[0085] According to this embodiment, in the evaluation using a confusion matrix in the predicted image evaluation step (S6), by expanding the area used for judgment in the image used for judgment by several pixels, false negatives and false positives caused by a shift of several pixels in the boundary are no longer detected, and false negatives and false positives that require correction can be extracted. [Explanation of symbols]

[0086] 10...main phase, 20...grain boundary phase, 21...two-particle grain boundary phase, 22...first triple junction grain boundary phase, 23...second triple junction grain boundary phase, 24...partially interrupted grain boundary phase, 25...undetectable grain boundary phase, 26...grain boundary phase isolated within main phase, 31...false negative region requiring correction, 32...false positive region requiring correction, 33...true positive region, 34...true negative region, 35...false negative region not requiring correction, 36...false positive region not requiring correction, 4...curtaining, 51...first image intensity range, 52...second image intensity range

Claims

1. A method for analyzing a material structure using an information processing device including a processing device, a storage device, an input device, and an output device, comprising: an input image set creation step of creating an input image set by performing image preprocessing on the material structure observation image set; a teacher image set creation step of creating a teacher image set by dividing each input image of the input image set into a plurality of regions; creating a first image set and a second image set different from the first image set from the input image set and the teacher image set; A machine learning model creation step of creating a machine learning model using the first image set; a predicted image set generation step of inputting the second image set into the machine learning model to generate a predicted image set; A method for analyzing material structure, comprising: a predicted image evaluation step of creating a morphologically processed image that expands or contracts an area contained in one of the predicted image or teacher image of the second image set, and performing an evaluation using a confusion matrix between the morphologically processed image and a non-morphological image.

2. 2. The method for analyzing a material structure according to claim 1, a measurement image set creation step of creating a measurement image set by correcting the predicted image set; and a structure shape measuring step of measuring characteristics of the material structure using the measurement image set.

3. 2. The method for analyzing a material structure according to claim 1, The predicted image evaluation step includes: creating an expanded predicted image by expanding a divided region included in the predicted image of the second image set; A method for analyzing material structure, comprising evaluation using false negative images calculated using a confusion matrix between the teacher images and the expansion prediction images of the second image set.

4. 2. The method for analyzing a material structure according to claim 1, The predicted image evaluation step includes: creating an expanded teacher image by expanding a divided region included in the teacher image of the second image set; A method for analyzing material structure, comprising evaluation using false positive images calculated using a confusion matrix between the predicted images and the expanded teacher images of the second image set.

5. 2. The method for analyzing a material structure according to claim 1, The predicted image evaluation step includes: creating a reduced predicted image by reducing a divided area included in the predicted image of the second image set; A method for analyzing material structure, comprising evaluation using false negative images calculated using a confusion matrix between the teacher images and the reduced predicted images of the second image set.

6. 2. The method for analyzing a material structure according to claim 1, The predicted image evaluation step includes: creating a reduced teacher image by reducing a divided area included in the teacher image of the second image set; A method for analyzing material structure, comprising evaluation using false positive images calculated using a confusion matrix between the predicted images and the reduced teacher images of the second image set.

7. 2. The method for analyzing a material structure according to claim 1, The input image set creation step includes: creating an image intensity histogram of the observed image and measuring the most frequent value in a specified first image intensity range; measuring a mode in a specified second image intensity range; The method for analyzing a material structure includes a step of normalizing the image intensity so that the mode and the mode become specified values.

8. 2. The method for analyzing a material structure according to claim 1, The input image set creation step includes: creating an image intensity histogram of the observed image, and measuring a mean value and a variance of the image intensity histogram in a specified first image intensity range; The material structure analysis method includes an image intensity normalization step of normalizing the image intensity so that the average value and the variance become specified values.

9. 2. The method for analyzing a material structure according to claim 1, The input image set creation step includes: A method for analyzing a material structure, comprising a step of rotating each observation image so that the direction of grooves formed by the ion beam curtain effect when an observation surface is created using an ion beam is aligned between input image sets.

10. 2. The method for analyzing a material structure according to claim 1, The teacher image set creation step includes: A step of creating a binary image by dividing an input image into a main phase and a grain boundary phase; creating a corrected image by correcting a partially interrupted grain boundary phase or an undetectable grain boundary phase in the binary image; a step of thinning the corrected image and then expanding it to create a thick line image; A material structure analysis method comprising the step of creating the teacher image by combining the binary image and the thick line image.

11. A material structure analysis system using an information processing device including a processing device, The processing device includes: An input image set is generated by performing image preprocessing on the observation image set of the material structure; A teacher image set is created by dividing each input image of the input image set into a plurality of regions; creating a first image set and a second image set different from the first image set from the input image set and the teacher image set; creating a machine learning model using the first set of images; inputting the second set of images into the machine learning model to generate a predicted set of images; A material structure analysis system that creates a morphologically processed image that expands or shrinks an area contained in one of the predicted images or teacher images of the second image set, and performs an evaluation using a confusion matrix between the morphologically processed image and the non-morphological image.

Images