Data preprocessing method, average crystal grain size measuring device, and computer program

Abstract

This invention provides a data preprocessing method applied to microstructure images before inputting them into a trained model, enabling the automation of average crystal grain size prediction. [Solution] The data preprocessing method includes the steps of: generating multiple cropped images with different image sizes from a microstructure image; resizing the image size of each of the multiple cropped images to the image size of the input image data to generate multiple pre-input images; inputting the multiple pre-input images into a trained model to obtain pre-predicted values ​​of multiple average crystal grain sizes and relating the image size of each of the multiple cropped images to each of the pre-predicted values ​​of the multiple average crystal grain sizes; and determining the optimal image size from among the image sizes of the multiple cropped images based on the pre-predicted values ​​of the multiple average crystal grain sizes and the image sizes of the multiple cropped images, and cropping the microstructure image to the optimal image size.

Description

Technical Field

[0001] The present disclosure relates to a method for data preprocessing applied to input data before input to a learned model, an average crystal grain size measuring apparatus, and a computer program.

Background Art

[0002] The crystal grain size in a metallic material greatly affects the material properties. As a method for evaluating the crystal grain size, measurement of the average crystal grain size using a microstructure image is known. The average crystal grain size is used, for example, as an evaluation index for the cooling rate during solidification or the performance of a crystal grain refiner. Conventionally, since the average crystal grain size has been manually measured from a microstructure image, there have been problems such as large variations in the measurement results and the time required for measurement.

[0003] In recent years, evaluation methods for crystal grain structures using machine learning have been proposed. Patent Document 1 describes a technique for estimating the crystal grain size from a micrograph image of a metallic material using a learned estimation model generated by supervised learning. Patent Document 2 describes a technique for estimating the crystal grain size by inputting a grain boundary image generated by extracting only the crystal grain boundaries from a microstructure image into a learned estimation model.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0005] According to the grain size estimation methods described in Patent Document 1 or 2, the process can be automated using machine learning without manual intervention, thereby reducing reliance on human expertise. However, even if the measurement of average grain size using microstructure images can be automated using machine learning, if the state of the grains contained in the microstructure image, for example, the proportion of grains, differs, fully automated measurement becomes difficult, and manual measurement may be partially necessary.

[0006] The present invention has been made in view of the above problems, and its objective is to provide a data preprocessing method applied to microstructure images before input to a trained model, an average crystal grain size measuring device, and a computer program that enable the automation of predicting the average crystal grain size even when the state of the crystal grains contained in the microstructure image differs. [Means for solving the problem]

[0007] According to embodiments of the present invention, the following solutions are provided.

[0008] [Item 1] A method for preprocessing data applied to a microstructure image of an alloy material before inputting it into a trained model that predicts the average grain size of crystal grains contained in the microstructure image, A cropped image generation step of generating multiple cropped images with different image sizes from the aforementioned microstructure image, A pre-input image generation step involves resizing the image size of each of the multiple cropped images to the image size of the input image data to be input to the trained model, thereby generating multiple resized pre-input images. A pre-prediction step in which the plurality of pre-input images are input to the trained model, pre-predicted values ​​of a plurality of average crystal grain sizes are obtained from the plurality of pre-input images, and the image size of each of the plurality of cropped images is associated with each of the pre-predicted values ​​of the plurality of average crystal grain sizes, An optimal size determination step is to determine the optimal image size from which the microstructure image should be extracted, based on the pre-predicted values ​​of the plurality of average grain sizes and the image sizes of the plurality of cropped images associated with the pre-predicted values ​​of the plurality of average grain sizes, and to extract the microstructure image at the optimal image size. A method that includes this.

[0009] [Item 2] The aforementioned optimal size determination step is: A calculation step of determining a theoretical value corresponding to the pre-predicted average crystal grain size associated with each of the remaining image sizes of the multiple cropped images, based on one pre-predicted value among the multiple pre-predicted average crystal grain sizes associated with one image size selected from the multiple image sizes of the multiple cropped images, An evaluation step to evaluate the degree of fit between a pre-predicted value of one or more average crystal grain sizes related to the remaining image size and one or more theoretical values ​​corresponding to the pre-predicted value of one or more average crystal grain sizes, The method described in item 1, including the method described in item 1.

[0010] [Item 3] The method according to item 2, wherein the optimal size determination step includes determining a plurality of goodness-of-fit values ​​for the image sizes of the plurality of cropped images by performing the calculation step and the evaluation step for each of the image sizes of the plurality of cropped images.

[0011] [Item 4] The method according to item 3, wherein the optimal size determination step determines the image size corresponding to the highest degree of fit among the multiple degrees of fit as the optimal image size.

[0012] [Item 5] The method according to any one of items 2 to 4, wherein the evaluation step includes evaluating the degree of fit based on the mean squared error between the pre-predicted value of one or more average grain sizes and the one or more theoretical values.

[0013] [Item 6] The step of generating the clipped image includes, for each of the plurality of image sizes, clipping different regions of the microstructure image with the image size to generate two or more pre-clipped images. The pre-prediction step includes, for each of the plurality of image sizes, inputting two or more pre-input images corresponding to the two or more pre-clipped images into the learned model, obtaining two or more pre-predicted values of the average crystal grain size from the two or more pre-input images, and obtaining an average value of the two or more pre-predicted values of the average crystal grain size. The method according to any one of Items 1 to 5, including associating the average value of the pre-predicted values of the two or more average crystal grain sizes with a corresponding one of the plurality of image sizes.

[0014] [Item 7] A method for predicting the average crystal grain size of crystal grains included in a microstructure image of an alloy material, the learned model used in the pre-prediction step is a first learned model, a prediction step of inputting an input image generated from the microstructure image with the optimal image size determined according to the method according to any one of Items 1 to 6 into a second learned model and obtaining a predicted value of the average crystal grain size of the input image, wherein the second learned model is the same as or different from the first learned model, and each of the first and second learned models is constructed to predict the average crystal grain size in pixel units of the image, a conversion step of converting the predicted value of the average crystal grain size in pixel units into a predicted value in the SI unit system, A method including.

[0015] [Item 8] The method according to Item 7, further including a display step of displaying the predicted value in the SI unit system and the input image on a display device.

[0016] [Item 9] The input image includes a scale bar, The conversion step includes obtaining the length of the scale bar included in the input image in the SI unit system and the length of the scale bar in pixel units, and obtaining the length in the SI unit system per pixel based on the length of the scale bar in the SI unit system and the length of the scale bar in pixel units. The method according to item 7 or 8, including the above.

[0017] [Item 10] A method for evaluating the value of dendrite arm spacing included in a microstructure image of an alloy material, comprising: a preprocessing step of generating an evaluation image from the microstructure image with the optimal image size determined according to the method according to any one of items 1 to 9, and an evaluation step of evaluating the value of dendrite arm spacing included in the evaluation image. The method including the above.

[0018] [Item 11] The method according to any one of items 1 to 10, wherein the alloy material is an aluminum alloy sheet.

[0019] [Item 12] An average crystal grain size measuring device comprising a processor configured to execute the method according to any one of items 1 to 11.

[0020] [Item 13] A computer program including a set of instructions for causing one or more computers to execute the method according to any one of items 1 to 11.

[0021] [Item 14] A data preprocessing method applied to a microstructure image of an alloy material before evaluating the value of dendrite arm spacing included in the microstructure image, comprising: A cropped image generation step of generating multiple cropped images with different image sizes from the aforementioned microstructure image, A pre-input image generation step comprising resizing the image size of each of the plurality of cropped images to the image size of input image data to be input to a trained model, thereby generating a plurality of resized pre-input images, wherein the trained model is constructed to predict the average grain size of crystal grains contained in the microstructure image, A pre-prediction step in which the plurality of pre-input images are input to the trained model, pre-predicted values ​​of a plurality of average crystal grain sizes are obtained from the plurality of pre-input images, and the image size of each of the plurality of cropped images is associated with each of the pre-predicted values ​​of the plurality of average crystal grain sizes, An optimal size determination step is to determine the optimal image size from which the microstructure image should be extracted, based on the pre-predicted values ​​of the plurality of average grain sizes and the image sizes of the plurality of cropped images associated with the pre-predicted values ​​of the plurality of average grain sizes, and to extract the microstructure image at the optimal image size. A method that includes this.

[0022] [Item 15] A method for evaluating the value of dendrite arm spacing contained in a microstructural image of an alloy material, A preprocessing step of generating an evaluation image from the microtissue image at the optimal image size determined according to the method described in item 14, An evaluation step to evaluate the dendrite arm spacing values ​​included in the evaluation image, A method that includes this.

[0023] The comprehensive or specific embodiments of this disclosure may be implemented by apparatus, systems, methods, integrated circuits, computer programs, or computer-readable non-temporary storage media, or any combination thereof. Computer-readable storage media may include volatile storage media or non-volatile storage media. Apparatus may consist of multiple devices. If apparatus consists of two or more devices, these two or more devices may be located in a single device or in two or more separate devices. [Effects of the Invention]

[0024] Exemplary embodiments of this disclosure provide a data preprocessing method applied to microstructure images before input to a trained model, an average grain size measuring device, and a computer program that enable the automation of average grain size prediction. [Brief explanation of the drawing]

[0025] [Figure 1] This figure shows an example of a pretreatment that can be applied to microstructure images with a relatively small number of crystal grains. [Figure 2] This figure shows an example of a pretreatment that can be applied to microstructure images with a relatively large number of crystal grains. [Figure 3] This figure shows an example of a resized image obtained by cropping a narrow area from a microstructure image with a relatively large number of crystal grains. [Figure 4] This is a flowchart showing the data preprocessing procedure in the embodiments of this disclosure. [Figure 5] This diagram illustrates multiple cropped areas with different image sizes. [Figure 6A] This figure shows examples of cropped images corresponding to each cropping area. [Figure 6B] This figure shows examples of cropped images corresponding to each cropping area. [Figure 6C] This figure shows examples of cropped images corresponding to each cropping area. [Figure 6D]This figure shows examples of cropped images corresponding to each cropping area. [Figure 6E] This figure shows examples of cropped images corresponding to each cropping area. [Figure 7] This graph plots multiple pre-predicted average crystal grain values ​​corresponding to multiple pre-input images. [Figure 8] This graph shows an example of the relationship between the predicted average crystal grain size per pixel relative to the image size and the theoretical value. [Figure 9] This flowchart shows the procedure for predicting the average grain size in embodiments of the present disclosure. [Figure 10] This flowchart shows the procedure for evaluating the dendrite arm spacing values ​​in embodiments of the present disclosure. [Figure 11A] This figure shows a typical image of a microstructure with a large number of crystal grains. [Figure 11B] This figure shows a typical image of a microstructure with a small number of crystal grains. [Figure 12] This block diagram shows a schematic hardware configuration of an average crystal grain size measuring device in an embodiment of the present disclosure. [Modes for carrying out the invention]

[0026] When predicting the average grain size of crystal grains contained in a microstructure image of an alloy material using machine learning, the number of crystal grains contained in the microstructure image must be within a certain range. Therefore, conventionally, when predicting the average grain size of crystal grains contained in a microstructure image with significantly different grain sizes, methods such as adjusting the magnification during imaging of the metal structure or having a human pre-process the acquired microstructure image were used to keep the number of crystal grains within a certain range.

[0027] Examples of microstructure images include polarized images taken with an optical microscope or polarizing camera. When the number of crystal grains present in or occupying a microstructure image is relatively small, the image size of the image data input to the trained model that predicts the average crystal grain size becomes large. Figure 1 shows an example of preprocessing that can be applied to a microstructure image with a relatively small number of crystal grains. Image 100 shown in Figure 1 is an example of a microstructure image with a relatively small number of crystal grains. The image size of Image 100 is 2048 × 2880 pixels. In this example, since the input image to be fed into the trained model requires many crystal grains, it is necessary to crop a relatively wide area of ​​Image 100 so that the number of crystal grains falls within a certain range and prepare a cropped image.

[0028] Image 110, shown in Figure 1, is an example of a cropped image obtained by cutting out the rectangular area indicated by the dashed line from the region of Image 100. The size of Image 110 in the example in Figure 1 is 2000 x 2000 pixels. Because Image 110 is still large, inputting it into the trained model at this size will increase the computational cost. Therefore, it is necessary to resize Image 110 to an image size suitable for input to the trained model. Here, "cropping" is the process of cutting out a specific area from a large original image to generate a smaller image, and the resolution does not change before and after cropping. On the other hand, "resizing" is the process of downsampling pixels in the horizontal and vertical directions of an image, and the resolution of the image changes before and after resizing.

[0029] Image 120, shown in Figure 1, is an example of a resized image obtained by resizing Image 110. In the example in Figure 1, Image 120 has a size of 200 x 200 pixels. In this example, resizing Image 110 reduces the image data size to 1 / 100th of its original size, thereby reducing computational costs.

[0030] On the other hand, resizing an image reduces its resolution. Figure 2 shows an example of preprocessing that can be applied to a microstructure image with a relatively large number of crystal grains. Image 101 in Figure 2 is an example of a microstructure image with a relatively large number of crystal grains. The image size of Image 101 is 1048 × 1280 pixels. Image 111 in Figure 2 is an example of a cropped image obtained by cutting out the rectangular area indicated by the dashed line from the region of Image 101. The size of Image 111 in the example in Figure 2 is 1000 × 1000 pixels. Image 121 in Figure 2 is an example of a resized image after resizing Image 111. The size of Image 121 in the example in Figure 2 is 200 × 200 pixels, similar to Image 120 in Figure 1.

[0031] As shown in Figure 2, applying preprocessing to microstructure images with a relatively large number of crystal grains reduces the clarity of the resized image. As a result, when such resized images are input into a trained model, the model's prediction accuracy decreases. To avoid this situation, it is necessary to crop the region of image 101 shown in Figure 2 into an area narrower than the dashed rectangular area.

[0032] Figure 3 shows an example of a resized image obtained by cropping a narrow area from a microstructure image with a relatively large number of crystal grains. For reference, Figure 3 also shows cropped image 111 and resized image 121 of image 101 shown in Figure 2. Image 112 in Figure 3 is an example of a cropped image obtained by cropping the rectangular area indicated by the dotted line from the region of image 101. The size of the rectangular area indicated by the dotted line is smaller than the size of the rectangular area indicated by the dashed line from image 111. The image size of image 112 in the example in Figure 3 is 300 × 300 pixels. Image 122 in Figure 3 is an example of a resized image obtained after resizing image 112. The size of image 122 in the example in Figure 3 is 200 × 200 pixels, the same as image 121.

[0033] Image 121 is a low-resolution, resized image where it is difficult to distinguish between crystal grains, while Image 122 is a resized image where it is possible to distinguish between crystal grains. In other words, the accuracy of crystal grain size is low in Image 121, while it is high in Image 122.

[0034] As explained above, conventionally, in order to accurately predict the average grain size using machine learning by inputting microstructure images into a pre-trained model, it was necessary for a person to manually determine the appropriate image size for cropping the original microstructure image so that the number of crystal grains would fall within a certain range. Naturally, since the state of the crystal grains contained in the microstructure image differs depending on the field of view or magnification at the time of shooting, manual preprocessing was required each time a microstructure image was acquired. Alternatively, it was necessary to change the camera's shooting magnification or field of view and reacquire an appropriate input image to feed into the pre-trained model. However, these tasks were problematic due to their reliance on individual expertise and increased workload.

[0035] In view of these challenges, the inventors of the present invention have devised a data preprocessing method applied to microstructure images before input to a trained model, which enhances the robustness of the prediction of average grain size by a trained model, even when the number of crystal grains contained in the microstructure image differs, and enables the automation of the prediction of average grain size. According to this disclosure, it is possible to eliminate reliance on individual expertise and reduce the amount of work required.

[0036] The data preprocessing method and average grain size measuring apparatus according to the embodiments of this disclosure will be described in detail below with reference to the attached drawings. However, unnecessary details may be omitted. For example, detailed explanations of already well-known matters and redundant explanations of substantially identical configurations or processes may be omitted. This is to avoid the following explanation becoming unnecessarily redundant and to facilitate understanding for those skilled in the art. Also, substantially identical configurations or processes may be given the same reference numerals.

[0037] The following embodiments are illustrative, and the data preprocessing method and average grain size measuring apparatus described herein are not limited to these embodiments. For example, the numerical values, shapes, materials, steps, and the order of those steps shown in the following embodiments are merely examples, and various modifications are possible as long as they do not create a technical inconsistency. Furthermore, it is possible to combine one embodiment with another as long as it does not create a technical inconsistency.

[0038] (Embodiment) <1. Data Preprocessing> The data preprocessing in this embodiment is applied to the microstructure image of the alloy material before inputting it into a trained model that predicts the average grain size of the crystal grains contained in the microstructure image. The alloy material in this embodiment is a wrought aluminum alloy. However, the alloy material is not limited to this and may be, for example, steel. The data preprocessing in this embodiment can be widely applied to microstructure images of various alloy materials.

[0039] Figure 4 shows a flowchart illustrating the data preprocessing procedure in this embodiment. The data preprocessing method includes a cropped image generation step S110, a pre-input image generation step S120, a pre-prediction step S130, and an optimal size determination step S140, and is implemented on one or more computers or processors.

[0040] (Cropped image generation step S110) The cropped image generation step S110 includes generating multiple cropped images with different image sizes from a microtissue image. The sizes of the multiple cropped images with different sizes are smaller than the image size of the original microtissue image.

[0041] Microstructure images are obtained, for example, by etching the surface of the alloy material with Barker's solution and then photographing the crystalline structure of the alloy material with an optical microscope or polarizing microscope. The magnification or field of view may vary, for example, depending on the type of alloy system or the location of observation of the metallic structure. As will be described later, the data preprocessing according to this embodiment enables the prediction of the average grain size based on microstructure images taken at any field of view or magnification.

[0042] The microtissue images acquired through imaging are cropped into multiple cropping sizes with different image dimensions, thereby generating multiple cropped images. These cropped images can be generated, for example, by specifying to a computer the location and size of the cropping region within the overall microtissue image.

[0043] Figure 5 illustrates multiple cropped regions with different image sizes. The microstructure image 103 shown in Figure 5 includes five different cropped regions A to E with image sizes 1 to 5, indicated by dashed lines. In the example shown in Figure 5, the shape of each cropped region is rectangular, but it is not limited to this. For clarity, each cropped region is shown near the center of the microstructure image 103. The number of image sizes for the cropped regions is not limited to five, but could be two, three, four, or six or more. The image size of the microstructure image 103 is 2048 × 2880 pixels, and the image sizes 1 to 5 for the five cropped regions A to E are 2000 × 2000 pixels, 1500 × 1500 pixels, 1000 × 1000 pixels, 500 × 500 pixels, and 250 × 250 pixels, respectively.

[0044] Figures 6A to 6E illustrate the cropped images corresponding to cropping regions A to E, respectively. Cropped images 113A to 113E are the images corresponding to cropping regions A to E, respectively. Therefore, the image sizes 1 to 5 for cropped images 113A to 113E are 2000 x 2000 pixels, 15000 x 1500 pixels, 1000 x 1000 pixels, 500 x 500 pixels, and 250 x 250 pixels, respectively.

[0045] The cropped image generation step S110 may include generating two or more pre-cropped images by cropping different regions of the microtissue image for each of the image sizes 1 to 5 of the cropped images 113A to 113E. For example, with image size 1 of cropped image 113A: 2000 x 2000 pixels, the microtissue image may be cropped into two or more different regions so as to include the entire field of view of the microtissue image (or cover the entire area of ​​the microtissue image), resulting in the generation of two or more pre-cropped images with different fields of view from each other. The same process may be applied to image sizes 2 to 5 of the cropped images 113B to 113E, resulting in the generation of two or more pre-cropped images with different fields of view from each other.

[0046] (Pre-input image generation step S120) The pre-input image generation step S120 includes resizing the image size of each of the multiple cropped images to the image size of the input image data to be input to the trained model, thereby generating multiple resized pre-input images.

[0047] In this embodiment, the trained model is a supervised predictive model and is constructed using a neural network (NN). An example of a neural network is a convolutional neural network (CNN). However, the type of neural network is not limited to CNNs and may be other types of neural networks. The trained model in this embodiment may be constructed to predict the average crystal grain size on a pixel-by-pixel basis in an image.

[0048] In this embodiment, the image size of the input image data for input to the trained model is, for example, 200 x 200 pixels. The image size of each of the multiple cropped images can all be resized by the computer to an image size of 200 x 200 pixels. Figures 6A to 6E illustrate the resized images 123A to 123E obtained by resizing multiple cropped images 113A to 113E. The image size of each of the resized images 123A to 123E is 200 x 200 pixels. Thus, although the image sizes 1 to 5 of the cropped images 113A to 113E are different from each other, the pixel size of the resized images 123A to 123E is all the same. Hereafter, the resized images will be referred to as "pre-input images".

[0049] In this way, multiple pre-input images 123A to 123E, each having the same image size, are generated from multiple cropped images 113A to 113E.

[0050] (Pre-prediction step S130) The pre-prediction step S130 includes inputting multiple pre-input images into a trained model, obtaining pre-predicted values ​​for multiple average crystal grain sizes from the multiple pre-input images, and associating the image size of each of the multiple cropped images with each of the pre-predicted values ​​for multiple average crystal grain sizes.

[0051] By inputting data from multiple pre-input images 123A to 123E into a trained model, multiple pre-predicted average crystal grain values ​​corresponding to each of the pre-input images 123A to 123E can be obtained.

[0052] In this embodiment, the pre-prediction step S130 may include inputting two or more pre-input images corresponding to two or more pre-cropped images generated in the cropped image generation step S110 into the trained model for each of the multiple image sizes, obtaining two or more pre-predicted values ​​for average crystal grain size from the two or more pre-input images, and calculating the average value of the two or more pre-predicted values ​​for average crystal grain size. Taking the image size 1: 2000 × 2000 pixels of cropped image 113A as an example, two or more pre-input images obtained by resizing two or more pre-cropped images that have different fields of view and all pixels of the same size (2000 × 2000 pixels) are input into the trained model, and two or more pre-predicted values ​​for average crystal grain size are obtained from these two or more pre-input images. After this, the average value of the pre-predicted values ​​for image size 1: 2000 × 2000 pixels is obtained by averaging the two or more pre-predicted values ​​for average crystal grain size. The same process is applied to the cropped images 113B to 113E with image sizes 2 to 5, and the average of the pre-predicted values ​​for each pixel size is obtained.

[0053] The pre-prediction step S130 may further include associating the average value of pre-predicted average crystal grain sizes for two or more images with a corresponding image size among a plurality of image sizes. For example, it is preferable to associate image sizes 1 to 5 for five types of cropped images 113A to 113E with the average value of the pre-predicted average crystal grain sizes for image sizes 1 to 5 for images 113A to 113E. By associating each image size with the average or representative value of the pre-predicted average crystal grain size for each image size, it is possible to improve the prediction accuracy of the average crystal grain size. Thus, either the pre-predicted average crystal grain size based on one cropped image, or the average or representative value of the pre-predicted average crystal grain size based on two or more pre-cropped images with different fields of view, can be associated with each image size. In this embodiment, the average value of the pre-predicted average crystal grain size based on two or more pre-cropped images with different fields of view is associated with each image size. In this specification, "pre-predicted average crystal grain size" strictly means the average value of this pre-predicted value.

[0054] Figure 7 shows an example graph plotting multiple pre-predicted average crystal grain values ​​corresponding to multiple pre-input images 123A to 123E. The horizontal and vertical axes of the graph represent the image size of the pre-input image and the pre-predicted average crystal grain value (or the average value of the pre-predicted values) per pixel, respectively.

[0055] The following are examples of how to input a model to predict the average crystal grain size: 27.9 pixels when the pre-input image 123A, whose original image is a cropped image 113A (image size 1: 2000×2000 image), is input to the trained model. The following are examples of how to input a model to predict the average crystal grain size: 29.9 pixels when the pre-input image 123B, whose original image is a cropped image 113B (image size 2: 1500×1500 image), is input to the trained model. The following are examples of how to input a model to predict the average crystal grain size: 34.4 pixels when the pre-input image 123C, whose original image is a cropped image 113C (image size 3: 1000×10000 image), is input to the trained model. The following are examples of how to input a model to predict the average crystal grain size: 40.1 pixels when the pre-input image 123D, whose original image is a cropped image 113D (image size 4: 500×500 image), is input to the trained model. The pre-predicted average crystal grain size of 38.9 pixels is shown when the pre-input image 123E, whose original image is a cropped image 113E (image size 5: 250×250 image), is input to the trained model. In this way, the image size of each of the multiple cropped images 113A to 113E before resizing, corresponding to the multiple pre-input images 123A to 123E, is associated with each of the multiple pre-predicted average crystal grain sizes (or the average of the pre-predicted values).

[0056] (Optimal size determination step S140) The optimal size determination step S140 includes determining the optimal image size from which to extract the microstructure image from among the image sizes of multiple cropped images, based on multiple pre-predicted average grain sizes (or the average value of the pre-predicted values) and the image sizes of multiple cropped images associated with the multiple pre-predicted average grain sizes, and then extracting the microstructure image at the optimal image size. The optimal size determination step S140 includes, for example, a calculation step and an evaluation step. The optimal image size is the image size of a cropped image that, when resized and input to a trained model as an input image, outputs the optimal pre-predicted average grain size.

[0057] (Calculation step) The computer determines a theoretical value corresponding to the pre-predicted average grain size associated with each of the remaining image sizes of the multiple cropped images 113A to 113E, based on one pre-predicted value among multiple pre-predicted average grain sizes associated with one image size selected from image sizes 1 to 5 of the multiple cropped images 113A to 113E.

[0058] First, one of the image sizes 1 to 5 of the multiple cropped images 113A to 113E is assumed to be the optimal image size from which to extract the microstructure image in order to input it into the trained model and obtain an appropriate predicted value for the average crystal grain size. In other words, one of the image sizes 1 to 5 of the multiple cropped images 113A to 113E is considered a candidate for the optimal image size.

[0059] The relationship between image size and average grain size can be theoretically calculated. For example, if a microstructure image containing grains with an average grain size of 500 pixels is cropped to two different image sizes, 2000×2000 pixels and 1000×1000 pixels, the average grain size in the cropped images will be 500 pixels in both cases, remaining unchanged between the two cropped images. However, the average grain size in the pre-input image at the resized image size (200×200 pixels) will differ between the two pre-input images. In this example, the average grain size in the pre-input image corresponding to the cropped image at image size 1 (2000×2000 pixels) is 50 pixels, while the average grain size in the pre-input image corresponding to the cropped image at image size 3 (1000×1000 pixels) is 100 pixels. Thus, when the cropped image size is multiplied by N, the average grain size in the pre-input image becomes 1 / N times smaller. By utilizing this relationship, it is possible to obtain a theoretical value corresponding to the predicted value that would be output when a pre-input image corresponding to a cropped image with an image size other than the candidate for the optimal image size is input to a trained model, using the candidate for the optimal image size and the pre-predicted average crystal grain size associated with the candidate for the optimal image size.

[0060] Figure 8 shows an example of the relationship between the predicted and theoretical values ​​of the average crystal grain size per pixel relative to the image size. The predicted and theoretical curves are shown as dotted and dashed lines, respectively. Figure 8 also shows the theoretical curve of the average crystal grain size obtained from an input image of the optimal image size as a solid line.

[0061] Of the five image sizes 1 to 5 mentioned above, for example, image size 1, which is 2000 x 2000 pixels, is considered a candidate for the optimal image size. In this case, the computer calculates a theoretical value corresponding to the predicted output when the four pre-input images 123B to 123E, which correspond to the four cropped images 113B to 113E that have the remaining four image sizes 2 to 5 (excluding image size 1: 2000 x 2000 pixels), are input to the trained model, using image size 1: 2000 x 2000 pixels and the pre-predicted average crystal grain size associated with image size 1: 2000 x 2000 pixels: 27.9 pixels.

[0062] To explain in more detail, the computer uses formula 1 to determine the theoretical values ​​(in pixels) of the average crystal grain size for four image sizes, corresponding to image sizes 2 through 5. The theoretical values ​​of the average crystal grain size for image sizes 2 through 5, obtained from formula 1, are 37.2 pixels, 55.8 pixels, 111.6 pixels, and 223.2 pixels, respectively. The 27.9 in formula 1 is the preliminary predicted value of the average crystal grain size associated with image size 1: 2000 × 2000 pixels. [Mathematics 1] Image size 2: Theoretical value for 1500 x 1500 pixels = 27.9 x 2000 / 1500 Image size 3: Theoretical value for 1000 x 1000 pixels = 27.9 x 2000 / 1000 Theoretical value for image size 4: 500 x 500 pixels = 27.9 x 2000 / 500 Theoretical value for image size 5: 250 x 250 pixels = 27.9 x 2000 / 250

[0063] (Evaluation Steps) The computer evaluates the fit between the pre-predicted average grain size values ​​(1 or more) associated with the remaining image sizes and the corresponding theoretical values ​​(1 or more) for each of these pre-predicted average grain size values. In the example formula in Equation 1, the computer evaluates the fit between the four pre-predicted average grain size values ​​associated with image sizes 2-5 [29.9 pixels, 34.4 pixels, 40.1 pixels, 38.9 pixels] and the four theoretical values ​​obtained from Equation 1 that correspond to these pre-predicted values ​​[37.2 pixels, 55.8 pixels, 111.6 pixels, 223.2 pixels].

[0064] The computer can evaluate the degree of fit based on an evaluation value. An example of an evaluation value is the mean squared error between a pre-predicted average grain size of 1 or more and a theoretical value of 1 or more. In this embodiment, the computer evaluates the degree of fit based on this mean squared error. In the above example, the mean squared error, which is an example of an evaluation value, is obtained based on the formula in Equation 2. A smaller evaluation value indicates a higher degree of fit, and a larger evaluation value indicates a lower degree of fit. [Math 2] Evaluation value = {(29.9-37.2)} 2 +(34.4-55.8) 2 +(40.1-111.6) 2 +(38.9-223.2) 2} / 4

[0065] The optimal size determination step S140 includes determining multiple goodness-of-fit values ​​for each of the multiple cropped image sizes by performing calculation and evaluation steps for each of the multiple cropped image sizes. In other words, assuming that each of the multiple cropped image sizes is the optimal image size, the computer performs calculation and evaluation steps to determine the goodness-of-fit value for each of the image sizes that would be considered candidates for the optimal image size. If there are five types of image sizes 1 to 5, as in this embodiment, five goodness-of-fit values ​​corresponding to the five types of image sizes 1 to 5 are determined.

[0066] The optimal size determination step S140 may further include determining the image size corresponding to the highest degree of fitness among multiple degrees of fitness as the optimal image size. The computer determines the image size corresponding to the highest degree of fitness among five degrees of fitness corresponding to five different image sizes as the optimal image size. In other words, the computer determines the image size corresponding to the lowest evaluation value among the evaluation values ​​corresponding to five different image sizes 1 to 5 as the optimal image size. For example, if the highest degree of fitness among five degrees of fitness corresponding to five different image sizes 1 to 5 corresponds to image size 3: 1000 x 1000 pixels, the computer determines image size 3 as the optimal image size. The computer then determines the pre-input image 123C, which is cropped and resized to image size 3, as the optimal input image to input into the trained model.

[0067] <2. Prediction of average crystal grain size> Figure 9 shows a flowchart illustrating the procedure for predicting the average grain size in this embodiment. The method for predicting the average grain size is for predicting the average grain size of grains contained in a microstructure image of an alloy material, and includes a prediction step S210 and a conversion step S220.

[0068] [Prediction step S210] The prediction step S210 includes inputting an input image generated from the microstructure image at an optimal image size determined according to the data preprocessing method described above (step S100) into a trained model, and obtaining a predicted value for the average crystal grain size of the input image. In this embodiment, the trained model is the same as the trained model into which the pre-input image is input during data preprocessing. However, these two trained models may be different. In this way, when a microstructure image to which data preprocessing has been applied is input to the trained model, the trained model outputs an optimal predicted value for the average crystal grain size. In this embodiment, this predicted value is output on a pixel-by-pixel basis.

[0069] [Conversion step S220] The conversion step S220 includes converting the predicted average grain size at the pixel level to a predicted value in the SI unit system. As mentioned above, the output from the trained model is the predicted average grain size at the pixel level.

[0070] The microstructure image to which data preprocessing has been applied, i.e., the input image to be fed into the trained model, may include the scale bar 150 shown in Figure 5. The computer can obtain the length of the scale bar 150 in SI units and the length of the scale bar 150 in pixels, and based on the length of the scale bar 150 in SI units and the length of the scale bar 150 in pixels, it can determine the length of each pixel in SI units.

[0071] A computer can generate a binary image by applying a binarization process to the data of an input image. As a result of binarization, it is possible to make only the pixel values ​​of the pixels of the scale bar black (or white). For example, a computer can identify the row with the most black pixels by raster scanning the entire image. The computer can then determine the number of black pixels in the row with the most black pixels and decide that this number of black pixels is the length of the scale bar 150 in pixels. In this way, the length of the scale bar 150 in pixels is obtained.

[0072] As illustrated in Figure 5, the length of the scale bar 150 in SI units (100 μm in the example in Figure 5) may be displayed near the scale bar 150. The computer can further extract the image region where the scale bar 150 and its length in SI units are displayed from the row with the most black pixels, and then, for example, apply optical character recognition (OCR) to the image data of the extracted region to read the length of the scale bar 150 in SI units contained in the extracted region. In this way, the length of the scale bar 150 in SI units is obtained.

[0073] The computer can determine the length per pixel in SI units based on the length of scale bar 150 in SI units and the length of scale bar 150 in pixel units. Based on the length per pixel in SI units, the computer can convert the predicted average crystal grain size per pixel into a predicted value in SI units.

[0074] The computer can output the predicted average grain size in SI units and the input image data to a display device and display them on the display device. Through this process, the predicted average grain size in SI units is presented to the user.

[0075] The microstructure image to which the data preprocessing in this embodiment has been applied is suitably used for measuring the average grain size of the crystal grains contained in the microstructure image or for measuring dendrite arm spacing (DAS). The predicted value of the average grain size is suitably used, for example, as an evaluation index for the performance of a grain refiner. DAS is suitably used, for example, for evaluating the cooling rate during solidification of an ingot. The average grain size obtained from the microstructure image of an ingot is generally larger than the DAS value. Therefore, a microstructure image cropped to an optimal image size for accurately predicting the average grain size can also be used for evaluating or measuring DAS.

[0076] Figure 10 shows a flowchart illustrating the procedure for evaluating DAS in this embodiment. This evaluation method is for evaluating or measuring DAS using a microstructural image of an alloy material cut out at an optimal image size, and includes a pretreatment step S100 and an evaluation step S310.

[0077] [Evaluation Step S310] The evaluation step S310 includes evaluating the DAS contained in the evaluation image generated by cropping the microstructure image to the optimal image size determined according to the data preprocessing method (step S100) described above. The evaluation image may be the same as the preprocessed input image that is input to the trained model for obtaining the predicted value of the average crystal grain size. In other words, the preprocessed input image can be used as the evaluation image. Thus, a trained model for predicting the average crystal grain size is used for preprocessing the evaluation of the DAS.

[0078] Image analysis software for performing DAS measurements has been developed and is commercially available. For example, such image analysis tools can be used to evaluate DAS. These image analysis tools enable DAS measurement using methods such as the intersection method or the quadratic branch method. By inputting evaluation images generated by applying preprocessing to the microtissue images to be evaluated into the image analysis tool, various advantages can be obtained compared to not applying preprocessing or visual inspection, such as reduced workload, shorter measurement time, and improved accuracy of measurement.

[0079] A pre-processed evaluation image can be input into a trained model (e.g., a neural network) built to predict DAS, and a predicted DAS value can be obtained. In this way, the DAS value can be automatically evaluated using a trained model, similar to the prediction of the average crystal grain size. Automated evaluation of DAS offers various advantages, such as reduced workload, shorter measurement times, and improved accuracy in measurement.

[0080] <3. Examples> The inventors of this invention verified the accuracy of predicting the average grain size when the data preprocessing method according to this embodiment was applied to microstructure images before inputting them into a trained model, using two microstructure images with significantly different numbers of crystal grains contained in the images.

[0081] A CNN with 5 hidden layers was used as the pre-trained model. The training data consisted of 27,216 images in total: microstructure images obtained by photographing ingots of 1000, 3000, and 5000 series alloys manufactured by DC casting, and extended microstructure images created by augmenting these images. The total of 27,216 images includes 17,417 training datasets, 5,444 test datasets, and 4,355 validation datasets. The average grain size of the crystal grains contained in each microstructure image was measured manually using the sectioning method. The input images used in the pre-trained model had a resolution of 200 x 200 pixels.

[0082] Figures 11A and 11B show representative microstructure images with a large number of crystal grains and a small number of crystal grains, respectively. The microstructure images were captured using a Nikon inverted metallurgical microscope (model number: ECLIPSE MA200).

[0083] Table 1 shows the predicted average grain size. Here, the number of grains was calculated by assuming each grain is equivalent to a circle and using the measured value as the diameter, from (total image area / area of ​​one grain). The error is expressed as the absolute difference between the measured value and the predicted value. The error ratio is expressed as (error / measured value) × 100.

[0084] The verification results confirmed that, within the range of 20 to 700 crystal grains, it is possible to predict with high accuracy the average grain size of crystal grains contained in both microstructure images with a large number of crystal grains and microstructure images with a small number of crystal grains.

[0085] [Table 1]

[0086] Conventionally, when inputting microstructure images into the same pre-trained model as in the examples, it was necessary to manually perform pre-processing such as changing the image size to an appropriate size so that the number of crystal grains would fall within a certain range, or changing the camera's magnification.

[0087] While measuring devices for determining the grain size distribution of crystals have already been developed and are commercially available, these devices are generally designed for measuring steel. Steel can be measured relatively easily by the sectioning method because it has few precipitates within its crystal grains. However, when measuring materials such as aluminum alloys, unlike steel, aluminum alloys have many precipitates within their crystal grains, making measurement by the sectioning method difficult. Therefore, even when using commercially available measuring devices, it is highly likely that it will be impossible to distinguish between crystal grains and precipitates.

[0088] According to the data preprocessing method of this embodiment, even when the number of crystal grains contained in the microstructure image differs, the robustness of the prediction of the average crystal grain size by the trained model is enhanced, and the prediction of the average crystal grain size is automated. In particular, even if the microstructure image of the object to be measured is taken at low magnification, the average crystal grain size can be predicted by applying data preprocessing to the microstructure image.

[0089] <4. Example Hardware Configuration of Measuring Device> The average crystal grain size measuring device in this embodiment (hereinafter simply referred to as the "measuring device") includes a processor configured or programmed to perform the data preprocessing method described above.

[0090] Figure 12 illustrates a block diagram of the schematic hardware configuration of the measuring device 200. A typical example of the measuring device 200 is a computer. The measuring device 200 comprises a communication device 210, a processor 220, a storage device 230, a ROM (Read Only Memory) 240, a RAM (Random Access Memory) 250, an input device 260, and a display device 270. These components are connected to each other via a bus so that they can communicate with one another.

[0091] The communication device 210 is a communication module for communicating with external devices via a network. For example, the communication device 210 can perform wired communication compliant with communication standards such as IEEE 1394 (registered trademark) or Ethernet (registered trademark). The communication device 210 can also perform wireless communication compliant with Bluetooth (registered trademark) or Wi-Fi standards, or cellular mobile communication such as 3G, 4G, or 5G.

[0092] The processor 220 is a semiconductor integrated circuit and includes a central processing unit (CPU). The processor 220 may be implemented by a microprocessor or microcontroller. The processor 220 sequentially executes a computer program stored in the ROM 240, which describes a set of instructions for performing at least one operation, thereby achieving the desired operation.

[0093] The measuring device 200 may, in addition to or instead of the processor 220, include an FPGA (Field Programmable Gate Array) equipped with a CPU, a GPU (Graphics Processing Unit), an ASIC (Application Specific Integrated Circuit), an ASSP (Application Specific Standard Product), or a combination of two or more circuits selected from these circuits.

[0094] The storage device 230 primarily functions as a data storage device. The storage device 230 is, for example, a magnetic storage device or a semiconductor storage device. An example of a magnetic storage device is a hard disk drive (HDD). An example of a semiconductor storage device is a solid-state drive (SSD).

[0095] ROM240 is, for example, writable memory (e.g., PROM), rewritable memory (e.g., flash memory), or read-only memory. ROM240 stores programs that control the operation of the processor 220. ROM240 does not have to be a single recording medium; it can be a collection of multiple recording media. Some of these collections may be removable memory.

[0096] RAM250 provides a workspace for temporarily unpacking the control program stored in ROM240 during boot-up. RAM250 does not need to be a single recording medium; it can be a collection of multiple recording media.

[0097] The input device 260 is a device for converting user instructions into data and inputting it into a computer. Examples of the input device 260 include a keyboard, mouse, or touch panel. Examples of the display device 270 include a liquid crystal display or an organic EL display.

[0098] The computer programs used for data preprocessing in embodiments of this disclosure may be manufactured and sold independently of the measuring apparatus. The computer programs may be provided, for example, on a computer-readable non-temporary storage medium. The computer programs may also be provided by download via a telecommunications line (e.g., the Internet). [Industrial applicability]

[0099] The technology disclosed herein can be widely used in data preprocessing in machine learning. [Explanation of symbols]

[0100] 200... Measuring devices, 210... Communication devices, 220... Processors, 230... Storage devices, 240... ROM, 250... RAM, 260... Input devices, 270... Display devices

Claims

1. A method for preprocessing data applied to a microstructure image of an alloy material before inputting it into a trained model that predicts the average grain size of crystal grains contained in the microstructure image, A cropped image generation step of generating multiple cropped images with different image sizes from the aforementioned microstructure image, A pre-input image generation step involves resizing the image size of each of the multiple cropped images to the image size of the input image data to be input to the trained model, thereby generating multiple resized pre-input images. A pre-prediction step in which the plurality of pre-input images are input to the trained model, pre-predicted values ​​of a plurality of average crystal grain sizes are obtained from the plurality of pre-input images, and the image size of each of the plurality of cropped images is associated with each of the pre-predicted values ​​of the plurality of average crystal grain sizes, An optimal size determination step is to determine the optimal image size from which the microstructure image should be extracted, based on the pre-predicted values ​​of the plurality of average grain sizes and the image sizes of the plurality of cropped images associated with the pre-predicted values ​​of the plurality of average grain sizes, and to extract the microstructure image at the optimal image size. A method that includes this.

2. The aforementioned optimal size determination step is: A calculation step of determining a theoretical value corresponding to the pre-predicted average crystal grain size associated with each of the remaining image sizes of the multiple cropped images, based on one pre-predicted value among the multiple pre-predicted average crystal grain sizes associated with one image size selected from the multiple image sizes of the multiple cropped images, An evaluation step to evaluate the degree of fit between one or more pre-predicted average grain size values ​​related to the remaining image size and one or more theoretical values ​​corresponding to the one or more pre-predicted average grain size values, The method according to claim 1, including the method described in claim 1.

3. The method according to claim 2, wherein the optimal size determination step includes determining a plurality of goodness-of-fit values ​​for the image sizes of the plurality of cropped images by performing the calculation step and the evaluation step for each of the image sizes of the plurality of cropped images.

4. The method according to claim 3, wherein the optimal size determination step determines the image size corresponding to the highest degree of fit among the plurality of degrees of fit as the optimal image size.

5. The method according to any one of claims 2 to 4, wherein the evaluation step includes evaluating the degree of fit based on the mean square error between the one or more pre-predicted average grain size values ​​and the one or more theoretical values.

6. The aforementioned cropped image generation step includes generating two or more pre-cropped images by cropping different regions of the microstructure image for each of the multiple image sizes, The pre-prediction step is performed for each of the multiple image sizes: Two or more pre-input images corresponding to the two or more pre-cut images are input to the trained model, two or more pre-predicted values ​​for average crystal grain size are obtained from the two or more pre-input images, and the average value of the two or more pre-predicted values ​​for average crystal grain size is obtained. The method according to any one of claims 1 to 4, comprising relating the average value of pre-predicted values ​​of two or more average crystal grain sizes to a corresponding image size among the plurality of image sizes.

7. A method for predicting the average grain size of crystal grains contained in a microstructure image of an alloy material, The pre-trained model used in the pre-prediction step is the first pre-trained model, A prediction step comprising inputting an input image generated from the microstructure image at the optimal image size determined according to the method of any one of claims 1 to 4 into a second trained model, and obtaining a predicted value of the average grain size of the input image, wherein the second trained model is the same as or different from the first trained model, and each of the first and second trained models is constructed to predict the average grain size on a pixel-by-pixel basis of the image, A conversion step of converting the predicted value of the average crystal grain size at the pixel level to a predicted value in the SI unit system, A method that includes this.

8. The method according to claim 7, further comprising a display step of displaying the predicted value in the SI unit system and the input image on a display device.

9. The aforementioned input image includes a scale bar. The aforementioned conversion step is, The length of the scale bar in the input image in the SI unit system, and the length of the scale bar in pixels are obtained. Based on the length of the scale bar in SI units and the length of the scale bar in pixel units, the length per pixel in SI units is determined. The method according to claim 7, including the method described in claim 7.

10. A method for evaluating the value of dendrite arm spacing contained in a microstructural image of an alloy material, A preprocessing step of generating an evaluation image from the microstructure image at the optimal image size determined according to the method described in any one of claims 1 to 4, An evaluation step to evaluate the dendrite arm spacing values ​​included in the evaluation image, A method that includes this.

11. The method according to any one of claims 1 to 4, wherein the alloy material is a wrought aluminum alloy.

12. An average crystal grain size measuring device comprising a processor configured to perform the method according to any one of claims 1 to 4.

13. A computer program comprising a set of instructions that causes one or more computers to execute the method according to any one of claims 1 to 4.

Images