Prediction device, prediction method, and program

Abstract

To provide a prediction device and the like capable of predicting process conditions taking into account the structural information of a material.SOLUTION: A prediction device 1 includes a generative model construction unit 21 that constructs a generative model 31 for generating a structural image of a material using a structural image obtained by imaging the material as learning data, a process condition prediction model construction unit 23 that constructs a process condition prediction model 33, which is a regression model that predicts process conditions for an arbitrary structural image, by learning paired data of the structural image of the material and the process conditions, and an image generation-prediction unit 26 that generates the structural image of the material and predicts the process conditions for the structural image using the generative model 31 constructed by the generative model construction unit 21 and the process condition prediction model 33 constructed by the process condition prediction model construction unit 23.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a technique for supporting material development. [Background technology]

[0002] In recent years, a technical field called materials informatics, which effectively utilizes information science, especially data science, to develop new materials, has been attracting attention. In materials informatics, data on various experimental conditions and results are associated and stored in a database, and information useful for the development of new materials is extracted by making full use of statistical analysis, machine learning, simulation, etc.

[0003] For example, Patent Documents 1 and 2 describe a technique for learning a regression model by relating process conditions, such as material composition conditions and heat treatment conditions, to quality characteristic values ​​of the material, and predicting the process conditions of the material from desired quality characteristic values ​​using the regression model. The techniques for learning and generating regression models as disclosed in Patent Documents 1 and 2 are highly accurate and are considered to be easily applicable in the field of material design based on numerical data. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 6617842 [Patent Document 2] Patent No. 6950119 Summary of the Invention [Problem to be solved by the invention]

[0005] However, there is a limit to further improving the accuracy of prediction of process conditions in the method of prediction using a regression model trained based only on the process conditions and numerical data of quality characteristic values ​​of a material as in Patent Documents 1 and 2. Therefore, in order to realize a more accurate prediction, it is considered to take into account information other than numerical data, such as structural information of the material.

[0006] The present invention has been made in view of the above problems, and has an object to provide a prediction device and the like capable of predicting process conditions taking into account the structural information of a material. [Means for solving the problem]

[0007] The first invention for solving the above-mentioned problems is a prediction device comprising: a generative model construction unit that constructs a generative model for generating a structural image of a material using a structural image obtained by imaging a material as learning data; a process condition prediction model construction unit that constructs a process condition prediction model, which is a regression model that predicts the process conditions of any structural image, by learning paired data of the structural image of the material and process conditions; and an image generation-prediction unit that generates a structural image of the material and predicts the process conditions of the structural image using the generative model constructed by the generative model construction unit and the process condition prediction model constructed by the process condition prediction model construction unit.

[0008] According to the first invention, by learning a generative model or a regression model using data on structural images and process conditions obtained through experiments, it becomes possible to predict process conditions taking into account the structural information of the material.

[0009] In the first invention, it is preferable that the number of epochs in the generative model construction unit and the process condition prediction model construction unit is determined based on the reconstruction accuracy of the tissue image generated by the generative model and the distribution of the process conditions predicted by the process condition prediction model. It is also preferable that the system further comprises an adjustment means for adjusting the number of epochs in the generative model construction unit and the process condition prediction model construction unit. This makes it possible to construct a learning model (generative model, process condition prediction model) that achieves both image reconstruction accuracy and process condition prediction accuracy.

[0010] It is also desirable that the image generation-prediction unit inputs an image obtained by performing super-resolution processing on the structural image of the material generated by the generative model into the process condition prediction model to predict the process conditions. In addition, it is desirable that the process condition prediction model construction unit uses, as the structural image of the material to be learned, an image obtained by reconstructing the structural image of the material using the generative model and then performing super-resolution processing on the image. By using images that have been subjected to super-resolution processing to train and predict the process condition prediction model, it is possible to improve the training and prediction performance.

[0011] In the first invention, it is preferable that the system further comprises a property prediction model constructing unit that constructs a property prediction model, which is a regression model that predicts the quality characteristic value of an arbitrary structural image, by learning paired data of the structural image of the material and the quality characteristic value, and the image generation-prediction unit predicts the quality characteristic value of the generated structural image of the material using the property prediction model. This makes it possible to further predict quality characteristics for the generated structural image of the material.

[0012] It is also preferable to further include a target image acquisition unit that acquires a tissue image of the material having a target quality characteristic from the tissue image of the material generated by the image generation-prediction unit. For example, the target image acquisition unit determines whether or not a quality characteristic value for a tissue image generated by the generative model predicted by the image generation-prediction unit satisfies a predetermined target condition, and if the quality characteristic value satisfies the predetermined target condition, sets the generated tissue image as a tissue image of the material having the target quality characteristic. This makes it possible to obtain a tissue image of a material having the target quality characteristic.

[0013] It is also preferable to further include a visualization unit that visualizes the process conditions and the predicted quality characteristic values ​​for the structural image of the material having the target quality characteristics, which makes it easier to evaluate the process conditions and the quality characteristics for the structural image of the material having the target quality characteristics.

[0014] It is also preferable that the number of epochs in the generative model construction unit and the property prediction model construction unit is determined based on the reconstruction accuracy of the tissue image generated by the generative model and the distribution of quality property values ​​predicted by the property prediction model. It is also preferable that the system further comprises an adjustment means for adjusting the number of epochs in the generative model construction unit and the property prediction model construction unit. This makes it possible to construct a learning model (generative model, property prediction model) that achieves both image reconstruction accuracy and quality property prediction accuracy.

[0015] It is also desirable that the image generation-prediction unit inputs an image obtained by performing super-resolution processing on the structural image of the material generated by the generative model into the characteristic prediction model, and predicts a quality characteristic value. In addition, it is desirable that the characteristic prediction model construction unit uses, as the structural image of the material to be learned, an image obtained by reconstructing the structural image of the material using the generative model and then performing super-resolution processing on the image. By using images that have been subjected to super-resolution processing to train and predict characteristics prediction models, it is possible to improve the training and prediction performance.

[0016] Furthermore, it is preferable that the image generation-prediction unit generates more tissue images including tissues that are strongly related to the target quality characteristics, or generates fewer tissue images including tissues that are weakly related to the target quality characteristics, compared to a case where tissue images are generated randomly, thereby making it possible to efficiently generate tissue images having characteristics close to the target quality characteristics.

[0017] It is also preferable that the distribution of quality characteristic values ​​of tissue images generated by the image generation-prediction unit is biased, so that tissue images having specific quality characteristics are generated more frequently, and tissue images that meet specific quality requirements can be preferentially generated.

[0018] A second invention is a prediction method including the steps of: a computer constructing a generative model for generating a structural image of a material using a structural image obtained by imaging a material as learning data; constructing a process condition prediction model, which is a regression model that predicts process conditions for an arbitrary structural image, by learning paired data of the structural image of the material and process conditions; and generating a structural image of the material and predicting process conditions for the structural image using the generative model constructed in the step of constructing the generative model and the process condition prediction model constructed in the step of constructing the process condition prediction model.

[0019] The third invention is a program for causing a computer to function as a generative model construction unit that constructs a generative model for generating a structural image of a material using a structural image obtained by imaging the material as learning data, a process condition prediction model construction unit that constructs a process condition prediction model, which is a regression model that predicts the process conditions of any structural image, by learning paired data of the structural image of the material and process conditions, and an image generation-prediction unit that generates a structural image of the material and predicts the process conditions for the structural image using the generative model constructed by the generative model construction unit and the process condition prediction model constructed by the process condition prediction model construction unit. Effect of the Invention

[0020] According to the present invention, it is possible to predict process conditions taking into account the structural information of the material. [Brief description of the drawings]

[0021] [Figure 1] FIG. 1 is a diagram showing a hardware configuration of a computer used as a prediction device 1. [Diagram 2] FIG. 2 is a block diagram showing a functional configuration of the prediction device 1. [Diagram 3] FIG. 13 is a diagram showing an example in which VQVAE is used as a generative model 31. [Figure 4]1 is a diagram showing an example of a structural image of a rare earth magnet generated by a generative model 31. FIG. [Diagram 5] FIG. 2 is a diagram showing an example of a structural image of graphite cast iron generated by a generation model 31. [Figure 6] FIG. 2 is a diagram illustrating a process condition prediction model constructing unit 23. [Figure 7] 1A to 1C are diagrams illustrating an example of a manufacturing process for a rare earth magnet. [Figure 8] FIG. 13 is a diagram showing an example of diffusion source application. [Figure 9] FIG. 13 is a diagram showing an example of heat treatment conditions in a diffusion process. [Figure 10] 13 is a diagram for explaining the generation of a texture image and the prediction of process conditions in the image generation-prediction unit 26. FIG. [Figure 11] 1 is a flowchart showing the flow of processing executed by the prediction device 1. [Figure 12] FIG. 13 is a diagram showing the difference in generated images depending on the number of epochs. [Figure 13] 13 is a graph showing the relationship between the number of epochs and the reconstruction loss. [Figure 14] The horizontal axis shows the measured values ​​for the diffusion source composition, and the vertical axis shows the predicted values. (a) shows the case of Fe composition, (b) shows the case of Ga composition, and (c) shows the case of Cu composition. [Figure 15] This is a diagram showing the heat treatment conditions (heating temperature, holding time) with the horizontal axis representing the actual measured values ​​and the vertical axis representing the predicted values. (a) shows the heat treatment conditions (heating temperature, holding time) for the first stage, and (b) shows the heat treatment conditions (heating temperature, holding time) for the second stage. [Figure 16] FIG. 13 is a diagram illustrating the relationship between the clarity of a generated image and the generation speed. [Figure 17] FIG. 13 is a diagram showing an example of applying super-resolution processing to a generated image. [Figure 18] FIG. 2 is a block diagram showing a functional configuration of a prediction device 1a. [Figure 19] FIG. 2 is a diagram illustrating a characteristic prediction model construction unit 22. [Figure 20]10 is a diagram for explaining the generation of a structure image in the image generation-prediction unit 26 and the prediction of quality characteristic values ​​and process conditions. FIG. [Figure 21] FIG. 13 is a diagram showing the distribution of predicted values ​​of quality characteristic values ​​of randomly generated images. [Figure 22] 1 is a flowchart showing the flow of processing executed by the prediction device 1a. [Figure 23] 1 is a diagram showing the magnetic flux density Br [T] with the horizontal axis representing the actual measured value and the vertical axis representing the predicted value of the characteristic prediction model 32. (a) shows the case of the learning data, (b) shows the case of the verification data, and (c) shows the case of the test data. [Figure 24] FIG. 13 is a diagram showing examples of images 5a, 5b, and 5c generated when high Br (magnetic flux density) and low HcJ (coercive force) are set as target characteristics. [Diagram 25] 13A to 13F are diagrams showing examples of images 5d, 5e, and 5f generated when low Br (magnetic flux density) and high HcJ (coercive force) are set as target characteristics. [Figure 26] 13A to 13C are diagrams showing examples of images 5g, 5h, and 5i generated when high Br (magnetic flux density)-high HcJ (coercive force) are set as target characteristics. [Figure 27] 13 is a diagram showing an example of visualization of predicted quality characteristics for generated structural images, where (a) shows a SHAP distribution map of magnetic flux density Br and a SHAP distribution map of coercive force HcJ for generated image 5g, (b) shows a SHAP distribution map of magnetic flux density Br and a SHAP distribution map of coercive force HcJ for generated image 5h, and (c) shows a SHAP distribution map of magnetic flux density Br and a SHAP distribution map of coercive force HcJ for generated image 5i. [Figure 28] FIG. 2 is a block diagram showing a functional configuration of a prediction device 1b. [Figure 29] FIG. 13 is a diagram for explaining an example in which a learning tissue image 41 is input to an encoder 211 to acquire a latent variable Z. [Diagram 30] FIG. 13 is a diagram for explaining an example of learning a conditional probability distribution (autoregressive model 34) based on a data string of a latent variable Z. [Diagram 31] FIG. 13 is a diagram illustrating an example in which an autoregressive model softmax function 342 is modified to a coefficient-added softmax function 342a by adding a coefficient T. [Diagram 32] FIG. 13 is a diagram showing a change in the function when the coefficient of a softmax function 342a is changed. [Diagram 33] 13 is a flowchart showing the flow of processing executed by the prediction device 1b. [Diagram 34] 13 is a graph showing the distribution of quality characteristic values ​​of generated images when the coefficient T is adjusted. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0022] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. In this embodiment, an example will be described in which a prediction device 1 is used to generate a structural image of a rare earth magnet and predict process conditions, etc. Note that the material is not limited to a rare earth magnet, and the present invention can be applied to various materials including, for example, metals, magnets, ceramics, resins, etc.

[0023] [First embodiment] Fig. 1 is a diagram showing an example of a hardware configuration of a prediction device 1 according to an embodiment of the present invention. For example, when a general personal computer is used as the prediction device 1, the prediction device 1 includes a control unit 101, a storage unit 102, a communication unit 103, an input unit 104, a display unit 105, a peripheral device I / F unit 106, and the like, which are connected via a bus 108, as shown in Fig. 1. Note that the configuration shown in Fig. 1 is only an example, and the prediction device 1 can have various configurations depending on the application and purpose.

[0024] The control unit 101 includes a CPU (Central Processing Unit), a ROM (Read Only Memory), and The CPU includes a memory unit 102, a RAM (Random Access Memory), etc. The CPU loads a program stored in a recording medium such as the storage unit 102 or a ROM into a work memory area on the RAM and executes the program, and drives and controls each unit connected via a bus 108 to realize each process of the prediction device 1, which will be described later.

[0025] The ROM is a non-volatile memory that permanently stores the computer's boot program, programs such as the BIOS, data, etc. The RAM is a volatile memory that temporarily stores programs and data loaded from the storage unit 102 or a recording medium such as a ROM, and also has a work area that the control unit 101 uses to perform various processes.

[0026] The storage unit 102 is a hard disk drive (HDD), a solid state drive (SSD), a flash memory, or the like, and stores programs executed by the control unit 101, data necessary for executing the programs, an operating system (OS), etc. In particular, in this embodiment, the storage unit 102 stores an application program for causing the prediction device 1 to execute a process described later.

[0027] The communication unit 103 includes a communication interface and a communication control circuit that mediate communication of the prediction device 1, and controls communication via a network. The network includes a local area network (LAN), a wide area network (WAN), the Internet, etc., and may be wired or wireless.

[0028] The input unit 104 includes input devices such as a keyboard, a mouse, or a touch panel, and various operation buttons, etc. The input unit 104 transmits input data and operation instructions to the control unit 101.

[0029] The display unit 105 includes a display such as a liquid crystal panel, and displays data such as images and text on the display according to instructions from the control unit 101.

[0030] The peripheral device I / F unit 106 is a port for connecting peripheral devices, and includes short-range wireless communication such as USB and Bluetooth (registered trademark). The control unit 101 transmits and receives data to and from peripheral devices via the peripheral device I / F unit 106. For example, a printer is connected to the peripheral device I / F unit 106.

[0031] Next, the functional configuration of the prediction device 1 will be described. 2 is a block diagram showing the functional configuration of the prediction device 1 according to this embodiment. As shown in the figure, the prediction device 1 includes a generative model construction unit 21, a process condition prediction model construction unit 23, a sampling unit 25, an image generation-prediction unit 26, and a visualization unit 29. A control unit 101 of the prediction device 1 reads out and executes a program stored in a storage unit 102 or a ROM, thereby realizing these functions.

[0032] The generative model construction unit 21 constructs a generative model 31 for generating a structural image of a material using a structural image 41 (hereinafter also referred to as a learning structural image 41) obtained by imaging a material as learning data. The learning structural image 41 used as learning data is an image of a target material imaged by an electron microscope or the like. The learning structural image 41 is linked to process conditions (actual measured values) and stored in a storage or the like that can be communicatively connected via the memory unit 102 or the communication unit 103.

[0033] FIG. 3 is a diagram for explaining the concept of constructing a generative model 31 by the generative model constructing unit 21. FIG. 3 shows an example of a vector quantized variational auto-encoder (VQVAE). The encoder 211 is a functional unit that compresses an image into a low-dimensional latent variable Z, and the decoder 212 is a functional unit that restores the original image from the latent variable Z. The encoder 211 and the decoder 212 are constructed, for example, by a neural network such as a CNN (convolutional neural network). As shown in the figure, a learning tissue image 41 obtained by imaging a material is used as learning data, and the encoder 211, the decoder 212, and the VQVAE including an embedding vector V are trained to construct a generative model 31 that generates an image (generated image) 51 similar to the learning tissue image 41.

[0034] Note that the generative model 31 is not limited to VQVAE, and may be VAE, VQVAE2, Generative Adversarial Networks (GAN), Gaussian Copula, or other unsupervised machine learning.

[0035] FIG. 4 is an example of a structural image (generated image) of a rare earth magnet generated by generative model 31. This image is generated by generative model 31 that has been trained using a structural image obtained by photographing a rare earth magnet as learning data. FIG. 5 is an example of a structural image (generated image) of graphite cast iron generated by generative model 31. The image is generated by generative model 31 that has been trained using a structural image obtained by photographing graphite cast iron as learning data.

[0036] Next, a description will be given of the process condition prediction model construction unit 23 in Fig. 2. The process condition prediction model construction unit 23 learns and constructs a regression model (process condition prediction model 33) that predicts the process conditions of an arbitrary structural image (generated image) by using learning data 43 that is a pair of a structural image of a material and a process condition.

[0037] Here, the structural image of the material in the training data 43 is an image (hereinafter also referred to as a training generated image 51) obtained by restoring (reconstructing) the structural image 41 (see FIG. 3) used in constructing the generative model 31 using the generative model 31. In other words, this is an image obtained by inputting the training structural image 41 to a trained encoder 211 to obtain a latent variable Z, and inputting the latent variable Z to a trained decoder 212.

[0038] 6 is a diagram for explaining the concept of constructing the process condition prediction model 33 by the process condition prediction model constructing unit 23. As shown in the figure, the process condition prediction model constructing unit 23 inputs learning data 43, which is a pair of a material structure image (learning generated image 51) and its process conditions, to a learning device 220, and constructs the process condition prediction model 33 by performing supervised learning.

[0039] The learning device 220 uses the learning generated images 51 of the learning data 43 as explanatory variables (input) and the process conditions 60 paired with the learning generated images 51 as objective variables (output), and learns the process condition prediction model 33 by CNN (convolutional neural network) or other known machine learning. When a structure image (generated image) with unknown process conditions is input, the learned process condition prediction model 33 predicts and outputs the process conditions 60 of the structure image (generated image).

[0040] Although the process condition prediction model 33 is trained using the training generated images 51 as explanatory variables, the process condition prediction model 33 may be trained using the training tissue images 41 as explanatory variables instead of the training generated images 51. The process condition prediction model 33 may also be trained using the latent variable Z used when the training generated images 51 are generated as explanatory variables.

[0041] Here, an example of the manufacturing process of a rare earth magnet will be described. FIG. 7 is a flow chart showing the manufacturing process of a rare earth magnet. As shown in the figure, after the steps of SC alloy preparation (step S1), hydrogen treatment (step S2), jet mill crushing (step S3), and magnetic field molding (step S4), sintering (step S5) and rough machining (step S6) are performed. After that, the steps of diffusion source application (step S7) and diffusion treatment (step S8) are performed, followed by finish grinding (step S9) and surface treatment (step S10) to manufacture a rare earth magnet. In this embodiment, of the above manufacturing steps, the process conditions in diffusion source application (step S7) and diffusion treatment (step S8) are predicted.

[0042] Fig. 8 is a diagram showing an example of diffusion source application in step S7. In the example shown, Pr-Fe-Cu-Ga is applied as a diffusion source to a base magnet of Nd-Fe-B in a container (however, the base magnet and the diffusion source are not limited to the example shown in the figure). In this embodiment, the diffusion source composition is set as a process condition to be predicted in the diffusion source application (Fig. 14).

[0043] FIG. 9 is a diagram showing an example of heat treatment conditions for the diffusion treatment in step S8. In this embodiment, a two-stage heat treatment is performed with the diffusion source applied to the base magnet. During the heat treatment, the diffusion source is diffused into the base magnet, causing a change in the material properties. In this embodiment, the heat treatment conditions for the first stage (heating temperature TH, holding time tH) and the heat treatment conditions for the second stage (heating temperature TL, holding time tL) are set as the process conditions to be predicted in the diffusion treatment (FIG. 15).

[0044] The process conditions to be predicted are not limited to the above examples, and various process conditions can be predicted. The manufacturing process illustrated in FIG. 7 is merely an example, and other manufacturing processes can be adopted.

[0045] The sampling unit 25 samples the latent variable Z to be input to the decoder 212 of the generative model 31 in order to generate a tissue image. The image generation-prediction unit 26 generates an arbitrary tissue image and predicts the process conditions of the generated tissue image using an image generation-prediction model that combines the generation model 31 constructed by the generation model construction unit 21 and the process condition prediction model 33 constructed by the process condition prediction model construction unit 23. Specifically, the image generation-prediction unit 26 inputs the latent variable Z sampled by the sampling unit 25 to the decoder 212 of the generation model 31 as shown in Fig. 10, and generates a generated image 51. In addition, the image generation-prediction unit 26 predicts the process conditions 60 for the generated generated image 51 using the process condition prediction model 33.

[0046] The visualization unit 29 displays the generated image 51 generated by the image generation-prediction unit 26 and its process conditions 60 on the display unit 105 (screen). The visualization unit 29 may also display the predicted process conditions 60 as a process condition map plotted on a graph or the structure image 5, or may display the SHAP value as a SHAP distribution chart plotted on a graph or the structure image 5.

[0047] Next, the flow of the process (prediction method) executed by the prediction device 1 will be described with reference to the flowchart in Fig. 11. Each function of the prediction device 1 is realized by the control unit 101 of the prediction device 1 executing the process of each step in Fig. 11.

[0048] The control unit 101 of the prediction device 1 acquires a tissue image 41 (learning tissue image 41) obtained by imaging a certain material from the memory unit 102 or the like as learning data, and learns and constructs a generative model 31 such as VQVAE (step S101).

[0049] The control unit 101 also uses the learning data 43 that pairs the tissue image and the process condition to construct a process condition prediction model 33, which is a regression model that predicts the process condition of an arbitrary tissue image (generated image) (step S102). Specifically, the control unit 101 constructs the process condition prediction model 33 by learning the process condition prediction model 33 using the learning generated image 51, which is an image obtained by restoring (reconstructing) the learning tissue image 41 by the generative model 31, as an explanatory variable (input) and the paired process condition 60 as an objective variable (output), thereby constructing the process condition prediction model 33 that predicts the process condition 60 from an arbitrary tissue image (generated image).

[0050] In steps S101 and S102, the learning tissue images 41 and the learning generated images 51 used as learning data may be divided into rectangular images of a predetermined size.

[0051] Next, the control unit 101 creates an image generation-prediction model by linking the generation model 31 constructed in step S101 and the process condition prediction model 33 constructed in step S102 (step S103). The image generation-prediction model outputs an arbitrary generated image 51 using the generation model 31, and also outputs a predicted value of the process condition 60 of the generated image 51 using the process condition prediction model 33.

[0052] Here, the operator verifies / tests the image generation-prediction model created in step S103 using validation data and test data, and judges whether the reconstruction accuracy of the generated image 51 and the distribution of the predicted values ​​of the process conditions 60 are appropriate (step S104). If they are inappropriate (step S104; No), the operator adjusts the hyperparameters of the generation model 31 (step S105), and reconstructs the generation model 31 (step S101) and the process condition prediction model 33 (step S102). The hyperparameters are, for example, the number of epochs.

[0053] In step S104, it is preferable that the control unit 101 displays the original image and the generated image 51 restored from the original image by the generative model 31 side by side, or displays the distribution of the process conditions 60 predicted by the process condition prediction model 33 on the display unit 105, so that the operator can visually check the reconstruction accuracy and the distribution of the predicted values ​​of the process conditions 60. Also, an "Adjust" button for moving to a hyperparameter adjustment screen and a "Next" button for moving to the next process may be displayed to accept input of appropriate / inappropriate.

[0054] If the operator determines that the reconstruction accuracy of the generated image 51 and the distribution of the predicted values ​​of the process conditions 60 are inappropriate (step S104; No), he / she operates, for example, an "adjust" button to transition to a hyperparameter adjustment screen. In this case, the control unit 101 displays the hyperparameter adjustment screen and accepts adjustments of the number of epochs, etc. from the operator (step S105). The function of "displaying the hyperparameter adjustment screen and accepting adjustments of the number of epochs, etc. from the operator" executed by the control unit 101 in step S105 is an example of the "adjustment means" according to the present invention. Note that the adjustment of the hyperparameters may be performed by directly rewriting the relevant parts of the program, rather than on the adjustment screen.

[0055] The operator repeats learning, verification, and testing of the generative model 31 and the process condition prediction model 33 while adjusting the hyperparameters (steps S101 to S105). Then, when the operator determines that the reconstruction accuracy of the generated image 51 and the distribution of the predicted values ​​of the process conditions 60 are appropriate (step S104; Yes), the learning process ends and the process proceeds to the next step S201.

[0056] Fig. 12 shows the difference in generated images depending on the number of epochs. Fig. 12(a) shows the original image, Fig. 12(b) shows the generated image when the number of epochs is 30 (image restored from the original image), and Fig. 12(c) shows the generated image when the number of epochs is 400 (image restored from the original image). As shown in the figure, it can be seen that when the number of epochs is 400, details can be reproduced more accurately than when the number of epochs is 30, and the reconstruction accuracy is higher.

[0057] Fig. 13 is a graph showing the relationship between the number of epochs and the reconstruction loss. The reconstruction loss is a value that represents the difference between an original image and an image (generated image) restored from the original image. The smaller the reconstruction loss, the higher the reconstruction accuracy of the image.

[0058] 12 and 13, increasing the number of epochs increases the reconstruction accuracy of the image, but if the reconstruction accuracy is too high, the distribution of the process conditions will not have an appropriate spread, and the prediction accuracy of the process conditions for an unknown generated image may decrease. Therefore, in this embodiment, in order to determine the optimal number of epochs that achieves both the reconstruction accuracy and the prediction accuracy of the process conditions, a process is provided in steps S104 to S105 of FIG. 11 to adjust the number of epochs while checking whether or not both the reconstruction accuracy and the distribution of the predicted values ​​of the process conditions are appropriate.

[0059] In step S104, the control unit 101 may evaluate the reconstruction accuracy and the distribution of the predicted values ​​of the process conditions using a predetermined criterion and automatically determine whether the reconstruction accuracy is appropriate or inappropriate. For example, the control unit 101 determines that the reconstruction accuracy is appropriate if the reconstruction loss is less than a predetermined threshold, and inappropriate if the reconstruction loss is equal to or greater than the threshold. In addition, the control unit 101 determines that the distribution of the predicted values ​​of the process conditions is appropriate if a statistical index such as the variance of the distribution satisfies a predetermined condition, and inappropriate if the statistical index does not satisfy the predetermined condition. Then, if it is determined that both the reconstruction accuracy and the distribution of the predicted values ​​of the process conditions are appropriate (step S104; Yes), the learning process is terminated and the process proceeds to step S201.

[0060] 14 and 15 are graphs showing the prediction performance of the trained process condition prediction model 33 (regression model). In both of FIG. 14 and FIG. 15, normalization processing is performed based on the actual measured values. FIG. 14 is a graph showing the diffusion source composition with the horizontal axis showing the actual measured values ​​and the vertical axis showing the predicted values, (a) is a graph of the composition Fe, (b) is a graph of the composition Ga, and (c) is a graph of the composition Cu. Each graph shows the coefficient of determination R2, which is a performance index of the regression model, and the coefficients of determination R2 are R2=0.9565, R2=0.9895, and R2=0.9895 for the compositions Fe, Ga, and Gu, respectively. FIG. 15 is a graph showing the heat treatment conditions (heating temperature, holding time) with the horizontal axis showing the actual measured values ​​and the vertical axis showing the predicted values, (a) is a graph of the heat treatment conditions (heating temperature, holding time) of the first stage, and (b) is a graph of the heat treatment conditions (heating temperature, holding time) of the second stage. The coefficients of determination R2 for the heating temperature and holding time in the first stage are R2 = 0.9530 and R2 = 0.9640, and the coefficients of determination R2 for the heating temperature and holding time in the second stage are R2 = 0.9726 and R2 = 0.976. From Figures 14 and 15, it can be seen that high prediction accuracy was achieved for both the diffusion source composition and the heat treatment conditions, and a process condition prediction model 33 with good prediction performance was created.

[0061] Returning to the explanation of the flowchart in FIG. The control unit 101 samples a latent variable Z to be input to the generation model 31 (decoder 212) of the image generation prediction model (step S201). In this embodiment, the latent variable Z is any data such as random noise.

[0062] The control unit 101 generates a tissue image (generated image 51) using an image generation-prediction model with the latent variable Z (any data such as random noise) sampled in step 201 as an input, and predicts process conditions 60 for the generated image 51 (step S202). The output data of the control unit 101 is data including the generated image 51 and the process conditions 60 predicted from the generated image 51.

[0063] Next, the control unit 101 performs a visualization process (step S203). In the visualization process, the control unit 101 displays the generated image 51 (tissue image) and the prediction result of the process condition 60 on the display unit 105. In addition, as a visualization method, a quality characteristic map generation method described in JP 2021-111360 A or the like may be used. In JP 2021-111360 A, a tissue image is cut into a rectangular region of a predetermined size, and the SHAP value of the magnetic flux density and the coercive force is calculated for each cut-out region, and a map of the SHAP value is generated. SHAP (SHapley Additive exPlanations) is one of the XAI (Explainable AI) technologies, and is a method for quantitatively calculating the influence of each explanatory variable on the prediction result output by the prediction model. This method may be applied to the case of process conditions to generate a map of the process conditions.

[0064] As described above, the prediction device 1 of this embodiment trains the generation model 31 using a structural image obtained by imaging a material as learning data, trains the process condition prediction model 33 using data paired with the structural image of the material and the process conditions as learning data, and constructs an image generation-prediction model by linking the generation model 31 and the process condition prediction model 33. This makes it possible to generate any structural image, and to construct a learning model that predicts the process conditions of the generated structural image, making it possible to predict process conditions taking into account the structural information of the material. Furthermore, when a structural image having an ideal structure is given, the structural image can be input into the process condition prediction model 33 to predict the process conditions that realize the ideal structure. Furthermore, when a structural image having an abnormal structure is given, the structural image can be input into the process condition prediction model 33 to predict the process cause of the abnormal structure.

[0065] (Application of super-resolution processing) Super-resolution processing may be applied to the images generated by the generative model 31. This makes it possible to obtain high-definition images without drastically increasing the computational cost. FIG. 16 is a diagram showing the relationship between the sharpness and generation speed of a generated image. As can be seen from the diagram, there is a trade-off between the sharpness and generation speed of a generated image. In other words, if an attempt is made to generate a generated image with high sharpness, the generation speed will be slow. Although it is possible to obtain a generated image with higher sharpness by increasing the complexity of the generative model 31, the calculation cost will be enormous when generating a large number of images. Therefore, instead of increasing the complexity of the generative model 31 itself, by performing super-resolution processing on the generated image generated by the generative model 31, it is possible to suppress an increase in the calculation cost while increasing the sharpness. This mitigates the trade-off between the sharpness and generation speed of the generated image.

[0066] 17 is a diagram showing an example of applying super-resolution processing to a generated image. As shown in the figure, super-resolution processing is applied to a generated image 51 output from the decoder 212 of the trained generative model 31. As a result, a sharpened generated image 51 is obtained. A known method for sharpening an image is used for the super-resolution processing.

[0067] Among the processes shown in Fig. 11, the super-resolution process can be applied during image generation (step S202 in Fig. 11). Specifically, in step S202 in Fig. 11, the control unit 101 (image generation-prediction unit 26) performs super-resolution process on a tissue image (generated image 51) generated by a generation model 31 of the image generation-prediction model to output a sharpened generated image 51, and inputs the sharpened generated image 51 to a property prediction model 32 of the image generation-prediction model to predict a quality property value 61.

[0068] Also, super-resolution processing can be applied during learning of property prediction model 32 (step S102 in FIG. 11). Specifically, in step S102 in FIG. 11, control unit 101 (property prediction model construction unit 22) performs super-resolution processing on learning generated images 51 obtained by restoring (reconstructing) learning tissue images 41 using generative model 31 to sharpen the learning generated images 51, as explanatory variables (input), and uses the paired quality property values ​​61 as objective variables (output), to learn property prediction model 32.

[0069] As described above, by applying super-resolution technology, it is possible to sharpen generated images 51 without increasing the complexity of generative model 31. As a result, even when generating a large number of generated images, it is possible to generate highly sharp images at a realistic generation speed.

[0070] [Second embodiment] Next, a second embodiment will be described below. In the second embodiment, a quality characteristic value is further predicted from a generated image. 18 is a block diagram showing the functional configuration of a prediction device 1a according to the second embodiment. The prediction device 1a further includes a characteristic prediction model construction unit 22 and a target image acquisition unit 28 in addition to the functional configuration of the prediction device 1 (FIG. 2).

[0071] As in the first embodiment, the generative model construction unit 21 constructs a generative model 31 for generating a structural image of a material using a structural image 41 (learning structural image 41) obtained by imaging a material as learning data. In the second embodiment, the learning structural image 41 is linked to quality characteristic values ​​(actual measured values) in addition to process conditions (actual measured values) and is stored in a storage unit 102 or a storage that can be communicatively connected via a communication unit 103. The quality characteristic values ​​are material characteristic values, such as magnetic flux density Br and coercive force HcJ.

[0072] The characteristic prediction model construction unit 22 uses learning data 42 that pairs material structural images and quality characteristic values ​​to learn and construct a regression model (characteristic prediction model 32) that predicts the quality characteristic values ​​of any structural image (generated image).

[0073] Here, the structural image of the material in the training data 42 is an image (training generated image 51) obtained by restoring (reconstructing) the structural image 41 (see FIG. 3) used in constructing the generative model 31 using the generative model 31. In other words, this is an image obtained by inputting the training structural image 41 to the trained encoder 211 to obtain a latent variable Z, and inputting this latent variable Z to the trained decoder 212.

[0074] 19 is a diagram for explaining the concept of constructing the property prediction model 32 by the property prediction model constructing unit 22. As shown in the figure, the property prediction model constructing unit 22 inputs learning data 42, which is a pair of a structural image of a material (learning generated image 51) and its quality characteristic value, to a learning device 221, and constructs the property prediction model 32 by performing supervised learning.

[0075] The learner 221 uses the training generated images 51 of the training data 42 as explanatory variables (input) and the quality characteristic value 61 paired with the training generated images 51 as a target variable (output), and trains the characteristic prediction model 32 by CNN (convolutional neural network) or other known machine learning. When a tissue image (generated image) with an unknown quality characteristic value is input, the trained characteristic prediction model 32 predicts and outputs the quality characteristic value 61 of the tissue image (generated image).

[0076] In this embodiment, property prediction model 32 is trained using training-use generated images 51 as explanatory variables, but property prediction model 32 can also be trained using training-use tissue images 41 (actual data) as explanatory variables. Also, property prediction model 32 can be trained using latent variable Z when training-use generated images 51 are generated as explanatory variables.

[0077] The image generation-prediction unit 26 generates an arbitrary tissue image and predicts the process conditions and quality characteristic values ​​of the generated tissue image using an image generation-prediction model linked to the generation model 31 constructed by the generation model construction unit 21, the process condition prediction model 33 constructed by the process condition prediction model 33, and the characteristic prediction model 32 constructed by the characteristic prediction model construction unit 22. Specifically, the image generation-prediction unit 26 inputs the latent variable Z randomly sampled by the sampling unit 25 to the decoder 212 of the generation model 31 as shown in FIG. 20, and randomly generates a generated image 51. Then, for the generated generated image 51, the image generation-prediction unit 26 predicts the process conditions 60 using the process condition prediction model 33, and predicts the quality characteristic value 61 using the characteristic prediction model 32.

[0078] FIG. 21 is a diagram showing the distribution of predicted values ​​of the quality characteristic value 61 of the generated image 51 (generated image of a rare earth magnet) randomly generated by the generative model 31. The vertical axis of the graph shows the magnetic flux density Br, and the horizontal axis shows the coercive force HcJ. Both the vertical and horizontal axes are normalized using actual data so that the minimum value is 0 and the maximum value is 1. The white dots "◯" show the distribution of the quality characteristic value of the experimental data, and the black dots "●" show the distribution of the predicted value of the quality characteristic value of the generated image 51 sampled from a large number of randomly generated generated images 51. As shown in the figure, the distribution of the predicted value of the quality characteristic value of the generated image 51 covers almost the entire range of the distribution of the quality characteristic value of the experimental data, so that the generative model 31 can generate generated images 51 with various characteristics. For example, as shown in the figure, it is possible to generate a generated image 51a with high Br-low HcJ characteristics, a generated image 51c with predicted characteristics of low Br-high HcJ, a generated image 51b with intermediate characteristics, etc. When the tissue structure is examined, it can be seen that the grain boundaries in generated image 51a are thin, the grain boundaries in generated image 51c are thick, and the grain boundaries in generated image 51b are intermediate between those in generated image 51a and generated image 51c.

[0079] The target image acquisition unit 28 acquires a structure image (generated image 5) of a material having the target quality characteristic from among the structure images (generated images 51) generated by the image generation-prediction unit 26. Specifically, the target image acquisition unit 28 determines whether or not the quality characteristic value 61 predicted by the image generation-prediction unit 26 satisfies a predetermined target condition. If the result of the determination is that the quality characteristic value 61 satisfies the predetermined target condition, the target image acquisition unit 28 acquires the generated image 51 as the structure image 5 of the material having the target quality characteristic.

[0080] The visualization unit 29 displays the predicted process conditions and quality characteristic values ​​for the structural image 5 of the material having the target quality characteristic acquired by the target image acquisition unit 28 on the display unit 105. The visualization unit 29 may also display the predicted process conditions 60 and quality characteristic values ​​61 as a process condition map or a characteristic map in which the predicted process conditions 60 and quality characteristic values ​​61 are plotted on a graph or the structural image 5, or may display the SHAP value as a SHAP distribution chart or the like in which the SHAP value is plotted on a graph or the structural image 5.

[0081] Next, the flow of the process (prediction method) executed by the prediction device 1a will be described with reference to the flowchart in Fig. 22. The control unit 101 of the prediction device 1a executes the process of each step in Fig. 22 to realize each function of the prediction device 1a.

[0082] The control unit 101 of the prediction device 1a acquires a tissue image 41 (learning tissue image 41) obtained by imaging a certain material from the memory unit 102 or the like as learning data, and learns and constructs a generative model 31 such as VQVAE (step S101a).

[0083] Next, in step S102a, the control unit 101 constructs a process condition prediction model 33, which is a regression model that predicts the process conditions of an arbitrary tissue image (generated image), using the learning data 43 that is a pair of a tissue image and a process condition. Specifically, the control unit 101 constructs the process condition prediction model 33 by learning the process condition prediction model 33 using a learning generated image 51, which is an image obtained by restoring (reconstructing) the learning tissue image 41 by the generative model 31, as an explanatory variable (input) and a pair of the learning generated image 51 as an objective variable (output), thereby predicting the process condition 60 from an arbitrary tissue image (generated image).

[0084] Furthermore, in step S102a, the control unit 101 uses the learning data 42 that is a pair of tissue images and quality characteristic values ​​to construct a property prediction model 32, which is a regression model that predicts the quality characteristic value of an arbitrary tissue image (generated image). Specifically, the control unit 101 constructs the property prediction model 32, which predicts the quality characteristic value 61 from an arbitrary tissue image (generated image), by learning the property prediction model 32 using the learning generated image 51, which is an image obtained by restoring (reconstructing) the learning tissue image 41 by the generation model 31, as an explanatory variable (input) and the paired quality characteristic value 61 as an objective variable (output).

[0085] In steps S101a and S102a, the learning tissue images 41 and the learning generated images 51 used as the learning data may be divided into rectangular images of a predetermined size.

[0086] Next, the control unit 101 creates an image generation-prediction model by linking the generation model 31 constructed in step S101a with the process condition prediction model 33 and the characteristic prediction model 32 constructed in step S102a (step S103a). The image generation-prediction model outputs an arbitrary generated image 51 by the generation model 31, and also outputs a predicted value of the process condition 60 of the generated image 51 by the process condition prediction model 33 and a predicted value of the quality characteristic value 61 of the generated image 51 by the characteristic prediction model 32.

[0087] Here, the operator verifies / tests the image generation-prediction model created in step S103a using validation data and test data, and judges whether the reconstruction accuracy of the generated image 51 and the distribution of the predicted values ​​of the process conditions 60 and the quality characteristic values ​​61 are appropriate (step S104a). If they are inappropriate (step S104a; No), the operator adjusts the hyperparameters of the generation model 31 (step S105a), and reconstructs the generation model 31 (step S101a), the process condition prediction model 33, and the characteristic prediction model 32 (step S102a). The hyperparameters are, for example, the number of epochs.

[0088] In step S104a, it is preferable that the control unit 101 displays the original image and the generated image 51 restored from the original image by the generative model 31 side by side, or displays the distribution of the process conditions 60 predicted by the process condition prediction model 33 and the distribution of the quality characteristic values ​​61 predicted by the characteristic prediction model 32 on the display unit 105, so that the operator can visually check the reconstruction accuracy and the distribution of the predicted values ​​of the process conditions 60 and the quality characteristic values ​​61. Also, an "Adjust" button for moving to a hyperparameter adjustment screen and a "Next" button for moving to the next process may be displayed to accept input of appropriate / inappropriate.

[0089] When the operator determines that the reconstruction accuracy of the generated image 51 and the distribution of the predicted values ​​of the process conditions 60 and the quality characteristic values ​​61 are inappropriate (step S104a; No), he / she operates, for example, an "adjust" button to move to a hyperparameter adjustment screen. In this case, the control unit 101 displays the hyperparameter adjustment screen and accepts adjustments of the number of epochs, etc. from the operator (step S105a). The function of "displaying the hyperparameter adjustment screen and accepting adjustments of the number of epochs, etc. from the operator" executed by the control unit 101 in step S105a is an example of the "adjustment means" according to the present invention. Note that the adjustment of the hyperparameters may be performed by directly rewriting the relevant parts of the program, rather than on the adjustment screen.

[0090] The operator repeats learning, verification, and testing of the generative model 31, the process condition prediction model 33, and the characteristic prediction model 32 while adjusting the hyperparameters (steps S101a to S105a). Then, when the operator determines that the reconstruction accuracy of the generated image 51 and the distribution of the predicted values ​​of the process conditions 60 and the quality characteristic values ​​61 are appropriate (step S104a; Yes), the learning process ends and the process proceeds to the next step S201a.

[0091] In step S104a, the control unit 101 may evaluate the reconstruction accuracy and the distribution of the predicted values ​​of the process conditions and quality characteristic values ​​based on a predetermined criterion and automatically determine whether the reconstruction accuracy is appropriate or inappropriate. For example, the control unit 101 determines that the reconstruction accuracy is appropriate if the reconstruction loss is less than a predetermined threshold and inappropriate if the reconstruction loss is equal to or greater than the threshold. In addition, the control unit 101 determines that the distribution of the predicted values ​​of the process conditions and quality characteristic values ​​is appropriate if a statistical index such as the variance of the distribution satisfies a predetermined condition and inappropriate if the statistical index does not satisfy the predetermined condition. Then, if it is determined that both the reconstruction accuracy and the distribution of the predicted values ​​of the process conditions and quality characteristic values ​​are appropriate (step S104a; Yes), the learning process is terminated and the process proceeds to step S201a.

[0092] FIG. 23 is a graph showing the prediction performance of the trained characteristic prediction model 32 (regression model), with the horizontal axis representing the actual measured value and the vertical axis representing the predicted value for magnetic flux density Br. Normalization processing is performed based on the actual measured value so that the minimum value of both the vertical and horizontal axes is 0 and the maximum value is 1. (a) shows the case where training data is used, (b) shows the case where verification data is used, and (c) shows the case where test data is used. The coefficient of determination R2 of the regression model is R2=0.9805 for the training data, R2=0.4286 for the verification data, and R2=0.4485 for the test data, which shows that a characteristic prediction model 32 with good prediction performance has been created.

[0093] Returning to the explanation of the flowchart in FIG. The control unit 101 samples the latent variable Z to be input to the generation model 31 (decoder 212) of the image generation prediction model (step S201a). In this embodiment, the latent variable Z is any data such as random noise.

[0094] The control unit 101 generates a tissue image (generated image 51) using an image generation-prediction model with the latent variable Z (any data such as random noise) sampled in step 201a as an input, and predicts a process condition 60 and a quality characteristic value 61 for the generated image 51 (step S202a). The output data of the control unit 101 is data including the generated image 51 and the process condition 60 and the quality characteristic value 61 predicted from the generated image 51. The control unit 101 generates a plurality of output data, and judges whether the quality characteristic value 61 for each output data satisfies the target condition (satisfies the target characteristic) (step S203a). The control unit 101 also judges whether a predetermined target number or more of images satisfying the target characteristic have been obtained (step S204a). If the predetermined target number of images satisfying the target characteristic are not obtained (No in either step S203a or step S204a), the process returns to step S201a, and the generation of output data using the image generation-prediction model is repeated. If a certain number of target images cannot be obtained within an effective time, it is also effective to return to START and perform parameter tuning.

[0095] 24 to 26 show examples of generated images 51 of rare earth magnets according to each target characteristic. 24 shows examples of generated images 5a, 5b, and 5c when the target characteristics are high Br (magnetic flux density) and low HcJ (coercive force). The target condition is the following formula (1). In the formula, Brpred is the predicted value of magnetic flux density, and HcJpred is the predicted value of coercive force.

[0096]

number

[0097] The sizes of generated images 5a, 5b, and 5c in Fig. 24 are rectangular small image sizes that are the same as the learning data. The predicted values ​​of the quality characteristics of generated image 5a are Br: 0.853[-], HcJ: 0.0805[-], the predicted values ​​of the quality characteristics of generated image 5b are Br: 0.868[-], HcJ: 0.0779[-], and the predicted values ​​of the quality characteristics of generated image 5c are Br: 0.888[-], HcJ: 0.0894[-].

[0098] 25 shows examples of generated images 5d, 5e, and 5f in the case where low Br (magnetic flux density)-high HcJ (coercive force) are set as the target characteristics. The target condition is the following formula (2).

[0099]

number

[0100] The predicted values ​​of the quality characteristics of generated image 5d in Figure 25 are Br: 0.325[-], HcJ: 0.896[-], the predicted values ​​of the quality characteristics of generated image 5e are Br: 0.345[-], HcJ: 0.915[-], and the predicted values ​​of the quality characteristics of generated image 5f are Br: 0.366[-], HcJ: 0.914[-].

[0101] 26 shows examples of generated images 5g, 5h, and 5i in the case where high Br (magnetic flux density)-high HcJ (coercive force) are set as the target characteristics. The target condition is expressed by the following formula (3).

[0102]

number

[0103] The predicted values ​​of the quality characteristics of generated image 5g in Figure 26 are Br: 0.817[-], HcJ: 0.811[-], the predicted values ​​of the quality characteristics of generated image 5h are Br: 0.868[-], HcJ: 0.821[-], and the predicted values ​​of the quality characteristics of generated image 5i are Br: 0.817[-], HcJ: 0.815[-].

[0104] Returning to the explanation of the flowchart in FIG. When the number of tissue images 5 that satisfy the target characteristics reaches a predetermined target number (Yes in both steps S203a and S204a), the control unit 101 then performs visualization processing (step S205a).

[0105] In the visualization process, the control unit 101 displays the predicted results of the process conditions 60 and the quality characteristic values ​​61 for the tissue image 5 on the display unit 105. The control unit 101 may also display the predicted process conditions 60 and the quality characteristic values ​​61 as a process condition map or a characteristic map in which the predicted process conditions 60 and the quality characteristic values ​​61 are plotted on a graph or the tissue image 5, or may display a SHAP distribution diagram in which the SHAP values ​​are plotted on a graph or the tissue image 5.

[0106] FIG. 27 shows a distribution map of SHAP values, which is an example of visualization processing of quality characteristic values. FIG. 27(a) shows a SHAP distribution map of magnetic flux density Br and a SHAP distribution map of coercive force HcJ of generated image 5g of FIG. 26. FIG. 27(b) shows a SHAP distribution map of magnetic flux density Br and a SHAP distribution map of coercive force HcJ of generated image 5h of FIG. 26. FIG. 27(c) shows a SHAP distribution map of magnetic flux density Br and a SHAP distribution map of coercive force HcJ of generated image 5i of FIG. 26. In this way, by visualizing the generated image and its quality characteristics, it becomes easier to evaluate the structural structure and quality characteristics of the material.

[0107] As described above, the prediction device 1a of the second embodiment trains the generation model 31 using the structural image obtained by imaging the material as training data, trains the characteristic prediction model 32 using the paired data of the structural image of the material and the quality characteristic value as training data, and constructs an image generation-prediction model by linking the generation model 31, the characteristic prediction model 32, and the process condition prediction model 33. This makes it possible to predict the process conditions and the quality characteristic value for the generated structural image. In addition, by acquiring a structural image 5 having the target quality characteristic from among the generated structural images and visualizing the structural image 5, the process conditions, and the quality characteristic value, it is possible to grasp the structural structure having the target characteristic and the process conditions for achieving the target characteristic.

[0108] As in the first embodiment, super-resolution processing may be applied to the generated image when generating the image (step S202a in FIG. 22) or when learning the characteristic prediction model 32 or the process condition prediction model 33 (step S102a in FIG. 22). [Third embodiment] Next, a third embodiment will be described. In the second embodiment, the generated images are generated randomly, but it is more efficient to generate more generated images close to the target quality characteristics. In the third embodiment, an improvement is made to the image generation method.

[0109] 28 is a block diagram showing the functional configuration of a prediction device 1b according to the third embodiment. The prediction device 1b further includes an autoregressive model construction unit 24 in addition to the configuration of the prediction device 1a (FIG. 18).

[0110] The autoregressive model construction unit 24 learns and constructs an autoregressive model 34 (conditional probability distribution) based on a latent variable Z obtained by inputting the learning tissue image 41 into a trained generative model 31 (encoder 211).

[0111] The learning of the autoregressive model 34 will be explained. First, the embedding vector V in VQVAE in FIG. 3 will be explained in detail. As shown in FIG. 3, D values ​​are stored in each embedding vector V, and there are K embedding vectors V. K and D are hyperparameters that are set in advance by the user. When VQVAE is trained, the D values ​​contained in each embedding vector V are optimized. As shown in FIG. 3, the latent variable Z can be expressed as a two-dimensional array composed of the indexes of the embedding vector V. For example, the "2" contained in the latent variable Z indicates the embedding vector V2 with the second index.

[0112] After learning the VQVAE in FIG. 3 (after constructing the generative model 31), the autoregressive model construction unit 24 inputs the learning tissue image 41 to the encoder 211 of the trained generative model 31 as shown in FIG. 29 to obtain a two-dimensional array of latent variables Z. The number of two-dimensional arrays of latent variables Z obtained is equal to the number of learning tissue images 41. In the two-dimensional array of latent variables, indexes are not randomly arranged, but are arranged according to a certain rule / pattern. The autoregressive model construction unit 24 learns this rule / pattern using an autoregressive model.

[0113] Specifically, the autoregressive model construction unit 24 converts the two-dimensional array of the latent variable Z into a one-dimensional data string, regards this data string as time-series data, and learns a conditional probability distribution (autoregressive model 34) according to the history of the indexes that have already been filled, as shown in Fig. 30. As shown in Fig. 30, the autoregressive model 34 has a structure composed of, for example, a neural network 341 and a softmax function 342.

[0114] The sampling unit 25 samples the latent variable Z using the autoregressive model 34 (conditional probability distribution) learned by the autoregressive model construction unit 24. Specifically, starting from the top left of the two-dimensional array of latent variables, the indices to be entered into each array are sampled in order based on the autoregressive model 34 (conditional probability distribution). At the time of sampling, as shown in the conditional probability distribution in FIG. 30, there are indices with a high probability of being entered into the array and indices with a low probability of being entered into the array.

[0115] Particularly in this embodiment, the sampling unit 25 does not perform sampling using the trained autoregressive model 34 as is, but performs sampling after adjusting the heights of the peaks and valleys of the conditional probability distribution included in the autoregressive model 34. This adjusts the probability of the index entering the array of the latent variable Z.

[0116] Generally, material structures contain precipitates, grain boundaries, voids, etc., but the parent phase accounts for the largest proportion. Therefore, indices with a high probability of being included in the array of latent variables Z (the peaks of conditional probability) tend to contain structure information with a high proportion of the parent phase. Conversely, indices with a low probability of being included in the array of latent variables Z (the valleys of conditional probability) tend to contain structure information with a low proportion of the parent phase.

[0117] As described above, by adjusting the height of the peaks and valleys of the conditional probability distribution, the probability of the indexes entering the array of the latent variable Z is adjusted, making it possible to generate generated images focusing on specific material structures. For example, it is possible to generate many microstructure images that include material structures that are closely related to the target quality characteristic, or generate fewer microstructure images that include material structures that are weakly related to the target quality characteristic.

[0118] Specifically, as shown in Fig. 31, the sampling unit 25 modifies the softmax function 342 expressing the conditional probability distribution into a coefficient-added softmax function 342a by adding a coefficient T. The coefficient-added softmax function 342a is expressed as follows.

[0119]

number

[0120] Then, the sampling unit 25 adjusts the height of the peaks and valleys of the conditional probability distribution by changing the coefficient T of the coefficientd softmax function 342a, and samples the latent variable Z based on the autoregressive model 34a having the adjusted conditional probability distribution (softmax function 342a).

[0121] FIG. 32 is a diagram showing the change in function when the coefficient of the softmax function 342a is changed. As shown in FIG. 32, the larger the value of the coefficient T, the lower the peak part of the softmax function 342a (conditional probability distribution) and the higher the valley part. As described above, the peak part of the conditional probability tends to include structural information with a high ratio of the matrix. Therefore, when sampling is performed with the coefficient T of the softmax function 342a set to a large value, the peak part of the conditional probability distribution becomes lower, and an index including structural information with a high ratio of the matrix is ​​less likely to appear when an image is generated. In addition, since the valley part of the conditional probability distribution becomes higher, an index including structural information with a low ratio of the matrix (with a high ratio of grain boundaries, precipitates, and voids) is more likely to appear. Conversely, when sampling is performed with the coefficient T of the softmax function 342a set to a small value, the peak part of the conditional probability distribution becomes lower, and an index including structural information with a high ratio of the matrix is ​​more likely to appear when an image is generated. In addition, since the valley of the conditional probability distribution becomes lower, an index containing structural information with a small proportion of the matrix (with a large proportion of grain boundaries, precipitates, and voids) is less likely to appear.

[0122] The coefficient T can be set appropriately according to the target quality characteristic. For example, when a high coercive force is set as the target characteristic, since grain boundaries are related to coercive force (see FIG. 21), it is efficient to generate many structural images with a small proportion of the matrix. In other words, it is desirable to set the coefficient T large. When sampling is performed with the coefficient T set large, the valley of the conditional probability distribution becomes higher, so that indices containing structural information with a small proportion of the matrix (a large proportion of grain boundaries) tend to appear, and many structural images with a small proportion of the matrix (a large proportion of grain boundaries) are generated. In this way, sampling is performed after controlling the proportion of the matrix using the coefficient T according to the target characteristic.

[0123] The image generation-prediction unit 26 inputs the latent variable Z sampled by the sampling unit 25 to the decoder 212 of the generation model 31, and generates a generated image 51. The image generation-prediction unit 26 also predicts a quality characteristic value 61 for the generated generated image 51 using the characteristic prediction model 32, and predicts a process condition 60 using the process condition prediction model 33. The distribution of the quality characteristic values ​​61 of the generated image 51 generated by the image generation-prediction unit 26 is not a uniform distribution, but a biased distribution (see FIG. 34).

[0124] Other functional configurations of the prediction device 1b (generative model construction unit 21, characteristic prediction model construction unit 22, process condition prediction model construction unit 23, target image acquisition unit 28, visualization unit 29) are similar to the functional configuration of the prediction device 1a (FIG. 18), so description thereof will be omitted. Also, the hardware configuration of the prediction device 1b is similar to the hardware configuration of the prediction devices 1 and 1a (FIG. 1).

[0125] Next, the flow of the process (prediction method) executed by the prediction device 1b will be described with reference to the flowchart in Fig. 33. The control unit 101 of the prediction device 1b executes the process of each step in Fig. 33 to realize each function of the prediction device 1b.

[0126] Steps S101a to S105a in Fig. 33 are the same as steps S101a to S105a in Fig. 22. That is, the control unit 101 learns and constructs a generative model 31 such as VQVAE using the learning tissue image 41 as learning data (step S101a). Next, the control unit 101 learns and constructs a process condition prediction model 33 using learning data 43 that is a pair of a learning generated image 51, which is an image obtained by restoring (reconstructing) the learning tissue image 41 by the generative model 31, and a process condition, and learns and constructs a characteristic prediction model 32 using learning data 42 that is a pair of the learning generated image 51 and a quality characteristic value (step S102a). Then, the control unit 101 creates an image generation-prediction model that combines the generative model 31 built in step S101a with the process condition prediction model 33 and the characteristic prediction model 32 built in step S102a (step S103a).

[0127] The operator verifies / tests the image generation-prediction model created in step S103a using validation data and test data, and judges whether the reconstruction accuracy of the generated image 51 and the distribution of the predicted values ​​of the process conditions 60 and the quality characteristic values ​​61 are appropriate (step S104a). If they are inappropriate (step S104a; No), the operator adjusts the hyperparameters of the generation model 31 (step S105a), and reconstructs the generation model 31 (step S101a), the process condition prediction model 33, and the characteristic prediction model 32 (step S102a).

[0128] When the operator judges that the reconstruction accuracy of the generated image 51 and the distribution of the predicted values ​​of the process conditions 60 and the quality characteristic values ​​61 are inappropriate (step S104a; No), the operator operates, for example, an "adjust" button to move to a hyperparameter adjustment screen. In this case, the control unit 101 displays the hyperparameter adjustment screen and accepts adjustments such as the number of epochs from the operator (step S105a). The function of "displaying the hyperparameter adjustment screen and accepting adjustments such as the number of epochs from the operator" executed by the control unit 101 in step S105a is an example of the "adjustment means" according to the present invention. Note that the adjustment of the hyperparameters may be performed by directly rewriting the corresponding parts of the program, not on the adjustment screen. The operator repeats learning, verification, and testing of the generation model 31, the process condition prediction model 33, and the characteristic prediction model 32 while adjusting the hyperparameters (steps S101a to S105a). Then, when the operator judges that the reconstruction accuracy of the generated image 51 and the distribution of the predicted values ​​of the process conditions 60 and the quality characteristic values ​​61 are appropriate (step S104a; Yes), the learning process ends and proceeds to the next step S106a.

[0129] In step S106a, the control unit 101 learns and constructs the autoregressive model 34 (conditional probability distribution). Specifically, the control unit 101 first inputs the learning tissue image 41 to the encoder 211 of the learned generative model 31 to obtain a two-dimensional array of latent variables Z (FIG. 29). The two-dimensional arrays of latent variables Z are obtained as many as the number of learning tissue images 41. Next, the control unit 101 learns the autoregressive model 34 based on the obtained latent variables Z. Specifically, as shown in FIG. 30, the control unit 101 converts the two-dimensional array of latent variables Z into a one-dimensional data string, regards this data string as time-series data, and learns a conditional probability distribution (autoregressive model 34) according to the history of the indexes that have already been filled. As shown in FIG. 30, the autoregressive model 34 has a structure composed of, for example, a neural network 341 and a softmax function 342.

[0130] Next, the control unit 101 samples the latent variable Z using the trained autoregressive model 34 (step S201a). At this time, the control unit 101 does not sample the latent variable Z using the trained autoregressive model 34 as it is, but adjusts the conditional probability distribution included in the trained autoregressive model 34, and samples the latent variable Z based on the adjusted conditional probability distribution. Specifically, the control unit 101 first modifies the softmax function 342 expressing the conditional probability distribution to a coefficient-added softmax function 342a to which a coefficient T is added. Then, the control unit 101 sets the coefficient T of the coefficient-added softmax function 342a to a predetermined value. This adjusts the height of the peaks and valleys of the conditional probability distribution included in the autoregressive model 34 (FIG. 32). The coefficient T may be set appropriately according to the target quality characteristic. Then, the control unit 101 samples the latent variable Z based on the autoregressive model 34a (FIG. 31) having the coefficient-added softmax function 342a to which the coefficient is set.

[0131] The subsequent processes (steps S202a to S205a in FIG. 33) are the same as those in the second embodiment (steps S202a to S205a in FIG. 22). That is, the control unit 101 generates a tissue image (generated image 51) by the image generation-prediction model using the latent variable Z sampled in step 201 as an input, and predicts the quality characteristic value 61 and the process condition 60 for the generated image 51 (step S202a). Next, the control unit 101 generates a plurality of output data, and determines whether the quality characteristic value 61 for each output data satisfies the target condition (satisfies the target characteristic) (step S203a), and also determines whether or not a predetermined target number or more of images that satisfy the target characteristic have been obtained (step S204a). If the number of images that satisfy the target characteristic does not reach the predetermined target number (No in either step S203a or step S204a), the process returns to step S201a, and the generation of output data by the image generation-prediction model is repeated. Then, when the number of tissue images 5 satisfying the target characteristics reaches a predetermined target number (Yes in both steps S203a and S204a), the control unit 101 performs visualization processing (step S205a) and ends the processing. In the visualization processing, the control unit 101 displays the prediction results of the process conditions 60 and the quality characteristic values ​​61 for the tissue images 5 satisfying the target characteristics on the display unit 105.

[0132] FIG. 34 is a graph showing the distribution of quality characteristic values ​​when the coefficient T is adjusted. The white points "◯" are plots of the predicted quality characteristic results when a generated image is generated by sampling the latent variable Z with the coefficient T of the coefficient softmax function 342a of the autoregressive model 34a as a standard value. The black points "●" are plots of the predicted quality characteristic results when a generated image is generated by sampling the latent variable Z with the coefficient T of the coefficient softmax function 342a of the autoregressive model 34a as a predetermined value larger than the standard value. From FIG. 34, it can be seen that the distribution of quality characteristic values ​​is biased toward the high coercivity side for the points "●" where the coefficient T is a large value. In this way, by adjusting the coefficient T, more generated images close to the target characteristic ("high coercivity" in the case of FIG. 25) can be generated.

[0133] The third embodiment has been described above. According to the third embodiment, the autoregressive model 34 is trained based on the latent variable Z obtained by inputting the learning tissue image 41 into the generative model 31 (encoder 211), and the latent variable Z is sampled based on the conditional probability distribution included in the autoregressive model 34 to generate a tissue image. At this time, by adjusting the heights of the peaks and valleys of the conditional probability distribution and then sampling the latent variable Z, it is possible to efficiently generate a tissue image having characteristics close to the target quality characteristics. This makes it possible to efficiently develop materials having the desired quality characteristics.

[0134] As in the first and second embodiments, super-resolution processing may be applied to the generated image when generating the image (step S202a in FIG. 33) or when learning the characteristic prediction model 32 or the process condition prediction model 33 (step S102a in FIG. 33).

[0135] Although the preferred embodiment of the present invention has been described above with reference to the accompanying drawings, the present invention is not limited to the above examples. It is clear that a person skilled in the art can come up with various modified or altered examples within the scope of the technical ideas disclosed in this application, and it is understood that these also naturally belong to the technical scope of the present invention. [Explanation of symbols]

[0136] 1, 1a, 1b... Prediction device 21...Generative model construction section 22...………………Characteristics prediction model construction section 23...Process condition prediction model construction section 24. Autoregressive model building section 25...Sampling section 26...Image generation and prediction section 28...Target image acquisition section 29……………………Visualization section 31. Generative model 32…………………Characteristics prediction model 33…………………Process condition prediction model 34. Autoregressive model 341...Neural network 342...Softmax function 342a……………Softmax function with coefficients 41…………………Tissue image (actual data) 42. Paired data of tissue image and quality characteristic value 43...................................Paired data of tissue image and process conditions 51...Tissue image (generated image) 60…………………Process conditions 61…………………Quality characteristic value 5. Tissue image with target characteristics

Claims

1. a generative model constructing unit that constructs a generative model for generating a structural image of a material using a structural image obtained by imaging the material as learning data; a process condition prediction model constructing unit that constructs a process condition prediction model, which is a regression model that predicts the process conditions of any structural image, by learning paired data of the structural image of the material and the process conditions; an image generation and prediction unit that generates a structural image of the material and predicts process conditions of the structural image by using the generative model constructed by the generative model construction unit and the process condition prediction model constructed by the process condition prediction model construction unit; A prediction device comprising:

2. The number of epochs in the generative model construction unit and the process condition prediction model construction unit is determined based on the reconstruction accuracy of the structure image generated by the generative model and the distribution of the process conditions predicted by the process condition prediction model. The prediction device according to claim 1 .

3. an adjustment unit for adjusting the number of epochs in the generation model construction unit and the process condition prediction model construction unit; The prediction device according to claim 2 , further comprising:

4. The image generation-prediction unit inputs an image obtained by performing super-resolution processing on the structural image of the material generated by the generative model into the process condition prediction model, and predicts the process conditions. The prediction device according to claim 1 .

5. The process condition prediction model construction unit uses, as a structural image of the material to be learned, an image obtained by performing super-resolution processing on an image reconstructed by the generative model of the structural image of the material. The prediction device according to claim 1 or 4.

6. a characteristic prediction model constructing unit that constructs a characteristic prediction model, which is a regression model that predicts the quality characteristic value of an arbitrary structural image, by learning paired data of a structural image of the material and a quality characteristic value, The image generation-prediction unit predicts quality characteristic values ​​of the structure image of the material generated using the characteristic prediction model. The prediction device according to claim 1 .

7. a target image acquisition unit that acquires a structural image of the material having a target quality characteristic from the structural image of the material generated by the image generation-prediction unit; The prediction device according to claim 6, further comprising:

8. The target image acquisition unit determines whether or not a quality characteristic value of a tissue image generated by the generative model predicted by the image generation-prediction unit satisfies a predetermined target condition, and when the quality characteristic value satisfies the predetermined target condition, sets the generated tissue image as a tissue image of the material having the target quality characteristic. The prediction device according to claim 7 .

9. a visualization unit that visualizes process conditions and predicted quality characteristic values ​​for a structure image of the material having the target quality characteristic; The prediction device according to claim 8, further comprising:

10. The number of epochs in the generative model construction unit and the characteristic prediction model construction unit is determined based on the reconstruction accuracy of the tissue image generated by the generative model and the distribution of quality characteristic values ​​predicted by the characteristic prediction model. The prediction device according to claim 6 .

11. an adjustment means for adjusting the number of epochs in the generative model construction unit and the characteristic prediction model construction unit; The prediction device according to claim 10, further comprising:

12. The image generation-prediction unit inputs an image obtained by performing super-resolution processing on the structural image of the material generated by the generative model into the characteristic prediction model, and predicts a quality characteristic value. The prediction device according to claim 6 .

13. The characteristic prediction model construction unit uses, as a structural image of the material to be learned, an image obtained by performing super-resolution processing on an image reconstructed from a structural image of the material by the generative model. The prediction device according to claim 6 or 12.

14. The image generation-prediction unit generates more tissue images including tissues that are strongly related to the target quality characteristic, or generates fewer tissue images including tissues that are weakly related to the target quality characteristic, compared to a case where tissue images are generated randomly.

8. The prediction device according to claim 6 or 7.

15. The distribution of quality characteristic values ​​of the tissue images generated by the image generation-prediction unit is biased.

8. The prediction device according to claim 6 or 7.

16. The computer A step of constructing a generative model for generating a structural image of a material using a structural image obtained by imaging the material as learning data; A step of constructing a process condition prediction model, which is a regression model for predicting the process conditions of any structural image, by learning paired data of the structural image of the material and the process conditions; generating a structural image of the material and predicting process conditions of the structural image using the generative model constructed in the step of constructing the generative model and the process condition prediction model constructed in the step of constructing the process condition prediction model; A prediction method including:

17. Computer, a generative model constructing unit that constructs a generative model for generating a structural image of a material using a structural image obtained by imaging the material as learning data; a process condition prediction model constructing unit that constructs a process condition prediction model, which is a regression model that predicts the process conditions of any structural image, by learning paired data of the structural image of the material and the process conditions; an image generation-prediction unit that generates a structural image of the material and predicts process conditions for the structural image by using the generative model constructed by the generative model construction unit and the process condition prediction model constructed by the process condition prediction model construction unit; A program to function as a

Images