Alloy performance prediction method fusing vision-language-process multi-modal data
By fusing visual, linguistic, and process-related multimodal data and employing self-supervised learning and random forest regressors, the problem of insufficient utilization of multimodal data in alloy material performance prediction is solved, achieving high-precision and low-cost alloy performance prediction.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- ZHEJIANG UNIV
- Filing Date
- 2025-12-22
- Publication Date
- 2026-05-01
AI Technical Summary
Existing methods for predicting the properties of alloy materials rely on single-modal data, neglecting the complementary value of multimodal data. This results in high manual annotation costs and limited generalization performance, making it difficult to comprehensively characterize complex alloy properties.
By integrating visual, linguistic, and technological multimodal data, and employing self-supervised learning to encode unstructured data, and using a random forest regressor to predict alloy properties, the reliance on manual annotation is reduced.
It significantly improves the accuracy and efficiency of alloy property prediction, reduces errors, lowers computational costs, and is applicable to a wide range of material systems.
Smart Images
Figure CN121963968A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of machine learning and alloy performance prediction, and relates to an alloy performance prediction method that integrates visual, linguistic, and process multimodal data. Background Technology
[0002] The mechanical properties of alloy materials are core indicators determining their feasibility and safety in engineering applications, and are influenced by various factors such as elemental composition, processing parameters, and microstructure. Traditional materials research and development typically employs a trial-and-error approach, preparing samples by repeatedly adjusting process parameters such as temperature and time during heat treatment. Furthermore, determining the mechanical properties of these samples (such as yield strength and ultimate tensile strength) requires tedious tensile testing. This method is time-consuming, costly, and resource-intensive.
[0003] To address the aforementioned issues, computational materials science (CTS) techniques for predicting the performance of alloy materials have been developed. Early computational techniques relied on classical physical metallurgical equations and finite element analysis. These attempted to solve mathematical equations describing the behavior of microscopic particles or macroscopic physical phenomena to predict the properties and evolution of materials. However, these methods heavily depend on prior human knowledge and often struggle to construct accurate equation models for complex nonlinear systems. Furthermore, their computational processes are complex and costly. With the continuous accumulation of experimental data on alloy material composition, processing, and microstructure, as well as the development of artificial intelligence methods, computational materials science has begun to utilize this experimental data to train AI-based performance prediction models. These models can model and uncover complex nonlinear relationships, overcoming the limitations of traditional computational materials science methods in effectively processing and utilizing unstructured data such as images and text. Domestic researchers first screened five key factors affecting the elongation of copper alloys, then established a support vector machine regression model, and used Bayesian optimization methods to predict the elongation of copper alloys, successfully obtaining copper alloys with excellent performance. International researchers first used scanning electron microscopy (SEM) to obtain microstructure images of magnesium alloys, then trained a convolutional neural network-based model to directly predict the mechanical properties of magnesium alloys from these images. In addition, some researchers treated the composition of high-entropy alloys as unstructured text data, training Bidirectional Encoder Representations from Transformers (BERT) to predict the yield strength of the alloys. While these methods have successfully constructed artificial intelligence models to predict the mechanical properties of alloys, they largely neglect the complementary value of multimodal data—such as structured process data, visual microstructure data, and unstructured textual descriptions of microstructure—for alloy performance prediction.
[0004] Furthermore, existing studies attempting to utilize two or more modal data typically rely on supervised learning methods to encode alloy material features. This approach not only introduces high manual annotation costs but also fails to comprehensively and deeply represent the complex information of the data, thus limiting the model's generalization performance. Therefore, there is an urgent need for an alloy performance prediction method that can effectively integrate multi-source heterogeneous data and automatically encode unstructured key features of alloy materials without relying on manual annotation, in order to overcome the current technological bottlenecks. Based on these needs, this invention proposes an alloy performance prediction method that integrates visual, linguistic, and process multimodal data. Summary of the Invention
[0005] To address the above problems, this invention provides a method for predicting alloy properties by integrating visual, linguistic, and process multimodal data.
[0006] This invention simultaneously considers process parameters in alloy preparation, visual information from microstructure images, and linguistic information describing the microstructure images. It trains a multimodal vector representation for predicting alloy properties using a ResNet50 visual encoder with open weights and a BERT language encoder. This multimodal vector representation is then input into a random forest regression model for training, resulting in an alloy property predictor that significantly reduces prediction errors.
[0007] The technical solution of this invention is: A method for predicting alloy properties by integrating visual, linguistic, and process multimodal data includes the following steps: 1. Construct a multimodal dataset for alloy materials, including process parameters such as temperature and time for solution treatment and aging treatment, visual information from SEM images, and textual information describing SEM images, along with the corresponding true values of the alloy's mechanical properties. Taking Inconel 718 nickel-based superalloy as an example, the steps for obtaining and constructing a multimodal dataset are as follows: Step 1-1: Collect process parameters during material preparation, including solution treatment time, solution treatment temperature, number of aging treatment stages, aging treatment time, and aging treatment temperature.
[0008] Steps 1-2: After the material preparation is completed, the alloy base material is first processed by electrical discharge machining to obtain tensile specimens. Then, the tensile specimens are polished step by step with sandpaper, followed by mechanical polishing of the specimens to a mirror finish using diamond polishing paste. Finally, the mechanically polished specimens are etched using an electrolytic etching solution.
[0009] Steps 1-3: Place the etched material under a scanning electron microscope to photograph its typical microstructure and collect SEM image data. During this process, repeat sampling of typical images at different locations is performed for data augmentation to improve the stability of subsequent model training. Natural language is used to describe the SEM images, collecting linguistic data on the descriptions.
[0010] Steps 1-4: Stretch the samples using a tensile testing machine and record their corresponding mechanical properties, including yield strength and ultimate tensile strength. After constructing the process parameters such as solution treatment and aging temperatures and times, SEM image visual information, SEM image text description information, and mechanical property data into corresponding quadruple sets, divide the training and test sets uniformly at a 9:1 ratio, ensuring that there is no overlap between the training and test set data.
[0011] 2. The visual encoder ResNet50 model is trained using contrastive learning, and the language encoder BERT model is trained using Masked Language Model (MLM) to obtain vector encodings of SEM images and their language descriptions. This invention employs a self-supervised learning training method to encode unstructured visual and language data, achieving the corresponding data encoding without manual annotation. The steps for encoding unstructured visual and language data are as follows: Step 2-1: Flip the original SEM image. The flipped image and the original image are used as positive samples in the contrastive learning process, while the flipped image and other images are used as negative samples. Using the positive and negative sample images as input, we obtain positive and negative sample vectors. We then train the ResNet50 visual encoder model using the InfoNCE loss function. The InfoNCE loss function formula is as follows: (1) Where N is the number of samples, and sim represents the cosine similarity. and These are the positive and negative sample encoding vectors of the selected sample i, respectively, and τ is a temperature parameter used to control the similarity score distribution.
[0012] Step 2-2: Input the original image into the trained ResNet50 to obtain the corresponding SEM visual image encoding vector.
[0013] Steps 2-3: Divide the original SEM image description language information into a token sequence, mask the random positions in the token sequence, and train BERT using the MLM method and the MSE loss function. The MSE loss function formula is as follows: (2) Where M is the set of hidden tokens, and y i These are the predicted value and the actual value.
[0014] Steps 2-4: Input the original SEM image description language information into the BERT after training to obtain the corresponding SEM description language encoding vector.
[0015] 3. Combine the process parameters such as the temperature and time of solution treatment and aging treatment with the vector codes obtained in step 2, and train a random forest regressor based on the corresponding mechanical properties of the alloy; This invention uses a random forest regressor as the model for predicting alloy properties. This hybrid architecture of deep learning and machine learning maintains excellent performance while reducing computational overhead and cost. The training steps for the alloy property predictor are as follows: Step 3-1: The process parameters collected in Step 1 and the visual and language encoding vectors obtained in Step 2 are spliced and fused to construct a multimodal vector for prediction. This vector integrates three key pieces of information: material preparation conditions, microstructure, and semantic description, laying a complete data foundation for subsequent high-precision prediction.
[0016] Step 3-2: Use the multimodal feature vectors generated in the above steps as input features, aligning them with the alloy mechanical property data collected in Step 1. Input the training set into the random forest regression algorithm. Guided by the objective of minimizing the mean square error of each decision tree in the forest, perform model fitting, and finally train an accurate alloy property predictor based on random forest. The calculation formula for the random forest predictor is: (3) Where B represents the number of decision trees. This represents the output of the decision tree, where x represents the multimodal vector input.
[0017] 4. The trained random forest regressor is used to predict alloy properties; Input the data from the test set into the predictor obtained in step 3 as required, and the model output will be the predicted value of the alloy properties. After collecting the quaternion information of the material to be predicted in step 1, execute steps 2 and 3 in sequence to obtain the corresponding alloy property predictor to predict the corresponding properties.
[0018] The features of this invention are: Comprehensive multimodal data fusion: Unlike existing technologies that typically use only a single type of process data or image data, this invention is the first to collaboratively fuse and jointly model structured process data, unstructured SEM image data, and their derived textual description data. This fusion fully leverages the complementarity between different modal data to construct a more comprehensive and three-dimensional digital characterization of the alloy state.
[0019] Broad versatility: The technical solution of this invention does not depend on a specific alloy system or process path, and its multimodal fusion framework and feature learning mechanism are applicable to a wide range of material systems. Furthermore, the "deep representation + random forest" hybrid model employed ensures high prediction accuracy while also offering the advantage of efficient training.
[0020] The innovative point of this invention is: Multimodal data fusion construction: This invention is the first to integrate structured process parameters, unstructured SEM images, and unstructured SEM image text descriptions for alloy property prediction. This multimodal data fusion fully utilizes the complementarity and collaboration between different modal data to construct a more comprehensive and three-dimensional digital characterization of the material state.
[0021] Self-supervised representation: This invention encodes images and text using self-supervised learning techniques such as contrastive learning and masked language modeling. The model can automatically learn high-level deep feature vectors with strong semantic information from the original images and text, eliminating the dependence on traditional handmade features and greatly improving the quality of feature representation and the prediction accuracy of the model.
[0022] Balancing High Performance and Efficiency: Random forests, as a powerful ensemble learning algorithm, typically achieve equal or even better prediction accuracy with significantly lower computational overhead than deep regression networks when processing well-represented low- to medium-dimensional features generated by deep networks. This results in faster overall model training, lower inference costs, and less reliance on hyperparameter tuning, making it easier to deploy and run. The advantages of this invention are: Firstly, it integrates multimodal data such as structured process parameters, unstructured SEM images, and unstructured SEM image text descriptions for alloy performance prediction. This multimodal data fusion fully utilizes the complementarity and collaboration between different modal data, constructing a more comprehensive and three-dimensional digital representation of the material state. Secondly, this invention encodes images and text using self-supervised learning techniques such as contrastive learning and masked language modeling. The model can automatically learn high-level deep feature vectors with strong semantic information from the original images and text, eliminating reliance on traditional hand-crafted features and greatly improving the quality of feature representation and the model's prediction accuracy. Random forests, as a powerful ensemble learning algorithm, typically achieve equal or even better prediction accuracy with far less computational overhead than deep regression networks when processing well-represented low- to medium-dimensional features generated by deep networks. This makes the overall model of this invention faster to train, lower inference costs, and less dependent on hyperparameter tuning. Using this method to predict the mechanical properties of alloys can significantly reduce errors. Attached Figure Description
[0023] Figure 1This is a flowchart of the method of the present invention. Detailed Implementation
[0024] The technical solution of the present invention will be clearly and completely explained and described below.
[0025] A method for predicting alloy properties by integrating visual, linguistic, and process multimodal data includes the following steps: 1. Construct a multimodal dataset for alloy materials, including process parameters such as temperature and time for solution treatment and aging treatment, visual information from SEM images, and textual information describing SEM images, along with the corresponding true values of the alloy's mechanical properties. Taking Inconel 718 nickel-based superalloy as an example, the steps for obtaining and constructing a multimodal dataset are as follows: Step 1-1: Collect process parameters during material preparation, including solution treatment time, solution treatment temperature, number of aging treatment stages, aging treatment time, and aging treatment temperature.
[0026] Steps 1-2: After the material preparation is completed, the alloy base material is first processed by electrical discharge machining to obtain tensile specimens. Then, the tensile specimens are polished step by step with sandpaper, followed by mechanical polishing of the specimens to a mirror finish using diamond polishing paste. Finally, the mechanically polished specimens are etched using an electrolytic etching solution.
[0027] Steps 1-3: Place the etched material under a scanning electron microscope to photograph its typical microstructure and collect SEM image data. During this process, repeat sampling of typical images at different locations is performed for data augmentation to improve the stability of subsequent model training. Natural language is used to describe the SEM images, collecting linguistic data on the descriptions.
[0028] Steps 1-4: Stretch the samples using a tensile testing machine and record their corresponding mechanical properties, including yield strength and ultimate tensile strength. After constructing the process parameters such as solution treatment and aging temperatures and times, SEM image visual information, SEM image text description information, and mechanical property data into corresponding quadruple sets, divide the training and test sets uniformly at a 9:1 ratio, ensuring that there is no overlap between the training and test set data.
[0029] 2. A ResNet50 visual encoder model is trained using contrastive learning, and a BERT language encoder model is trained using masked language modeling to obtain vector encodings of SEM images and their language descriptions; This invention employs a self-supervised learning training method to encode unstructured visual and language data, achieving the corresponding data encoding without manual annotation. The steps for encoding unstructured visual and language data are as follows: Step 2-1: Flip the original SEM image. The flipped image and the original image are used as positive samples in the contrastive learning process, while the flipped image and other images are used as negative samples. Using the positive and negative sample images as input, we obtain positive and negative sample vectors. We then train the ResNet50 visual encoder model using the InfoNCE loss function. The InfoNCE loss function formula is as follows: (1) Where N is the number of samples, and sim represents the cosine similarity. and These are the positive and negative sample encoding vectors of the selected sample i, respectively, and τ is a temperature parameter used to control the similarity score distribution.
[0030] Step 2-2: Input the original image into the trained ResNet50 to obtain the corresponding SEM visual image encoding vector.
[0031] Steps 2-3: Divide the original SEM image description language information into a token sequence, mask the random positions in the token sequence, and train BERT using the MLM method and the MSE loss function. The MSE loss function formula is as follows: (2) Where M is the set of hidden tokens, and y i These are the predicted value and the actual value.
[0032] Steps 2-4: Input the original SEM image description language information into the BERT after training to obtain the corresponding SEM description language encoding vector.
[0033] 3. Combine the process parameters such as the temperature and time of solution treatment and aging treatment with the vector codes obtained in step 2, and train a random forest regressor based on the corresponding mechanical properties of the alloy; This invention uses a random forest regressor as the model for predicting alloy properties. This hybrid architecture of deep learning and machine learning maintains excellent performance while reducing computational overhead and cost. The training steps for the alloy property predictor are as follows: Step 3-1: The process parameters collected in Step 1 and the visual and language encoding vectors obtained in Step 2 are spliced and fused to construct a multimodal vector for prediction. This vector integrates three key pieces of information: material preparation conditions, microstructure, and semantic description, laying a complete data foundation for subsequent high-precision prediction.
[0034] Step 3-2: Use the multimodal feature vectors generated in the above steps as input features, aligning them with the alloy mechanical property data collected in Step 1. Input the training set into the random forest regression algorithm. Guided by the objective of minimizing the mean square error of each decision tree in the forest, perform model fitting, and finally train an accurate alloy property predictor based on random forest. The calculation formula for the random forest predictor is: (3) Where B represents the number of decision trees. This represents the output of the decision tree, where x represents the multimodal vector input.
[0035] 4. The trained random forest regressor is used to predict alloy properties; Input the data from the test set into the predictor obtained in step 3 as required, and the model output is the predicted value of the alloy properties. On the alloy property prediction task test set, the method of this invention shows an average difference of 10.8 MPa between the predicted and actual yield strength, with an average error percentage of 1.07%. Compared with methods that do not use multimodal data, the average error is reduced by 12.6 MPa; the average difference between the predicted and actual ultimate tensile strength is 8.2 MPa, with an average error percentage of 0.71%. Compared with methods that do not use multimodal data, the average error is reduced by 26.1 MPa. After collecting the quaternion information of the material to be predicted in step 1, steps 2 and 3 are executed sequentially to obtain the corresponding alloy property predictor to predict the corresponding properties.
[0036] The embodiments described in this specification are merely examples of implementations of the inventive concept. The scope of protection of this invention should not be considered as limited to the specific forms described in the embodiments. The scope of protection of this invention also extends to equivalent technical means that can be conceived by those skilled in the art based on the inventive concept.
Claims
1. A method for predicting alloy properties by integrating visual, linguistic, and process multimodal data, comprising the following steps: S1, construct a multimodal dataset of alloy materials, including process parameters such as temperature and time of solid solution and aging treatment, visual information of SEM images, textual information describing SEM images, and three modal inputs and corresponding true values of alloy mechanical properties. S2, using contrastive learning to train the visual encoder ResNet50 model and masked language modeling to train the language encoder BERT model to obtain vector encodings of SEM images and their language descriptions; S3, the process parameters such as the temperature and time of solution treatment and aging treatment are concatenated and fused with the vector code obtained in the second step, and a random forest regressor is trained according to the corresponding mechanical properties of the alloy. S4. The trained random forest regressor is used to predict the alloy properties.
2. The method as described in claim 1, characterized in that, Step S1 includes the following steps: Step 1-1: Collect process parameters during material preparation, including solution treatment time, solution treatment temperature, number of aging treatment stages, aging treatment time, and aging treatment temperature. Steps 1-2: After the material preparation is completed, the alloy base material is first processed by electrical discharge machining to obtain tensile specimens. Then, the tensile specimens are polished step by step with sandpaper, followed by mechanical polishing of the specimens to a mirror finish using diamond polishing paste. Finally, the mechanically polished specimens are etched using an electrolytic etching solution. Steps 1-3: Place the etched material under a scanning electron microscope to photograph its typical microstructure and collect SEM image data. During this process, repeat sampling of typical images at different locations is performed for data augmentation to improve the stability of subsequent model training. Natural language is used to describe the SEM images, collecting linguistic data on the descriptions. Steps 1-4: Stretch the samples using a tensile testing machine and record their corresponding mechanical properties, including yield strength and ultimate tensile strength. After constructing the process parameters such as solution treatment and aging temperatures and times, SEM image visual information, SEM image text description information, and mechanical property data into corresponding quadruple sets, divide the training and test sets uniformly at a 9:1 ratio, ensuring that there is no overlap between the training and test set data.
3. The method as described in claim 1, characterized in that, Step S2 includes the following steps: Step 2-1: Flip the original SEM image. The flipped image and the original image are used as positive samples in the contrastive learning process, while the flipped image and other images are used as negative samples. Using the positive and negative sample images as input, we obtain positive and negative sample vectors. We then train the ResNet50 visual encoder model using the InfoNCE loss function. The InfoNCE loss function formula is as follows: （1） Where N is the number of samples, and sim represents the cosine similarity. and These are the positive and negative sample encoding vectors of the selected sample i, respectively. It is a temperature parameter used to control the distribution of similarity scores. Step 2-2: Input the original image into the trained ResNet50 to obtain the corresponding SEM visual image encoding vector. Steps 2-3: Divide the original SEM image description language information into a token sequence, mask the random positions in the token sequence, and train BERT using the MLM method and the MSE loss function. The MSE loss function formula is as follows: （2） Where M is the set of hidden tokens, and These are the predicted value and the actual value. Steps 2-4: Input the original SEM image description language information into the BERT after training to obtain the corresponding SEM description language encoding vector.
4. The method as described in claim 1, characterized in that, Step S3 includes the following steps: Step 3-1: The process parameters collected in Step 1 and the visual and language encoding vectors obtained in Step 2 are spliced and fused to construct a multimodal vector for prediction. This vector integrates three key pieces of information: material preparation conditions, microstructure, and semantic description, laying a complete data foundation for subsequent high-precision prediction. Step 3-2: Use the multimodal feature vectors generated in the above steps as input features, aligning them with the alloy mechanical property data collected in Step 1. Input the training set into the random forest regression algorithm. Guided by the objective of minimizing the mean square error of each decision tree in the forest, perform model fitting, and finally train an accurate alloy property predictor based on random forest. The calculation formula for the random forest predictor is: （3） Where B represents the number of decision trees. This represents the output of the decision tree, where x represents the multimodal vector input.
5. The method as described in claim 1, characterized in that, Step S4 includes the following steps: Input the data from the test set into the predictor obtained in step 3 as required, and the model output will be the predicted value of the alloy properties. After collecting the multimodal input information required in step 1, execute steps 2 and 3 in sequence to obtain the corresponding alloy property predictor to predict the corresponding properties.