A material mechanical property inference method based on multi-modal pre-training representation

Abstract

The application discloses a material mechanical property inference method based on multi-modal pre-training representation. In view of the problems that the existing method needs to rely on expensive scanning electron microscope images in the inference stage, and the prediction accuracy is low in the small sample scene, the application constructs a multi-modal network containing an image encoder, a table encoder, a shared projection branch and a modal specific projection branch, and carries out joint pre-training through cross-modal alignment loss, intra-modal decoupling loss based on contrast logarithmic ratio upper bound mutual information estimation and table mask recovery loss. Then, the pre-trained table encoder and its auxiliary projection head are migrated to the downstream multi-task regression network, and the mechanical properties can be predicted only by inputting process parameters. The method can also maintain stable performance in the small sample scene of iterative sample collection, significantly reduces the material representation cost, and improves the prediction accuracy and generalization ability.

Description

Technical Field

[0001] This invention relates to the interdisciplinary field of materials informatics and artificial intelligence, specifically to a method, apparatus, and computer-readable storage medium for reasoning about the mechanical properties of materials based on multimodal self-supervised pre-training characterization. This method is applicable to material systems where there is a complex coupling relationship between process parameters, microstructure, and mechanical properties. Background Technology

[0002] The core challenge of materials design lies in the fact that the relationship between process, structure, and performance is usually not a linear mapping of a single variable or scale, but rather a complex coupled system composed of experimental conditions, microstructure, and performance. Existing methods for predicting materials properties often employ traditional machine learning or supervised deep learning models to directly establish regression relationships between process parameters and performance labels. These methods typically require a large number of labeled samples, while mechanical performance testing is often costly and time-consuming, resulting in a limited number of labeled samples. Furthermore, training models solely using process parameters makes it difficult to fully learn the influence of microstructure on mechanical properties, easily leading to insufficient generalization ability and large fluctuations in prediction results.

[0003] Another approach attempts to input microstructure images along with process parameters into the model to improve performance prediction accuracy. However, microstructure characterization typically relies on equipment such as scanning electron microscopes, which incurs high sampling and sample preparation costs and may not be available during the actual inference phase. If the model depends on image modalities in both the training and inference phases, it will be difficult to meet the application requirements for rapid process screening and material performance prediction.

[0004] Therefore, there is an urgent need for a method that can fully utilize the pairwise relationship between process parameters and microstructure images under unlabeled or poorly labeled conditions, absorb microstructure knowledge during the pre-training stage, and perform mechanical performance inference based solely on process parameters even in the inference stage when image modalities are missing. Summary of the Invention

[0005] This invention aims to address the problems of insufficient prediction accuracy and stability of existing material mechanical property prediction methods with a small number of labeled samples, difficulty in utilizing unlabeled microstructure images, and reliance on expensive image modalities during the inference stage.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] To address the aforementioned issues, this invention provides a method for inferring material mechanical properties based on multimodal pre-trained characterization. In the pre-training stage, this method utilizes unlabeled tabular data of process parameters and image data of microstructures to establish a shared semantic space between process parameters and microstructures through cross-modal comparative learning; it separates the shared semantics from modal-specific information through intramodal decoupling constraints; and it enhances the table encoder's ability to model the dependencies between process parameters through table masking modeling.

[0008] After pre-training, at least one of the table encoder and table-specific projection (auxiliary projection head) branches is transferred to the downstream mechanical property prediction model, and a multi-task regression head is trained on a small number of samples with mechanical property labels. During the inference phase, only the process parameters of the material to be tested are input, and one or more mechanical property prediction values ​​can be output.

[0009] In one alternative implementation, the three training objectives of cross-modal contrastive learning, mutual information decoupling, and mask modeling constitute a weighted total loss, enabling the model to simultaneously learn cross-modal consistency, modality-specific information, and tabular feature dependencies.

[0010] In one optional implementation, the few-shot training adopts an iterative sample acquisition method. First, the downstream predictor is trained using an initial small number of labeled samples. Then, labeled samples are added round by round according to a preset sample budget, and the model is updated to verify and improve the model's performance inference ability under a low label budget.

[0011] The beneficial effects of this invention are as follows:

[0012] 1. In the pre-training stage, this invention utilizes the pairing relationship between microstructure images and process parameters to enable the table encoder to inherit the structural knowledge contained in the image modality. Even without inputting microstructure images in the inference stage, it can still make relatively accurate predictions of mechanical properties.

[0013] 2. This invention learns shared semantics through cross-modal alignment and preserves the specific information of the table modality and the image modality through intra-modal decoupling, avoiding the dominance of the representation space by high-dimensional image modality or strong signal modality due to simple fusion, thereby improving the representation transfer capability.

[0014] 3. This invention enhances the robustness to local dependencies and perturbations of process parameters by using mask table modeling, enabling the model to have more stable predictive performance under conditions of fluctuating experimental conditions, missing features, or small sample training.

[0015] 4. This invention can use a multi-task regression approach to simultaneously predict multiple mechanical performance indicators, reducing the cost of training a separate model for each performance indicator, and improving overall inference efficiency by utilizing the correlation between different performance indicators.

[0016] 5. This invention is applicable to small-sample performance prediction scenarios, and can complete model transfer and fine-tuning with fewer mechanical property labels, reducing material testing and labeling costs. It is suitable for material process screening, performance prediction and experimental scheme optimization. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of the multimodal self-supervised pre-training process of the present invention;

[0018] Figure 2 This is a schematic diagram of the mechanical performance inference process of downstream multi-task regression of tabular modal data in this invention;

[0019] Figure 3 This is a schematic diagram of the small-sample iterative training and performance prediction process in this invention. Detailed Implementation

[0020] The present invention will be further described below with reference to embodiments. It should be understood that the embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the scope of protection of the present invention. Without departing from the concept of the present invention, those skilled in the art can replace or adjust the specific network structure, parameters, loss weights, sample types and prediction indicators.

[0021] Example 1: Data Preprocessing

[0022] The process parameter table data undergoes missing value processing, outlier processing, and normalization or standardization. Categorical or binary variables are encoded using one-hot encoding or binary encoding. Microstructure images are cropped, scaled, and normalized. During training, image enhancement methods such as random cropping, random flipping, rotation, and brightness or contrast perturbation can be further employed to improve model robustness.

[0023] In the unlabeled pre-training stage, mechanical property labels are not required; only the pairwise relationship between process parameters and microstructure images is needed. In downstream small-sample scenarios, a small number of samples with mechanical property test results are selected as labeled samples.

[0024] Example 2: Construction of Multimodal Self-Supervised Networks

[0025] Building an image encoder Table encoder Image sharing projection branch Table shared projection branch Image-specific projection branch and the tabular special projection branch The image encoder uses ResNet or Visual Transformer; the table encoder uses a multilayer perceptron or other neural network capable of processing table features.

[0026] For the k-th sample, the microstructure image The input image encoder obtains the image representation process parameter table data Input the table encoder to obtain the table representation Then, the image shared representation is obtained through the shared projection branch. Shared representation with tables Image-specific representations were obtained through modal-specific projection branches, respectively. and table-specific characteristics .

[0027] The shared projection branch is used to learn the common semantics that can be aligned between process parameters and microstructure, while the modality-specific projection branch is used to preserve the unique information of each mode.

[0028] Example 3: Cross-modal contrastive learning

[0029] Within a training batch, images of the same sample share representations. Shared representation with tables As positive sample pairs, image-shared representations and table-shared representations between different samples are used as negative sample pairs. Cosine similarity is used to measure the similarity between representations, and a temperature coefficient is set. Adjust the smoothness of the contrastive learning distribution.

[0030] The contrast loss from image to table direction can be expressed as:

[0031] The contrast loss from the table to the image direction is calculated in the same way. Cross-modal contrast loss. This is the average or weighted sum of the losses in both directions.

[0032] Through the above training, the process condition representation and microstructure representation of the same material sample are close to each other in the shared semantic space, while the representations of different samples are far apart, thus enabling the model to learn the nonlinear correspondence between process conditions and micromorphology.

[0033] Example 4: Intramodal Information Decoupling

[0034] To avoid the mixing of shared semantics and modality-specific information in the same representation branch, this embodiment applies decoupling constraints to shared representations and specific representations within the same modality. Specifically, it constrains image shared representations separately. Image-specific representation Table sharing representation Table-specific characteristics The mutual information between the two types of representations makes them as complementary as possible in terms of information content.

[0035] Since mutual information is difficult to calculate accurately and directly, this embodiment uses a mutual information upper bound estimation method for optimization. For example, a comparison logarithm ratio upper bound estimator can be used to estimate it. and The upper bound is determined, and the estimated value is used as the decoupling loss. Part of it. During training, the decoupling loss is minimized, causing the shared projection branch to focus more on cross-modal consistency factors, while the modality-specific projection branch focuses more on supplementary information specific to that modality.

[0036] For the task of predicting the mechanical properties of materials, the process parameter relationships preserved by the table-specific branch can supplement the general representation of the table encoder in the downstream inference, so that the model can still obtain stable performance improvement when only process parameters are input.

[0037] Example 5: Mask Table Modeling

[0038] Data from the process parameter table Generate a binary mask ,in Each dimension corresponds to a process feature. A subset of features are selected as masked features according to a preset probability, and these are replaced with replacement values ​​sampled from the edge distribution of that feature, or replaced with preset mask values, to obtain perturbation table data. .

[0039] Disrupt the table data The input is a table encoder, and the output is either the mask position prediction result or the masked feature reconstruction result, processed by a mask prediction head. Mask modeling loss. Binary cross-entropy, mean squared error, or a combination of both can be used to help the model identify which process features are replaced and learn the statistical dependencies between different process parameters.

[0040] Masked table modeling enables the table encoder to learn the local structural relationships between multiple process parameters instead of relying on a single strongly correlated feature. Therefore, it has better robustness in experimental noise, missing local features, or small sample fine-tuning.

[0041] Example 6: Pre-training optimization objective

[0042] Cross-modal contrast loss Decoupling loss and mask modeling loss The combination is the total pre-training loss. .in , and The weighting coefficients can be determined through validation set performance, cross-validation, or empirical settings.

[0043] In one specific implementation, the image encoder uses ResNet50, the table encoder uses a multilayer perceptron, and the number of training epochs is set to 200. The first few epochs use a learning rate warm-up and an annealing learning rate scheduling strategy. The above network structure and training parameters are not limiting conditions; other encoders or training strategies can be selected according to the data scale and material system.

[0044] Example 7: Prediction of downstream mechanical properties

[0045] After pre-training, extract the table encoder. and the tabular special projection branch At least one of these features is used as the feature extraction component of the downstream mechanical performance prediction model. The downstream model can consist of a shared feature layer and multiple regression output heads, each corresponding to a mechanical performance index.

[0046] During training, the process parameters of samples labeled with mechanical properties are input into the table encoder to obtain a pre-trained table representation; optionally, this representation is further input into the table-specific projection branch to obtain an enhanced table-specific representation; then it is input into the multi-task regression head to output predicted values ​​such as mechanical properties.

[0047] The training loss can be a weighted sum of the mean square errors of each performance index. For mechanical performance indices with different dimensions, the labels can be standardized first, and then destandardized after output to obtain the prediction results at the actual physical quantity scale.

[0048] Example 8: Small Sample Performance Prediction

[0049] In scenarios with small sample sizes, the number of labeled samples is limited. One approach is to freeze the pre-trained table encoder and train only the downstream regression head to reduce the risk of overfitting; alternatively, some layers of the table encoder can be unfrozen later in the training process for joint fine-tuning with the regression head.

[0050] In one embodiment, a small number of initially labeled samples are randomly or representatively selected to train a multi-task regression model. Then, candidate samples are gradually added according to a preset number of rounds, with one or more samples added in each round, continuously updating the downstream model until the labeling budget is reached. This process is used to obtain the highest possible performance prediction capability with limited experimental costs.

[0051] Since the pre-training stage has already learned the process-structure correspondence using a large number of unlabeled process parameters and microstructure images, the downstream small-sample training does not need to learn the material characterization from scratch. Therefore, it can achieve higher early prediction performance, a smoother learning curve, and better final convergence performance with a lower label budget.

[0052] Example 9: Inference Flow in the Missing Image Modality

[0053] In practical applications, the material under test may not have been characterized by scanning electron microscopy, or, for cost and efficiency reasons, it may be undesirable to acquire images of its microstructure. This invention does not require input of image modes during the inference phase; only a table of process parameters for the material under test needs to be input.

[0054] Specifically, the process parameters to be tested are standardized and encoded in the same way as in the training phase; the processed process parameters are input into a pre-trained table encoder to obtain a table representation; the table representation is input into a table-specific projection branch and a multi-task regression head to output one or more mechanical property prediction values.

[0055] In this way, microstructure images participate in representation learning as a knowledge source during the pre-training stage, but can be omitted during the inference stage, thus balancing model accuracy and practical application convenience.

[0056] Example 10: Experimental Verification

[0057] To verify the material mechanical property prediction effect of the present invention under the condition of missing image mode, paired process parameter table data and microstructure image data were used for pre-training, and only process parameter table data were input in the downstream prediction stage, without inputting microstructure image data.

[0058] In one experimental verification, the method of the present invention was compared with traditional machine learning methods, single-modal table self-supervised learning methods, and existing multimodal self-supervised learning methods. The evaluation metrics included average R² (coefficient of determination), multiple mechanical performance indicators R², transfer gain, and the trend of R² change during small-sample iterative training.

[0059] Experimental results show that the method of the present invention has a high average R² under only the table modal reasoning condition, and performs well in multiple indicators such as fracture strength, yield strength, elastic modulus, tangent modulus and elongation at break, indicating that the structural knowledge provided by the microstructure image can be transferred to the table encoder through pre-training.

[0060] In the small-sample iterative training experiment, a small number of labeled samples were first used to train the downstream multi-task regression model, and then labeled samples were added and the model was updated round by round. Experimental results show that the method of the present invention has good early prediction ability in the low-label budget stage, and maintains an overall upward trend as the number of labeled samples increases.

[0061] The experimental verification results of this embodiment are summarized in the table below.

[0062] Table 1 Summary of Experimental Verification Results

[0063] Missing image modality prediction Pre-training uses tables and images; inference only requires input of process parameters. The average R² is superior to the multi-class comparison method. We prove that knowledge of image structure can be transferred to table branches. Multi-index mechanical property prediction Five mechanical performance indicators were evaluated separately. The performance of multiple indicators is relatively balanced. We demonstrate that multi-task regression has stable predictive power. Pre-trained table encoder transfer Compare training from scratch with transferring pre-trained encoders. Multiple metrics gained after migration. The table encoder representation is proven to be transferable. Table-specific projection branch contribution Compare migrating only the encoder with migrating specific branches simultaneously. Some indicators have improved or become more stable. It is demonstrated that modal-specific information can supplement general characterization. Small sample iterative training Start with a small number of labeled samples and gradually increase the number of samples in each round. It performed well in the early stages with a low annotation budget, and its overall performance improved as the sample size increased. The results demonstrate that the method is suitable for small-sample prediction scenarios where labeling costs are high.

[0064] As shown in Table 1, the method of the present invention can maintain high prediction accuracy in the inference stage of images lacking microstructure, and supports prediction of multiple mechanical performance indicators. The pre-trained table encoder and the table-specific projection branch can both provide effective gains for downstream prediction and improve label efficiency and prediction stability under small sample labeling conditions.

[0065] Example 11: Applicable Material Systems and Extension Methods

[0066] This invention is not limited to a specific material system. For any material system with process parameters, microstructure images, and mechanical property labels, as long as a pairwise relationship between process parameters and microstructure images can be established, this invention can be used for pre-training and mechanical property inference.

[0067] When the material system has other available modes, such as spectra, diffraction patterns, composition descriptions, or textual experimental records, they can also be incorporated as additional modes into the cross-modal alignment and modality-specific decoupling framework without departing from the inventive concept.

[0068] The mechanical performance indicators described in this invention are not limited to material performance indicators such as fracture strength, yield strength, elastic modulus, tangential modulus, elongation at break, hardness, toughness, fatigue life, compressive strength, and flexural strength.

[0069] In summary, this invention transfers structural knowledge from microstructure images to directly obtainable process parameter representations through cross-modal alignment, intra-modal decoupling, and mask table modeling. It then combines this with small-sample downstream training to achieve material mechanical property inference. This method is particularly suitable for material development scenarios where acquiring microstructure images is costly, mechanical property labels are scarce, and rapid process screening is required.

Claims

1. A method for inferring the mechanical properties of materials based on multimodal pre-trained characterization, characterized in that, The process includes the following steps: S1, acquiring paired multimodal data of material samples, wherein the paired multimodal data includes tabular data of material processing parameters and microstructure image data corresponding to the material processing parameters; S2, inputting the microstructure image data into an image encoder to obtain image representation, and inputting the tabular data of material processing parameters into a table encoder to obtain table representation; S3, inputting the image representation and table representation into a shared projection branch respectively to obtain image shared representation and table shared representation, and performing cross-modal contrastive learning based on the paired sample relationship, so that the image shared representation and table shared representation of the same sample are close to each other, and the shared representations of different samples are far apart; S4, inputting the image representation and table representation into the corresponding modality-specific projection branch respectively to obtain image-specific representation and table-specific representation, and performing cross-modal contrastive learning based on the paired sample relationship, so that the image shared representation and table shared representation of the same sample are close to each other, and the shared representations of different samples are far apart; S5. Decoupling constraints are applied to shared and specific representations within the same modality; S6. Mask modeling training is performed on the material process parameter table data to learn the local dependencies between process parameters and improve the table encoder's anti-disturbance capability; S7. The image encoder, table encoder, shared projection branch, and modal specific projection (auxiliary projection head) branch are pre-trained based on cross-modal contrastive learning loss, decoupling loss, and mask modeling loss; S8. At least one of the pre-trained table encoder and table specific projection branch is transferred to the mechanical property prediction model, which is a multi-task regression model and fine-tuned using material samples with mechanical property labels; S9. In the inference stage, the process parameter table data of the material to be tested is input, and one or more mechanical property prediction values ​​of the material to be tested are output.

2. The method according to claim 1, characterized in that, The cross-modal contrastive learning adopts a positive and negative sample construction method, which uses the image-shared representation and table-shared representation of the same material sample as positive sample pairs, and the image-shared representation and table-shared representation of different material samples as negative sample pairs, and calculates the contrast loss based on cosine similarity and temperature coefficient.

3. The method according to claim 1, characterized in that, The decoupling constraints include mutual information minimization constraints, which are achieved by estimating and minimizing the upper bound of mutual information between shared representations and modality-specific representations. This makes the shared projection branch focus on preserving cross-modal consistent semantics, and the modality-specific projection branch focus on preserving the proprietary information of the corresponding modality.

4. The method according to claim 3, characterized in that, The mutual information upper bound is calculated using a contrastive logarithmic ratio upper bound estimator, and decoupling losses are applied to image shared representation and image specific representation, and table shared representation and table specific representation, respectively.

5. The method according to claim 1, characterized in that, The mask modeling training includes: generating a binary mask for some features in the process parameter table data according to a preset probability; replacing the masked features with replacement features obtained by edge distribution sampling or preset mask values ​​to form perturbation table data; inputting the perturbation table data into a table encoder, and predicting the masked position or reconstructing the masked features through a mask prediction head.

6. The method according to claim 1, characterized in that, The total pre-training loss in step S6 is a weighted sum of cross-modal contrastive learning loss, decoupling loss, and mask modeling loss, and the weight of each loss term is determined based on the performance of the validation set or a preset empirical value.

7. The method according to claim 1, characterized in that, In the scenario of a small sample with scarce labeled samples, the small sample labeled set in step S7 is constructed by an iterative sample acquisition method. The iterative sample acquisition method includes first selecting labeled samples to train the mechanical performance prediction model, and then adding candidate samples and updating the model in a preset number of rounds until the labeled budget is reached.

8. The method according to claim 1, characterized in that, The pre-training phase uses microstructure image data to provide structural knowledge, while the inference phase, in the absence of microstructure image data, completes mechanical property inference solely based on material process parameter table data.

9. A material mechanical property reasoning device, characterized in that, include: The data acquisition module is used to acquire tabular data of process parameters and corresponding microstructure image data of material samples; The encoding module is used to extract image representations and table representations through an image encoder and a table encoder, respectively. The cross-modal alignment module is used for cross-modal comparative learning based on pairwise sample relationships; The decoupling module is used to apply decoupling constraints to shared representations and modality-specific representations; The mask modeling module is used to train mask modeling on process parameter table data. The transfer prediction module is used to transfer at least one of the pre-trained table encoder and table-specific projection branches to the mechanical property prediction model and output the predicted values ​​of material mechanical properties.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 8.

Images