Transfer learning method for solving multi-fidelity data fusion rigidity problem

Abstract

The invention discloses a transfer learning method for solving a multi-fidelity data fusion rigidity problem, which aims to solve the rigidity problem existing in model training, and comprises the following steps: taking low-fidelity data in a multi-fidelity modeling method as a trainable parameter, and carrying out constraint by using a designed transfer path to establish an intermediate domain; the rigid phenomenon caused by obvious difference of high-fidelity data volume and low-fidelity data volume in the process of processing multi-source data fusion by a block type network framework is solved. An original migration task is divided into a multi-segment migration process, and a migration path is constrained by establishing a plurality of intermediate domains. According to the path, the model migration loss is at a minimum value, the model parameters are always in a stable state in the updating process, and data participating in migration module parameter updating are increased through establishment of the intermediate domain, so that the model can capture more data features.

Description

Technical Field

[0001] The present invention belongs to the field of multi-fidelity modeling. More specifically, the present invention relates to a transfer learning method for solving the rigidity problem of multi-fidelity data fusion. Background Art

[0002] In the computational modeling of physical systems, due to the limitation of computational costs, there is little high-precision data available, which makes it difficult to meet the engineers' design requirements for the system. To solve this problem, most existing methods use artificial intelligence prediction methods that fuse multi-source data features, using a large amount of cheap low-precision data to provide prior information and a small amount of high-precision data to improve prediction accuracy. Represented by the method (MFDNN) proposed by Meng et al. (Meng X, Karniadakis G E. A composite neural network that learns from multi-fidelity data: Application to function approximation and inverse PDE problems[J]. Journal of Computational Physics, 2020, 401: 109020.), a block-based network (such as Figure 1 ) provides a module that preserves the characteristics of low-fidelity data And high and low fidelity mapping module Predict high-precision data and achieve excellent prediction results. This block-based network structure is based on As a low-fidelity proxy model The module outputs the predicted low-fidelity data ,Will and low-fidelity data for prediction As a migration module input, thereby predicting high-fidelity data This block-based model helps to fully preserve low-precision data, and the high-precision data features will not affect the low-precision data features.

[0003] The parameter update of the block structure is affected by the two-part loss gradient. At the same time, due to the high cost of high-precision data, the amount of high-precision data involved in training is small, which means , the gradient of each sample in the low-precision data is averaged by more data, making The change range of is smaller, and its parameters are updated slowly. The fast parameter update in the training process leads to an unbalanced parameter update speed in the overall model training process, resulting in multi-scale changes in the same system and rigidity problems. This causes the model to output significantly different prediction results under slightly changed initial conditions, seriously affecting the stability and accuracy of the model. Summary of the invention

[0004] An object of the present invention is to solve at least the above problems and / or disadvantages and to provide at least the advantages which will be described hereinafter.

[0005] In order to achieve these purposes and other advantages of the present invention, a transfer learning method for solving the rigidity problem of multi-fidelity data fusion is provided. In the computational modeling of physical systems, the multi-fidelity modeling method process established by using a block-based structure includes the following steps: S1. Low-fidelity data model in block structure Input low-fidelity data independent variables , get the predicted low-fidelity data , S2. Low-fidelity data for prediction and high-fidelity data independent variables Input to the high-low fidelity mapping module in the block structure , get the predicted high-fidelity data ; In S2's high and low fidelity mapping module Low-fidelity data for predictions in To high-fidelity data Multiple intermediate thresholds are established on the transfer learning path to update the overall neural network parameters to: In the above formula, Represents the neural network parameters of the overall training of the module t+1 times, Represents a low-fidelity data model with the same number of training times parameter, Represents high- and low-fidelity mapping modules under the same number of training times The model parameters in represents the intermediate domain sample trained t+1 times, that is, the t+1th intermediate domain, represents the intermediate domain samples trained t times, , Represent the learning rates of the low-fidelity data model and the high- and low-fidelity mapping modules, respectively. , Represent the sample size of high-fidelity and low-fidelity data, respectively. represents the loss value of the i-th sample in the low-fidelity prediction value, represents the gradient of the parameter update of the high- and low-fidelity mapping module, represents the gradient of the intermediate domain parameter update, represents the i-th sample in the r-th intermediate domain, M represents the number of intermediate domains, represents the neural network parameters of the low-fidelity data model after training t times, represents the i-th sample in the high-fidelity domain, represents the i-th sample in the low-fidelity domain.

[0006] Preferably, the low-fidelity data model Low-fidelity data for prediction As the first intermediate threshold, the high-low fidelity mapping module Low-fidelity data through prediction The gradient in the neural network model is calculated by comparing the loss value of the intermediate domain data with the high-fidelity data. Loss value between To calculate the update gradient required for the intermediate domain, the obtained update gradient continuously updates the predicted low-fidelity data , the low-fidelity data of the prediction after each update As the intermediate domain for the next migration, until the data of the intermediate threshold is updated to ; Among them, the intermediate domain loss value Obtained by the following formula: In the above formula, represents the connection coefficient, Represents the number of samples in the middle threshold.

[0007] The present invention has at least the following beneficial effects: It solves the rigidity phenomenon caused by the obvious difference in the amount of high-fidelity and low-fidelity data in the process of multi-source data fusion processed by the block network framework. The original migration task is divided into a multi-stage migration process, and the migration path of the model is constrained by establishing multiple intermediate domains. This path makes the model migration loss at a minimum value, and the model parameter update process is always in a stable state. In addition, by establishing the intermediate domain, the data involved in the migration module parameter update is increased, so that the model can capture more data features.

[0008] Other advantages, objectives and features of the present invention will be embodied in part through the following description, and in part will be understood by those skilled in the art through study and practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 A schematic diagram of a block-based structural framework used in the multi-fidelity modeling method of the present invention; Figure 2 It is a model training flow chart of the multi-fidelity modeling method in the present invention; Figure 3 This is a diagram of the traditional transfer learning training process; Figure 4 This is a diagram of the segmented rigid transfer learning training process in the present invention; Figure 5 This is a sampling diagram of a nonlinear function example in Embodiment 1 of the present invention; Figure 6 It is an absolute error diagram of the prediction results of the traditional block-based MFDNN in Example 1; Figure 7 is an absolute error diagram of the prediction results of the segmented rigid migration framework of the present invention in Example 1; Figure 8 This is a sampling diagram of a nonlinear function example in Embodiment 2 of the present invention; Fig. 9 It is an absolute error diagram of the prediction results of the traditional block-based MFDNN in Example 2; Fig.10 FIG. 4 is an absolute error diagram of the prediction results of the segmented rigid migration framework of the present invention in Example 1. FIG. DETAILED DESCRIPTION

[0010] The present invention is further described in detail below in conjunction with the accompanying drawings so that those skilled in the art can implement the invention with reference to the description.

[0011] like Figure 2-Figure 4 The multi-fidelity modeling method based on this block-based structure is shown in the figure. The parameter update during the training process can be expressed as: in, Represents the neural network parameters of the overall training of the module t+1 times, Represents a low-fidelity data model with the same number of training times parameter, Represents high- and low-fidelity mapping modules under the same number of training times In the parameter update process, the parameters of iteration t+1 are calculated by the parameters of iteration t, learning rate, and loss value through the gradient descent method, which can be specifically expressed as: in, denote the learning rates of the low-fidelity data model and the migration module, and denote the loss values ​​of the low-fidelity data model and the high- and low-fidelity mapping module, respectively. Represents the model update gradient. This gradient is averaged from the training samples: in, Represent the high and low fidelity sample sizes, respectively. Represents high- and low-fidelity data. and Represents the high-fidelity prediction value and the low-fidelity prediction value respectively. The loss value of the sample, thus, the overall neural network parameters can be obtained as: From the above formula, we can see that the parameter update of the block structure is affected by the two parts of the loss gradient. At the same time, due to the high cost of high-precision data, the amount of high-precision data involved in training is small, which means , the gradient of each sample in the low-precision data is averaged by more data, making The change range of is smaller, and its parameters are updated slowly. The fast parameter update in the training process leads to an unbalanced parameter update speed in the overall model training process, resulting in multi-scale changes in the same system and rigidity problems. This causes the model to output significantly different prediction results under slightly changed initial conditions, seriously affecting the stability and accuracy of the model.

[0012] By adopting a rigid transfer learning method framework to establish an intermediate domain, the multi-fidelity modeling task is transformed into a piecewise transfer task, improving the module The amount of data involved in training fundamentally balances the scale of parameter changes and suppresses rigidity problems. The segmented migration in can be expressed as: By establishing the intermediate domain, the overall neural network parameter update can be changed to: In the above formula, Represents the neural network parameters of the overall training of the module t+1 times, Represents a low-fidelity data model with the same number of training times parameter, Represents high- and low-fidelity mapping modules under the same number of training times The model parameters in represents the intermediate domain sample trained t+1 times, that is, the t+1th intermediate domain, represents the intermediate domain samples trained t times, , Represent the learning rates of the low-fidelity data model and the high- and low-fidelity mapping modules, respectively. , Represent the high and low fidelity sample sizes, respectively. , Represents high- and low-fidelity data, represents the loss value of the i-th sample in the low-fidelity prediction value, represents the gradient of the parameter update of the high- and low-fidelity mapping module, represents the gradient of the intermediate domain parameter update, represents the i-th sample in the r-th intermediate domain, M represents the number of intermediate domains, represents the neural network parameters of the low-fidelity data model after training t times, represents the i-th sample in the high-fidelity domain, represents the i-th sample in the low-fidelity domain.

[0013] Among them, the intermediate domain loss value Obtained by the following formula: In the above formula, represents the connection coefficient, Represents the number of samples in the middle threshold.

[0014] Among them, the low-fidelity data model Low-fidelity data for prediction As the first intermediate domain, the high-low fidelity mapping module Low-fidelity data obtained through prediction The loss between the intermediate domain and the high-fidelity domain is used to calculate the gradient in the neural network model, so that the model continues to approach the intermediate domain. It is used to calculate the update gradient required for the intermediate domain. The data generated after each round of iterative update is an intermediate domain. The first intermediate domain is , the last intermediate domain is As the intermediate domain keeps approaching the high-fidelity domain, and the model keeps approaching the intermediate domain, the model will eventually approach the high-fidelity domain.

[0015] The present invention solves the rigidity phenomenon caused by the obvious difference in the amount of high-fidelity and low-fidelity data in the process of multi-source data fusion processing by the block network framework. The original migration task is divided into a multi-stage migration process, and the model migration path is constrained by establishing multiple intermediate domains. This path makes the model migration loss at a minimum value, and the model parameter update process is always in a stable state. In addition, by establishing the intermediate domain, the data involved in the migration module parameter update is increased, so that the model can capture more data features.

[0016] Example 1 This is illustrated by using nonlinear numerical functions. High-fidelity data and low-fidelity data are obtained using the following functions: In the above formula, x represents the dependent variable.

[0017] The multi-fidelity modeling experiment was conducted by setting different numbers of high-fidelity data training points. First, 20 high-fidelity data training points were used for multi-fidelity experiments. The distribution of high-fidelity and low-fidelity data is as follows: Figure 5 As shown, Figure 6 It represents the experimental results of the traditional block-based MFDNN. Figure 7 It can be seen from the experiment that when more high-fidelity training points are involved in the training, the results of the two methods can effectively show the periodicity of nonlinear functions.

[0018] Example 2 We used 11 high-fidelity training points for multi-fidelity experiments, and the data distribution is as follows: Figure 8 As shown, Fig. 9 and Fig.10 The experimental results of MFDNN and the rigid migration network of the present invention are shown in Figure 2. Fig. 9 It is known that in The traditional block-based network architecture is difficult to capture the periodicity of functions, and the prediction results are poor. The main reason is that reducing the amount of high-fidelity data makes the model training and updating process rigid, which shows that the input with small differences predicts the output results with obvious differences. Fig.10 It can be seen that the prediction results of the rigid migration network show the periodic law of the function. The main reason is that the model adopts the minimum energy constraint migration path during the training process, which makes the gradient scale in the migration module stable and fundamentally suppresses the model rigidity phenomenon. The rigidity phenomenon exhibited by the traditional block network architecture MFDNN when there is a significant difference between high-fidelity and low-fidelity data, as well as the potential of the rigid migration network in dealing with rigid problems, among which, Figure 5-Figure 10 In the equation, the X-axis represents the dependent variable and the Y-axis represents the independent variable.

[0019] In Example 1 and Example 2, it can be seen that when the amount of low-fidelity data is the same, by changing the high-fidelity data participating in the multi-fidelity modeling experiment When the amount of data is large, and when more high-fidelity data training points participate in the training, the results of the traditional block-based architecture MFDNN and the rigid migration network of the present invention can effectively show the periodicity of nonlinear functions. However, when there are fewer high-fidelity data training points, the traditional block-based network architecture is difficult to capture the periodicity of the function, and the prediction results are poor. The rigid migration network of the present invention can effectively show the periodicity of nonlinear functions.

[0020] Although the embodiments of the present invention have been disclosed as above, they are not limited to the applications listed in the specification and the implementation modes, and they can be fully applied to various fields suitable for the present invention. For those familiar with the art, additional modifications can be easily implemented. Therefore, without departing from the general concept defined by the claims and the scope of equivalents, the present invention is not limited to the specific details and the illustrations shown and described herein.

Claims

1. A transfer learning method for solving the rigidity problem of multi-fidelity data fusion. In the computational modeling of physical systems, the multi-fidelity modeling method established by using a block-based structure includes the following steps; S1. Low-fidelity data model in block structure Input low-fidelity data independent variables , get the predicted low-fidelity data , S2. Low-fidelity data for prediction and high-fidelity data independent variables Input to the high-low fidelity mapping module in the block structure , get the predicted high-fidelity data ; It is characterized in that The high and low fidelity mapping module in S2 Low-fidelity data for predictions in To high-fidelity data Multiple intermediate thresholds are established on the transfer learning path to update the overall neural network parameters to: In the above formula, Represents the neural network parameters of the overall training of the module t+1 times, Represents a low-fidelity data model with the same number of training times parameter, Represents high- and low-fidelity mapping modules with the same number of training times The model parameters in represents the intermediate domain sample trained t+1 times, that is, the t+1th intermediate domain, represents the intermediate domain samples trained t times, , Represent the learning rates of the low-fidelity data model and the high- and low-fidelity mapping modules, respectively. , Represent the sample size of high-fidelity and low-fidelity data, respectively. represents the loss value of the i-th sample in the low-fidelity prediction value, represents the gradient of the parameter update of the high- and low-fidelity mapping module, represents the gradient of the intermediate domain parameter update, represents the i-th sample in the r-th intermediate domain, M represents the number of intermediate domains, represents the neural network parameters of the low-fidelity data model after training t times, represents the i-th sample in the high-fidelity domain, represents the i-th sample in the low-fidelity domain.

2. The transfer learning method for solving the rigidity problem of multi-fidelity data fusion according to claim 1, It is characterized in that The low-fidelity data model Low-fidelity data for prediction As the first intermediate threshold, the high-low fidelity mapping module Low-fidelity data through prediction The gradient in the neural network model is calculated by comparing the loss value of the intermediate domain data with the high-fidelity data. Loss value between To calculate the update gradient required for the intermediate domain, the obtained update gradient continuously updates the predicted low-fidelity data , the low-fidelity data of the prediction after each update As the intermediate domain for the next migration, until the data of the intermediate threshold is updated to ; Among them, the intermediate domain loss value Obtained by the following formula: In the above formula, represents the connection coefficient, Represents the number of samples in the middle threshold.

Images