Parameter identification method system of shield tunneling simulation model based on federated reinforcement learning
Through federal reinforcement learning technology, each client independently trains and aggregates model parameters, solving the accuracy and data security of the shield simulation model parameter identification, and realizing the efficient and secure parameter identification of the shield simulation model in different environments. train.
Patent Information
- Application Number
- CN202310188885.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-02-28
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2043-02-28
AI Technical Summary
Due to different regional conditions, it is difficult to accurately determine the important parameters affecting the simulation model. Due to data privacy and legal constraints, multiple data sources cannot directly exchange data, resulting in the 'data silo' phenomenon, affecting the simulation. Model accuracy and security.
Using a method based on federated reinforcement learning, each client independently establishes and trains a decision model, encrypts the transmission and aggregation of model gradient information and data through the central coordinator, builds global gradient information, and gradually optimizes simulation model parameters to ensure data security.
It improves the training speed and reliability of the shield simulation model, can realize high-precision parameter identification in different environments, alleviate the heterogeneity of the equipment, and protects the security of sensitive data and prevents data theft.
Smart Images

Figure CN116306268B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of shield construction simulation, and more specifically, relates to a shield tunneling simulation model parameter identification method and system based on federated reinforcement learning. Background Art
[0002] With the development of underground space technology, subway construction has been widely used in various places. The shield machine is a key equipment for tunnel excavation. The types of equipment parameters and operating data used in the excavation process are increasing, and the amount of data is also increasing. In the process of excavation, numerical simulation methods are usually used to simulate the excavation of the shield machine. The process of building a simulation model only determines the basic form of the model. In order to improve the accuracy and reliability of the model, the accuracy of the model parameters must be controlled.
[0003] However, since there is no unified standard for some parameters required by the simulation model, most of them are based on the engineering experience of engineers and construction personnel, and not all parameters have measured data. At the same time, the data monitored in various regions are affected by factors such as test methods, environmental conditions, and operating conditions, and there are deviations, making it difficult to determine the important parameters that affect the simulation model. Since the simulation calculation process is a dynamic solution process of nonlinear equations, the number of parameters involved in the modeling and analysis process is huge, and it is difficult to accurately determine the important parameters that affect the simulation model. Therefore, a method for identifying parameters of shield simulation models is needed to identify and verify the parameters required for simulation, improve the accuracy of simulation, and play an important role in realizing the safe and efficient excavation of shield machines. At the same time, in the actual construction process, if only the data of a single project is used, due to the small number of training data samples, the trained model often cannot achieve a high prediction accuracy, so it can be combined with data from other regions for joint training. However, due to the constraints on data privacy and security such as laws and regulations, policy supervision, commercial secrets, and personal privacy, companies will not open these data to the outside world, and multiple data sources cannot directly exchange data, which will form a "data island" phenomenon.
[0004] Therefore, a method is needed that can not only accurately identify the parameters of the shield simulation model, but also protect some sensitive data from various places participating in the learning and training, and prevent malicious theft of data information. Summary of the invention
[0005] In view of the above defects or improvement needs of the prior art, the present invention provides a shield tunneling simulation model parameter identification method based on federated reinforcement learning, which aims to solve the technical problem that the existing shield simulation model parameter identification method is difficult to accurately determine the important parameters affecting the simulation model due to different regional conditions.
[0006] To achieve the above object, according to one aspect of the present invention, a method for identifying parameters of a shield tunneling simulation model based on federated reinforcement learning is provided, comprising:
[0007] S1. Each client shall, based on the local working conditions, geological conditions, shield parameters and original monitoring data, preliminarily formulate the values of the undetermined parameters in the numerical model to be identified, assign initial values to them, establish a parameterized shield tunneling simulation model, and perform sensitivity analysis on the parameters to be identified in the model;
[0008] S2. The central coordinator delegates the reinforcement learning model to each client, and each client establishes a local decision model and establishes an environmental model based on the original data and simulation results of the shield tunneling process in the region collected by each client to simulate the current state S t ;
[0009] S3. Each client independently determines the current state S t Determine the private strategy and select the corresponding action according to the private strategy to obtain the next state S t+1 , and according to the objective function, S t+1 To evaluate, each client trains its own model in the state-action-reward-state cycle, and uploads the model gradient information, the number of training data and its evaluation results obtained during the reinforcement learning training process to the central coordinator after homomorphic encryption;
[0010] S4. After the central coordinator decrypts the model gradient information, the number of training data and the evaluation results uploaded by each client, it selectively aggregates the client model parameters with greater convergence potential, constructs new global gradient information and obtains new reinforcement learning model data;
[0011] S5. The central coordinator returns the latest reinforcement learning model data to each client. Each client updates the local reinforcement learning model by integrating the latest reinforcement learning model data sent by the central coordinator, performs reinforcement learning training according to the latest reinforcement learning model to obtain the parameter values to be determined, and then performs simulation. The local monitoring data is compared with the simulation results and the error between the two is analyzed. Steps S3-S5 are repeated until the error is within an acceptable range. The identified parameter value is the final value of the model parameter to be identified.
[0012] Preferably, the shield tunneling simulation model is established by numerical simulation software. Specifically, a parameterized basic model is first established based on the original data available on the client, including dimensions, material properties, contact and boundary conditions, and then other parameter values that are missing or are based on experience when establishing the simulation model are determined;
[0013] The method for determining the values of other parameters that are missing or based on experience when establishing the simulation model is as follows: first, determine the value range of each of the N undetermined parameters based on the project overview and engineering experience, randomly select the specific values of the N undetermined parameters as the input required to establish the shield tunneling simulation model, and set the N unidentified parameters A m The initial value of m = 1 ~ N, A m The initial value of is input into the shield tunneling simulation model for calculation and solution to obtain the initial simulation result;
[0014] The original monitoring data includes the thrust and torque of the shield machine during the shield tunneling process, the specific data values corresponding to the surface settlement and soil disturbance, the monitoring point locations and time tags.
[0015] Preferably, the method for performing sensitivity analysis on the model parameters to be identified in step S1 includes: combining engineering experience and the Sobol method to perform sensitivity analysis on the model parameters to be identified, the sampling method is Monte Carlo sampling, the number of sampling samples is set to C, and the initial sample matrices X, Y and the construction matrix XY are obtained. m , m = 1 ~ N, X, Y, XY m As the input value of the shield tunneling simulation model, the simulation result vectors f(X), f(Y) and f(XY m ), and the influence coefficient of each parameter to be identified is obtained according to the simulation result vector and the following formula: Tm The calculation method is:
[0016]
[0017] Among them, Z is a new constructed matrix including matrix X and matrix Y, Var(Z) represents the variance of the output results of the shield tunneling simulation model, the larger the influence coefficient of the parameter to be identified, the greater the impact on the model, C represents the total number of set sampling samples, m represents the mth parameter to be identified, and j represents the jth sampling sample.
[0018] Preferably, the central coordinator is responsible for establishing encrypted communication with the client and realizing aggregation of the shared model; multiple parallel clients are composed of heterogeneous devices and are independently distributed geographically without affecting each other.
[0019] Preferably, the environmental model is established by simulation data results and original monitoring data, and an objective function is set, and the objective function is:
[0020]
[0021] Among them, G is the objective function under the current parameter value to be identified, Z sim is the model simulation value of a certain indicator to be observed, Zrea is the measured value of a certain index in the local monitoring data to be observed; the subscripts 1, 2, 3... represent the total number of monitoring indicators in the local monitoring data;
[0022] Construct a reward function r according to the objective function G described above:
[0023] When the objective function G’ > G, r = r + 1,
[0024] When the objective function G’ < G, r = r,
[0025] where G’ is the objective function of the model under the parameter value to be identified in the previous iteration, and G is the objective function under the parameter value to be identified in the current iteration step;
[0026] The initial value of the set reward function r is 0, and the training purpose is to maximize the reward function; a value is randomly selected within the range of the parameter values to be identified as an action of the reinforcement learning model. When r = r + 1, the decision-making model will, according to the fluctuation range ΔA of the current parameter value to be identified m (STm) change all the parameters to be identified to determine the next parameter value to be identified, or randomly change the value of a certain parameter to be identified while fixing the values of other parameters to be identified; when r = r, the decision-making model will change the current parameter value to be identified back to the parameter value of the previous step, and on this basis, change the parameter value to be identified again until r = r + 1; when r = R, the model stops training, and R is a preset threshold;
[0027] The fluctuation range ΔA of the current parameter value to be identified m (STm) Specifically, according to the influence coefficients of each parameter to be identified, set the change values ΔA of N parameters to be identified m (STm) , m = 1~N, where S Tm is the influence coefficient of the m-th parameter to be identified. The next parameter search range will be determined according to the size of the influence coefficient. The greater the influence coefficient of the parameter to be identified, the finer the set parameter search range later, that is, the smaller the value of ΔA m (STm) ;
[0028] After that, change the parameter to be identified according to the reward function r. Specifically:
[0029] When the reward function r = r + 1, select the values of N simulation model parameters to be identified as A m (STm) +ΔA m (STm) , or randomly change the value of a certain parameter to be identified while fixing the values of other parameters to be identified;
[0030] When r=r, the decision model will change the current parameter value to be identified back to the parameter value of the previous step, and then change the parameter value to be identified again on this basis until r=r+1;
[0031] Then, the N simulation model parameter values to be identified are input into the simulation model to obtain new simulation results and updated to the new state S. t+1 And give a reward r;
[0032] With the new state S t+1 As input, the policy network π is used to calculate the probability distribution of each action, and then new actions are randomly sampled according to the probability distribution.
[0033] Preferably, the homomorphic encryption method is that the central coordinator establishes a key pair, the client holds a public key, and the central coordinator holds a private key. The client uses the public key to encrypt the acquired model gradient information, the number of training data and its evaluation results, and the central coordinator uses the private key to decrypt the model gradient information, the number of training data and its evaluation results.
[0034] Preferably, in step S4, the method for selectively aggregating client model parameters with greater convergence potential is: the central coordinator screens the data of each client according to the model gradient information, the number of training data and its evaluation results, and in the first round of iterations, eliminates a group of model gradient information with the worst evaluation results, and uses the model gradient information of other groups to construct new global gradient information to obtain new model data; at the same time, the model gradient information with higher evaluation results is stored, and in subsequent iterative training, the model gradient information with higher evaluation results stored in the previous iteration is aggregated, thereby improving the convergence and accuracy of the global model parameters.
[0035] According to another aspect of the present invention, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method for identifying parameters of a shield tunneling simulation model based on federated reinforcement learning as described in any of the preceding items is implemented.
[0036] According to another aspect of the present invention, a shield tunneling simulation model parameter identification system based on federated reinforcement learning is provided, comprising a computer-readable storage medium as described above and a processor, the processor being used to call and process a computer program stored in the computer-readable storage medium.
[0037] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects compared with the prior art:
[0038] (1) When the target tasks of shield tunneling are similar, multiple clients share the shield tunneling simulation model training experience obtained from different environments, which improves the training speed of the model and speeds up the learning process. At the same time, since the data of the corresponding customers in each region can be used in the process of training the aggregation model of the central coordinator, there are enough data samples and types to train the aggregation model, which improves the reliability of the model when facing shield tunneling tasks in different environments.
[0039] (2) Devices of clients in different regions may have different computing and communication capabilities, and some devices may not meet the basic requirements of training, but strategies are needed to guide actions. The strategy provided by the present invention enables all clients to equally obtain the shared model of the target task, alleviating the problem of device heterogeneity.
[0040] (3) Local data in different regions and projects will not participate in learning. Only intermediate parameters will participate in model training without transmitting their original data. This is not only conducive to updating the simulation model and reducing time costs, but also protects some local sensitive data, ensures the security of model parameter data during the transmission process, and prevents malicious theft of data information. BRIEF DESCRIPTION OF THE DRAWINGS
[0041] Figure 1 It is a flow chart of a shield tunneling simulation model parameter identification method based on federated reinforcement learning provided by the present invention;
[0042] Figure 2 It is a schematic diagram of an interactive process of an implementation example of the application of the present invention. DETAILED DESCRIPTION
[0043] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.
[0044] A method for identifying parameters of a shield tunneling simulation model based on federated reinforcement learning in this embodiment is described in detail in Figure 1 As shown, Figure 1 A schematic flow chart of a method for identifying parameters of a shield tunneling simulation model based on federated reinforcement learning provided in an embodiment of the present invention, the method comprising five steps:
[0045] S1. Each client shall, based on the local working conditions, geological conditions, shield parameters and original monitoring data, preliminarily formulate the values of the undetermined parameters in the numerical model to be identified, assign initial values to them, establish a parameterized shield tunneling simulation model, and perform sensitivity analysis on the parameters to be identified in the model;
[0046] S2. The central coordinator delegates the reinforcement learning model to each client, and each client establishes a local decision model and establishes an environmental model based on the original data and simulation results of the shield tunneling process in the region collected by each client to simulate the current state S t ;
[0047] S3. Each client independently determines the current state S t Determine the private strategy and select the corresponding action according to the private strategy to obtain the next state S t+1 , and according to the objective function, S t+1 To evaluate, each client trains its own model in the state-action-reward-state cycle, and uploads the model gradient information, the number of training data and its evaluation results obtained during the reinforcement learning training process to the central coordinator after homomorphic encryption;
[0048] S4. After the central coordinator decrypts the model gradient information, the number of training data and the evaluation results uploaded by each client, it selectively aggregates the client model parameters with greater convergence potential, constructs new global gradient information and obtains new reinforcement learning model data;
[0049] S5. The central coordinator returns the latest reinforcement learning model data to each client. Each client updates the local reinforcement learning model by integrating the latest reinforcement learning model data sent by the central coordinator, performs reinforcement learning training according to the latest reinforcement learning model to obtain the parameter values to be determined, and then performs simulation. The local monitoring data is compared with the simulation results and the error between the two is analyzed. Steps S3-S5 are repeated until the error is within an acceptable range. The identified parameter value is the final value of the model parameter to be identified.
[0050] Preferably, the shield tunneling simulation model is established by numerical simulation software. Specifically, a parameterized basic model is first established based on the original data available on the client, including dimensions, material properties, contact and boundary conditions, and then other parameter values that are missing or are based on experience when establishing the simulation model are determined;
[0051] The method for determining the values of other parameters that are missing or based on experience when establishing the simulation model is as follows: first, determine the value range of each of the N undetermined parameters based on the project overview and engineering experience, randomly select the specific values of the N undetermined parameters as the input required to establish the shield tunneling simulation model, and set the N unidentified parameters Am The initial value of m = 1 ~ N, A m The initial value of is input into the shield tunneling simulation model for calculation and solution to obtain the initial simulation result;
[0052] The original monitoring data includes the thrust and torque of the shield machine during the shield tunneling process, the specific data values corresponding to the surface settlement and soil disturbance, the monitoring point locations and time tags.
[0053] Preferably, the method for performing sensitivity analysis on the model parameters to be identified in step S1 includes: combining engineering experience and the Sobol method to perform sensitivity analysis on the model parameters to be identified, the sampling method is Monte Carlo sampling, the number of sampling samples is set to C, and the initial sample matrices X, Y and the construction matrix XY are obtained. m , m = 1 ~ N, X, Y, XY m As the input value of the shield tunneling simulation model, the simulation result vectors f(X), f(Y) and f(XY m ), and the influence coefficient of each parameter to be identified is obtained according to the simulation result vector and the following formula: Tm The calculation method is:
[0054]
[0055] Among them, Z is a new constructed matrix including matrix X and matrix Y, Var(Z) represents the variance of the output results of the shield tunneling simulation model, the larger the influence coefficient of the parameter to be identified, the greater the impact on the model, C represents the total number of set sampling samples, m represents the mth parameter to be identified, and j represents the jth sampling sample.
[0056] Preferably, the central coordinator is responsible for establishing encrypted communication with the client and realizing aggregation of the shared model; multiple parallel clients are composed of heterogeneous devices and are independently distributed geographically without affecting each other.
[0057] Preferably, the environmental model is established by simulation data results and original monitoring data, and an objective function is set, and the objective function is:
[0058]
[0059] Among them, G is the objective function under the current parameter value to be identified, Z sim is the model simulation value of a certain indicator to be observed, Z rea is the measured value of a certain indicator in the local monitoring data to be observed; the subscripts 1, 2, 3... represent the total number of monitoring indicators in the local monitoring data;
[0060] Construct the reward function r according to the objective function G:
[0061] When the objective function G’ > G, r = r + 1,
[0062] When the objective function G’ < G, r = r,
[0063] where G’ is the objective function of the model under the parameter values to be identified in the previous iteration, and G is the objective function under the parameter values to be identified in the current iteration step;
[0064] The initial value of the set reward function r is 0, and the training objective is to maximize the reward function; a value will be randomly selected within the range of the parameter values to be identified as an action of the reinforcement learning model. When r = r + 1, the decision-making model will, according to the fluctuation range ΔA of the current parameter values to be identified m (STm) change all the parameters to be identified to determine the next parameter values to be identified, or randomly change the value of a certain parameter to be identified while fixing the values of other parameters to be identified (for example: change the value of the parameter A1 to be identified, and fix other parameters to be identified A2,..., A N ); when r = r, the decision-making model will change the current parameter values to be identified back to the parameter values of the previous step, and on this basis, change the parameter values to be identified again until r = r + 1; when r = R, the model stops training, and R is a preset threshold;
[0065] The fluctuation range ΔA of the current parameter values to be identified m (STm) Specifically, according to the influence coefficients of each parameter to be identified, set the change values ΔA of N parameters to be identified m (STm) , m = 1 ~ N, where S Tm is the influence coefficient of the mth parameter to be identified. The next parameter search range will be determined according to the magnitude of the influence coefficient. The greater the influence coefficient of the parameter to be identified, the finer the set parameter search range will be later, that is, the smaller the value of ΔA m (STm) ;
[0066] After that, change the parameters to be identified according to the reward function r. Specifically:
[0067] When the reward function r = r + 1, select the values of N simulation model parameters to be identified as A m (STm) + ΔA m (STm) , or randomly change the value of a certain parameter to be identified while fixing the values of other parameters to be identified;
[0068] When r=r, the decision model will change the current parameter value to be identified back to the parameter value of the previous step, and then change the parameter value to be identified again on this basis until r=r+1;
[0069] Then, the N simulation model parameter values to be identified are input into the simulation model to obtain new simulation results and updated to the new state S. t+1 And give a reward r;
[0070] With the new state S t+1 As input, the policy network π is used to calculate the probability distribution of each action (that is, when r = r + 1, according to the fluctuation range ΔA of the current parameter value to be identified m (STm) All parameters to be identified are changed to determine the value of the next parameter to be identified or a certain parameter value to be identified is randomly changed, and the probability of the two actions of fixing the other parameter values to be identified is fixed), and then a new action is obtained by random sampling according to the probability distribution.
[0071] Preferably, the homomorphic encryption method is that the central coordinator establishes a key pair, the client holds a public key, and the central coordinator holds a private key. The client uses the public key to encrypt the acquired model gradient information, the number of training data and its evaluation results, and the central coordinator uses the private key to decrypt the model gradient information, the number of training data and its evaluation results.
[0072] Preferably, in step S4, the method for selectively aggregating client model parameters with greater convergence potential is: the central coordinator screens the data of each client according to the model gradient information, the number of training data and its evaluation results, and in the first round of iterations, eliminates a group of model gradient information with the worst evaluation results, and uses the model gradient information of other groups to construct new global gradient information to obtain new model data; at the same time, stores one or more groups of model gradient information with higher evaluation results, and in subsequent iterative training, aggregates the model gradient information with higher evaluation results stored in previous iterations, thereby improving the convergence and accuracy of global model parameters.
[0073] Preferably, the clients have the same reinforcement learning model, and each client can observe its own environment independently within its environment; the agent in environment i and the agents in other environments have similar models, including reward functions, and environment i is independent of other environments and does not depend on the states and actions of other environments; each agent interacts with its own environment to learn the optimal strategy.
[0074] Next, the method of the present invention is further described by taking the parameter identification of elastic modulus E and Poisson's ratio μ as an example:
[0075] S1, each client preliminarily formulates the parameters to be identified in the numerical model and determines the parameter range according to the local working conditions, geological conditions, shield parameters and original monitoring data. In this example, the parameter values to be identified are the elastic modulus E and Poisson's ratio μ of the soil. Then, a value of the two parameters to be identified, the elastic modulus E and Poisson's ratio μ of the soil, is selected within the parameter range to be determined as the initial value of the simulation model, and a parameterized shield tunneling simulation model is established using numerical modeling software, and a sensitivity analysis of the parameters to be identified in the model is performed;
[0076] Specifically, the shield tunneling simulation model is established by numerical software, which establishes a parametric basic model based on some existing original data of the client, including size, material properties, contact and boundary conditions. The method for determining the values of other parameters that are missing or based on experience when establishing the simulation model is as follows: Figure 2 As shown, each client collects relevant original data from the corresponding area, firstly determines the value range of the elastic modulus E and Poisson's ratio μ of the soil according to the engineering profile, and randomly selects a specific value of the elastic modulus E and Poisson's ratio μ as the initial value, and inputs the initial values of the elastic modulus E and Poisson's ratio μ into the shield tunneling simulation model for calculation and solution to obtain the initial simulation results; the original monitoring data includes the thrust and torque of the shield machine during the shield tunneling process, as well as the specific data values, monitoring point locations and time tags corresponding to the surface settlement and soil disturbance;
[0077] The sensitivity analysis method of the model parameters to be identified includes: combining engineering experience and the Sobol method to perform sensitivity analysis of the model parameters to be identified. The sampling method is Monte Carlo sampling to obtain the initial sample matrices A and B and the construction matrix AB. m (m=1,2) and use it as the input value of the shield tunneling simulation model to obtain the simulation result vectors f(A), f(B) and f(XY m ). And according to the simulation result vector and the following formula, the influence coefficient of each parameter to be identified is calculated. The calculation method of the influence coefficient of each parameter to be identified is:
[0078]
[0079] Wherein, Z is a new constructed matrix including matrix X and matrix Y, Var(Z) represents the variance of the output result of the shield tunneling simulation model, the larger the influence coefficient of the parameter to be identified, the greater the influence on the model, C represents the total number of sampling samples set, m represents the mth parameter to be identified, and j represents the jth sampling sample;
[0080] S2. The central coordinator distributes the reinforcement learning model to each client. Each agent in the reinforcement learning model establishes a local decision-making model and an environment model based on the original data and simulation results of the local shield tunneling process collected by each client, which is used to simulate the current state St.
[0081] S3. Each agent can independently observe the state of its environment, determine a private policy according to the current state, and select corresponding actions according to the policy. Then, it is evaluated by the next state St+1 according to the objective function. All agents train their respective models in the state-action-reward-state loop, and at the same time, the model gradient information, the number of training data, and their evaluation results obtained during the reinforcement learning training process are homomorphically encrypted and uploaded to the central coordinator.
[0082] The environment model is established from the simulation data results and the original monitoring data, and an objective function is set. The objective function is:
[0083]
[0084] where G c is the objective function under the current value of the parameter to be identified, Z sim is the model simulation value of a certain index to be observed, and Z rea is the measured value of a certain index in the local monitoring data to be observed; the subscripts 1, 2, 3... represent the total number of monitoring indexes in the local monitoring data. In this example, the monitoring indexes include ground settlement, horizontal displacement of soil layers, and formation disturbance conditions.
[0085] According to the objective function G, a reward function r is constructed: when the objective function G’ > G, r = r + 1; when the objective function G’ < G, r = r, where G’ is the objective function of the model under the previous value of the parameter to be identified, and G is the objective function of the model under the current value of the parameter to be identified; the initial value of the set reward function r is 0, and the purpose of training the model is to maximize the reward function; a value is randomly selected within the possible value range of the parameter to be identified as an action of the reinforcement learning model. When r = r + 1, the decision-making model will change all the parameters to be identified according to the fluctuation range ΔE (ST1) and Δμ (ST2) of the current value of the parameter to be identified, determine the value of the parameter to be identified next time or randomly change the value of a certain parameter to be identified while fixing the values of other parameters to be identified; when r = r, the decision-making model will change the current value of the parameter to be identified back to the previous parameter value, and then change the value of the parameter to be identified again until r = r + 1; a threshold R can be set for the reward function r. When r = R, the model stops training.
[0086] The fluctuation range ΔE (ST1)and Δμ (ST2) Specifically, according to the influence coefficient of each parameter to be identified, the change value ΔE of the elastic modulus E and Poisson's ratio μ of the soil is set (ST1) and Δμ (ST2) , where S Tm is the influence coefficient of the parameter to be identified. The parameter search range in the next step will be determined according to the size of the influence coefficient. The larger the influence coefficient of the parameter to be identified, the finer the parameter search range will be. (ST1) or Δμ (ST2) The smaller the value of is, let E=E+ΔE (ST1) , μ=μ+Δμ (ST2) , and then the elastic modulus of the changed soil is E+ΔE (ST1) and Poisson's ratio μ+Δμ (ST2) Input into the simulation model to obtain new simulation results and update to the new state S t+1 And give a reward r;
[0087] Then, the parameters to be identified are changed according to the reward function. Specifically, when the reward function r = r + 1, the elastic modulus E of the soil is selected as E = E + ΔE (ST1) , the parameter value of Poisson's ratio μ is μ=μ+Δμ (ST2) Or randomly change the value of a parameter to be identified and fix the values of other parameters to be identified (for example, change the value of the elastic modulus E of the soil and fix the Poisson's ratio μ). When r=r, the decision model will change the current parameter value to be identified back to the parameter value of the previous step, and then change the parameter value to be identified again on this basis until r=r+1. For example, the elastic modulus of the soil in the previous step is E' and the Poisson's ratio is μ'. After the next step, the elastic modulus of the soil is E", and the Poisson's ratio is μ". The reward function is calculated to get r=r. In the next step, the elastic modulus of the soil is E' and the Poisson's ratio is μ'. The parameter values are changed again and substituted into the model for calculation.
[0088] With the new state S t+1 As input, the policy network π is used to calculate the probability distribution of each action, and then new actions are randomly sampled according to the probability distribution;
[0089] Each client has the same reinforcement learning model. Each client can observe its own environment independently within its environment. The agent in environment i has a similar model to the agents in other environments, including the reward function. Environment i is independent of other environments and does not depend on the states and actions of other environments. Each agent interacts with its own environment to learn the optimal strategy.
[0090] The central coordinator is responsible for establishing encrypted communication with the client and realizing the aggregation of the shared model. Multiple parallel clients can be composed of heterogeneous devices (for example, IoT devices, smartphones, and computers, etc.) and are geographically independent and independent of each other. There is no specific requirement for the number of agents, and each client can freely choose to join or leave.
[0091] Encrypted transmission is established between each client and the central coordinator, using homomorphic encryption. The central coordinator establishes a key pair, the client holds the public key, and the central coordinator holds the private key. The client uses the public key to encrypt the obtained model gradient information, the number of training data, and its evaluation results, and the central coordinator uses the private key to decrypt the model gradient information, the number of training data, and its evaluation results.
[0092] S4, the central coordinator decrypts the model gradient information, the number of training data and the evaluation results uploaded by each client based on the established global model, selectively aggregates the client model parameters with greater convergence potential, and constructs new global gradient information and obtains new model data;
[0093] The central coordinator selectively aggregates client model parameters with greater convergence potential. It screens each client data according to the model gradient information, the number of training data and its evaluation results. In the first round of iteration, the group of model gradient information with the worst evaluation results is eliminated, and other groups of gradients are used to construct new global gradient information to obtain new model data; at the same time, one or more groups of model gradient information with higher evaluation results are stored. In subsequent iterative training, the model gradient information stored in the previous iteration is aggregated to improve the convergence and accuracy of the global model parameters;
[0094] S5, the central coordinator returns the latest model data to each client, and each client updates the local model by fusing the global model data, performs reinforcement learning training based on the latest model to obtain the pending parameter values, and then performs simulation, compares the local monitoring data with the simulation results, and analyzes the error between the two, repeating steps 3-5 until the error is within an acceptable range.
[0095] At this point, each client can establish a simulation model that conforms to actual working conditions, guide safe construction and perform real-time predictions through the simulation model.
[0096] In some other examples, there may be multiple simulation model parameters to be identified.
[0097] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.
Claims
1. A method for identifying parameters of a shield tunneling simulation model based on federated reinforcement learning, characterized in that: Including: S1. Each client initially determines the values of the undetermined parameters in the numerical model to be identified according to the local working conditions, geological conditions, shield parameters, and original monitoring data, assigns initial values to them, establishes a parameterized shield tunneling simulation model, and conducts a sensitivity analysis on the parameters to be identified in the model; S2. The central coordinator delegates the reinforcement learning model to each client, and each client establishes a local decision model and establishes an environmental model based on the original data and simulation results of the shield tunneling process in the region collected by each client to simulate the current state S t ; S3. Each client independently determines the current state S t Determine the private strategy and select the corresponding action according to the private strategy to obtain the next state S t+1 , and according to the objective function, S t+1 To evaluate, each client trains its own model in the state-action-reward-state cycle, and uploads the model gradient information, the number of training data and its evaluation results obtained during the reinforcement learning training process to the central coordinator after homomorphic encryption; S4. After decrypting the model gradient information, the number of training data, and their evaluation results uploaded by each client, the central coordinator selectively aggregates the client model parameters with greater convergence potential, constructs new global gradient information, and obtains new reinforcement learning model data; S5. The central coordinator returns the latest reinforcement learning model data to each client. Each client updates its local reinforcement learning model by integrating the latest reinforcement learning model data sent by the central coordinator, conducts reinforcement learning training according to the latest reinforcement learning model to obtain the undetermined parameter values, then conducts a simulation, compares the local monitoring data with the simulation results, and analyzes the error between the two. Repeat steps S3 - S5 until the error is within an acceptable range. The identified parameter values are the final values of the parameters to be identified in the model; The environmental model is established from the simulation data results and the original monitoring data, and a target function is set. The target function is: Among them, G is the objective function under the current parameter value to be identified, Z sim is the model simulation value of a certain indicator to be observed, Z rea is the measured value of a certain indicator in the local monitoring data to be observed; the subscripts 1, 2, 3... represent the total number of monitoring indicators in the local monitoring data; Construct a reward function r according to the target function G: When the target function G’ > G, r = r + 1, When the target function G’ < G, r = r, where G’ is the target function of the model under the parameter values to be identified in the previous iteration, and G is the target function of the model under the parameter values to be identified in the current iteration step; The initial value of the reward function r is set to 0. The training goal is to maximize the reward function. A value is randomly selected from the range of the parameter value to be identified as an action of the reinforcement learning model. When r=r+1, the decision model will be based on the fluctuation range ∆A of the current parameter value to be identified. m (STm) All parameters to be identified are changed to determine the value of the next parameter to be identified, or the value of a parameter to be identified is randomly changed while the values of other parameters to be identified are fixed; when r=r, the decision model will change the current parameter value to be identified back to the parameter value of the previous step, and then change the parameter value to be identified again on this basis until r=r+1; when r=R, the model stops training, and R is the preset threshold; The fluctuation range of the current parameter value to be identified is ∆A m (STm) Specifically, according to the influence coefficient of each parameter to be identified, the change value ∆A of N parameters to be identified is set. m (STm) , m=1~N, where S Tm is the influence coefficient of the mth parameter to be identified. The parameter search range in the next step will be determined according to the size of the influence coefficient. The larger the influence coefficient of the parameter to be identified, the finer the parameter search range will be. m (STm) The smaller the value of ; After that, change the parameters to be identified according to the reward function r. Specifically: When the reward function r=r+1, select N simulation model parameter values to be identified as A m (STm) +∆A m (STm) , or, randomly change the value of a parameter to be identified and fix the values of other parameters to be identified; When r = r, the decision model will change the current parameter value to be identified back to the parameter value of the previous step, and then change the parameter value to be identified again on this basis until r = r + 1; Then, the N simulation model parameter values to be identified are input into the simulation model to obtain new simulation results and updated to the new state S. t+1 And give a reward r; With the new state S t+1 As input, the policy network π is used to calculate the probability distribution of each action, and then new actions are randomly sampled according to the probability distribution.
2. According to the method for identifying parameters of a shield tunneling simulation model based on federated reinforcement learning in claim 1, it is characterized in that: The shield tunneling simulation model is established by numerical simulation software. Specifically, first establish a parameterized basic model according to the existing original data of the client, including dimensions, material properties, contact, and boundary conditions, and then determine the values of other missing parameters or parameters taken according to experience when establishing the simulation model; The method for determining the values of other parameters that are missing or based on experience when establishing the simulation model is as follows: first, determine the value range of each of the N undetermined parameters based on the project overview and engineering experience, randomly select the specific values of the N undetermined parameters as the input required to establish the shield tunneling simulation model, and set the N unidentified parameters A m The initial value of m=1~N, A m The initial value of is input into the shield tunneling simulation model for calculation and solution to obtain the initial simulation result; The original monitoring data includes the specific data values, monitoring point positions, and time tags corresponding to the thrust, torque of the shield machine, ground settlement, and soil disturbance during the shield tunneling process.
3. The method for identifying parameters of a shield tunneling simulation model based on federated reinforcement learning according to claim 2 is characterized in that: The method for performing sensitivity analysis on the model parameters to be identified in step S1 includes: combining engineering experience and the Sobol method to perform sensitivity analysis on the model parameters to be identified, the sampling method is Monte Carlo sampling, the number of sampling samples is set to C, and the initial sample matrix X, Y and the construction matrix XY are obtained. m , m=1~N, X, Y, XY m As the input value of the shield tunneling simulation model, the simulation result vector is obtained f ( X ), f ( Y )and f ( XY m ), and the influence coefficient of each parameter to be identified is obtained according to the simulation result vector and the following formula. S Tm The calculation method is: where Z is a new constructed matrix including matrix X and matrix Y, Var(Z) represents the variance of the output results of the shield tunneling simulation model. The greater the influence coefficient of the parameter to be identified, the greater the influence on the model. C represents the total number of sampling samples set, m represents the m-th parameter to be identified, and j represents the j-th sampling sample.
4. The method for identifying parameters of a shield tunneling simulation model based on federated reinforcement learning according to claim 1 is characterized in that: The central coordinator is responsible for establishing encrypted communication with the client and realizing the aggregation of the shared model; multiple parallel clients are composed of heterogeneous devices and are geographically independently distributed without affecting each other.
5. The method for identifying parameters of a shield tunneling simulation model based on federated reinforcement learning according to claim 1, characterized in that: The homomorphic encryption method is that the central coordinator establishes a key pair, the client holds a public key, and the central coordinator holds a private key. The client uses the public key to encrypt the obtained model gradient information, the number of training data and its evaluation results, and the central coordinator uses the private key to decrypt the model gradient information, the number of training data and its evaluation results.
6. The method for identifying parameters of a shield tunneling simulation model based on federated reinforcement learning according to claim 1, characterized in that: In step S4, the method for selectively aggregating client model parameters with greater convergence potential is as follows: the central coordinator screens the data of each client according to the model gradient information, the number of training data and its evaluation results, and in the first round of iterations, eliminates a group of model gradient information with the worst evaluation results, and uses the model gradient information of other groups to construct new global gradient information to obtain new model data; at the same time, stores the model gradient information with higher evaluation results, and in subsequent iterative training, aggregates the model gradient information with higher evaluation results stored in the previous iteration, thereby improving the convergence and accuracy of the global model parameters.
7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the shield tunneling simulation model parameter identification method based on federated reinforcement learning as described in any one of claims 1 to 6.
8. A shield tunneling simulation model parameter identification system based on federated reinforcement learning, characterized in that: The method comprises the computer-readable storage medium as claimed in claim 7 and a processor, wherein the processor is used to call and process the computer program stored in the computer-readable storage medium.
