A friction coefficient calibration method for large cylinder forging simulation based on reinforcement learning

Abstract

A reinforcement learning-based friction coefficient calibration method for large-scale cylindrical forging simulation uses the temperature field and reduction as inputs for the cylindrical forging simulation, and load data as the evaluation benchmark for the simulation output load deviation. A reinforcement learning model is constructed. Based on the reinforcement learning algorithm and finite element simulation technology, the deviation between the simulated upsetting load and the actual upsetting load is used as the reward function to calculate the cylindrical upsetting finite element model, obtain the simulated time-load curve of the cylindrical upsetting, calculate the deviation between the actual and simulated upsetting loads, and generate a reward. The friction coefficient is then updated based on the reward, thereby improving the accuracy of the simulation prediction of the forging pressure and ensuring that the finite element numerical simulation of large-scale cylindrical forging can effectively guide the formulation of the process and production process.

Description

Technical Field

[0001] The present invention relates to the technical field of cylinder forging, and in particular to a friction coefficient calibration method for large cylinder forging simulation based on reinforcement learning. Background Art

[0002] As a crucial component of heavy equipment such as nuclear power high-pressure vessels and petroleum industry hydrogenation reactors, cylindrical forgings are being manufactured with increasing scale and precision. Currently, the steel ingots required for cylindrical forgings have risen to 200 tons. If the forging pressure is incorrectly calculated, production execution will not meet the forging process requirements, and quality issues such as coarse grains, shrinkage, and cracks caused by forging interruptions are very likely to occur. Therefore, a high level of compatibility between the forging equipment load and the process is required.

[0003] With the development and application of finite element and computer technology, numerical simulation is increasingly widely used in the formulation of forging process parameters. By simulating the cylinder forging through finite element simulation, the load information during forming can be obtained. However, there are still many problems with the consistency between many numerical simulation results and actual production, and they can only provide limited reference for process design. The accuracy of the load simulation results is insufficient, which is closely related to the friction coefficient. Therefore, in order to improve the accuracy of the cylinder load simulation, achieve the maximum pressure required for the forging process, and then guide the purpose of process optimization, it is necessary to determine the friction coefficient that conforms to actual production. The friction coefficient is one of the key input parameters in forging load simulation, and its change will significantly affect the accuracy of the simulated load. However, the current friction coefficient mostly relies on empirical values ​​and fails to accurately match the actual production situation, resulting in a large deviation between the simulation results and the actual production data. Summary of the Invention

[0004] In order to solve the problem of low load prediction accuracy in large cylinder forging simulation, the present invention provides a friction coefficient calibration method for large cylinder forging simulation based on reinforcement learning.

[0005] The technical solution adopted by the present invention to solve the above technical problems is: a large cylinder forging simulation friction coefficient calibration method based on reinforcement learning, comprising the following steps:

[0006] Step 1: Collect the temperature field, reduction, and load data of the barrel forging process, correlate the temperature field, reduction, and load data at the same time, use the temperature field and reduction as inputs for the barrel forging simulation, and use the load data as a criterion for judging the load deviation of the simulation output;

[0007] Step 2: Build a reinforcement learning model. Define the friction coefficient μ of the cylinder material as the state space, with a value range of 0.05 to 0.95; define the fine-tuning Δμ of the friction coefficient value as the action space, with a value range of ±0.01; and define the friction coefficient reward function r as the negative value of the deviation between the actual and simulated loads in the cylinder upsetting process:

[0008] ...Formula (1);

[0009] In formula (1), r is the reward function value, is the simulated load value at the i-th second, is the actual load value at the i-th second;

[0010] Step 3: Implement the Deep Q-Network (DDPG) algorithm using TensorFlow’s TF-Agents reinforcement learning library. The DDPG algorithm uses a policy network to generate actions Δμ and a critic network to evaluate the quality of the actions Δμ.

[0011] Step 4: Establish the geometric model of the ingot body, upsetting cover and turntable for the cylinder upsetting simulation, import the geometric model into the finite element simulation software for assembly, divide the mesh, define the cylinder ingot body material, set the billet temperature, mold temperature, upsetting ratio, mold pressing speed and heat exchange coefficient;

[0012] Step 5: Initialize the state space by setting the friction coefficient μ, run the finite element simulation software, calculate the cylinder upsetting finite element model, obtain the simulation time-load curve of the cylinder upsetting, calculate the deviation between the actual and simulated loads of the upsetting process and generate a reward, and then update the friction coefficient μ based on the reward;

[0013] The policy network is trained and updated by minimizing the loss function, which is:

[0014] ...Formula (2);

[0015] In formula (2), represents the probability that the policy network outputs action a in state s, represents the state-action value function;

[0016] The training update process of the policy network is expressed as:

[0017] ...Formula (3);

[0018] In formula (3), T is a constant less than 1;

[0019] The critic network is trained and updated by minimizing the temporal query score TD function, which is:

[0020] ...Formula (4);

[0021] In formula (4), y is the TD target, and the calculation formula is:

[0022] ...Formula (5);

[0023] In formula (5), s' is the next state, a' is the next output action, and γ is the discount factor;

[0024] The training update process of the critic network is expressed as:

[0025] ...Formula (6);

[0026] When the friction coefficient reward function r is minimized, the training is stopped and the friction coefficient μ is output as the calibrated large cylinder forging simulation friction coefficient.

[0027] Preferably, in step 4, the number of grids for the ingot body is 100,000, and the number of grids for the upsetting cover and the turntable is 60,000 each.

[0028] Preferably, in step five, the interaction trajectory between the agent and the environment is stored using experience replay, including state s, action a, reward r, and next state s'.

[0029] According to the above technical solution, the beneficial effects of the present invention are:

[0030] The present invention uses a reinforcement learning algorithm based on finite element simulation technology, takes the deviation between the simulation load and the production load data as the reward function, and calibrates the friction coefficient of the cylinder forging simulation, thereby improving the simulation prediction accuracy and ensuring that the numerical simulation of the cylinder forging can effectively guide process optimization and actual production. DETAILED DESCRIPTION

[0031] A method for calibrating the friction coefficient of a large cylinder forging simulation based on reinforcement learning is proposed. This embodiment uses a large cylinder with a 206T ingot shape and a 12Cr2Mo1V material. The method specifically includes the following steps:

[0032] Step 1: Use OPC, Modbus protocol, ACCESS database, etc. to collect the temperature field, reduction, and load data of the cylinder forging process with ingot shape 206T and material 12Cr2Mo1V. Correlate the temperature field, reduction, and load data at the same time, use the temperature field and reduction as the input of the cylinder forging simulation, and use the load data as the evaluation benchmark for the simulation output load deviation.

[0033] Step 2: Build a reinforcement learning model. Define the friction coefficient μ of the cylinder material as the state space, with a value range of 0.05 to 0.95; define the fine-tuning Δμ of the friction coefficient value as the action space, with a value range of ±0.01; and define the friction coefficient reward function r as the negative value of the deviation between the actual and simulated loads in the cylinder upsetting process:

[0034] ...Formula (1);

[0035] In formula (1), r is the reward function value, is the simulated load value at the i-th second, is the actual load value at the i-th second.

[0036] Step 3: Implement the Deep Q-Network (DDPG) algorithm using TensorFlow’s TF-Agents reinforcement learning library. The DDPG algorithm uses a policy network to generate actions Δμ and a critic network to evaluate the quality of the actions Δμ.

[0037] Step 4. Create a geometric model of the 206T cylinder upsetting simulation, including the ingot body: φ2700×3700mm, the upsetting cover: φ3500×500mm, and the turntable: φ5500×1000mm. Import the geometric model into the finite element simulation software for assembly, and divide the ingot body into 100,000 grids, and the upsetting cover and turntable into 60,000 grids each.

[0038] Then, define the material of the cylindrical ingot body, set the true stress and strain, density, Young's modulus and other properties of the cylindrical ingot body 12Cr2Mo1V material, set the billet temperature to 1200℃, the mold temperature to 250℃, the upsetting ratio to 2.0, the mold pressing speed to 35mm / s, and the heat exchange coefficient to 2000W / (m 2 •K).

[0039] Step 5. Initialize the state space by setting the friction coefficient μ. Run the finite element simulation software to calculate the cylindrical upsetting finite element model to obtain the simulated time-load curve of the cylindrical upsetting. Use experience replay to store the interaction trajectory between the intelligent agent and the environment, including state s, action a, reward r, and next state s'. Calculate the deviation between the actual and simulated loads of the upsetting process and generate a reward. Then, update the friction coefficient μ based on the reward.

[0040] The policy network is trained and updated by minimizing the loss function, which is:

[0041] ...Formula (2);

[0042] In formula (2), represents the probability that the policy network outputs action a in state s, represents the state-action value function.

[0043] The training update process of the policy network is expressed as:

[0044] ...Formula (3);

[0045] In formula (3), T is a constant less than 1.

[0046] The critic network is trained and updated by minimizing the temporal query score TD function, which is:

[0047] ......Formula (4).

[0048] In formula (4), y is the TD target, and the calculation formula is:

[0049] ...Formula (5);

[0050] In formula (5), s' is the next state, a' is the next output action, and γ is the discount factor.

[0051] The training update process of the critic network is expressed as:

[0052] ......Formula (6).

[0053] When the friction coefficient reward function r is minimized, the training is stopped and the friction coefficient μ is output as the calibrated large cylinder forging simulation friction coefficient.

[0054] The calibrated friction coefficient value, μ, was substituted into finite element simulation software to construct a finite element simulation model for the 206T cylinder punching process. The simulated punching time-load data was calculated and compared with actual production data to determine the maximum deviation. The results demonstrate the accuracy and versatility of the calibrated friction coefficient for 12Cr2Mo1V cylinders.

Claims

1. A friction coefficient calibration method for large cylinder forging simulation based on reinforcement learning, characterized in that: The following steps are involved: Step 1: Collect the temperature field, reduction, and load data of the barrel forging process, correlate the temperature field, reduction, and load data at the same time, use the temperature field and reduction as inputs for the barrel forging simulation, and use the load data as a criterion for judging the load deviation of the simulation output; Step 2: Build a reinforcement learning model. Define the friction coefficient μ of the cylinder material as the state space, with a value range of 0.05 to 0.95; define the fine-tuning Δμ of the friction coefficient value as the action space, with a value range of ±0.01; and define the friction coefficient reward function r as the negative value of the deviation between the actual and simulated loads in the cylinder upsetting process: ...Formula (1); In formula (1), r is the reward function value, is the simulated load value at the i-th second, is the actual load value at the i-th second; Step 3: Implement the Deep Q-Network (DDPG) algorithm using TensorFlow’s TF-Agents reinforcement learning library. The DDPG algorithm uses a policy network to generate actions Δμ and a critic network to evaluate the quality of the actions Δμ. Step 4: Establish the geometric model of the ingot body, upsetting cover and turntable for the cylinder upsetting simulation, import the geometric model into the finite element simulation software for assembly, divide the mesh, define the cylinder ingot body material, set the billet temperature, mold temperature, upsetting ratio, mold pressing speed and heat exchange coefficient; Step 5: Initialize the state space by setting the friction coefficient μ, run the finite element simulation software, calculate the cylinder upsetting finite element model, obtain the simulation time-load curve of the cylinder upsetting, calculate the deviation between the actual and simulated loads of the upsetting process and generate a reward, and then update the friction coefficient μ based on the reward; The policy network is trained and updated by minimizing the loss function, which is: ...Formula (2); In formula (2), represents the probability that the policy network outputs action a in state s, represents the state-action value function; The training update process of the policy network is expressed as: ...Formula (3); In formula (3), T is a constant less than 1; The critic network is trained and updated by minimizing the temporal query score TD function, which is: ...Formula (4); In formula (4), y is the TD target, and the calculation formula is: ...Formula (5); In formula (5), s' is the next state, a' is the next output action, and γ is the discount factor; The training update process of the critic network is expressed as: ...Formula (6); When the friction coefficient reward function r is minimized, the training is stopped and the friction coefficient μ is output as the calibrated large cylinder forging simulation friction coefficient.

2. The method for calibrating the friction coefficient of large cylinder forging simulation based on reinforcement learning according to claim 1 is characterized in that: In step 4, the number of grids for the ingot body is divided into 100,000, and the number of grids for the upsetting cover and the turntable is divided into 60,000.

3. The method for calibrating the friction coefficient of large cylinder forging simulation based on reinforcement learning according to claim 1 is characterized in that: In step 5, the interaction trajectory between the agent and the environment is stored using experience replay, including state s, action a, reward r, and next state s'.

Images