Bearing heat treatment parameter dynamic optimization method based on deep Q network (DQN)

Abstract

The invention discloses a bearing heat treatment parameter optimization method based on a deep Q network (DQN). Firstly, a state space and an action space in the heat treatment process of the bearing are defined, the state comprises physical sizes (outer diameter, inner diameter and height) of the bearing and heat treatment parameters (temperatures of a first zone to a fourth zone, mesh belt frequency and the number of charging layers), and the action is adjustment of the parameters. Thirdly, a deep neural network approximate Q function is constructed, and a reward function is designed based on the closeness degree of the quenching hardness and the target hardness by maximizing expected reward training; the intelligent agent interacts with the environment, executes actions and observes results, and stores and samples historical experience by using an experience playback mechanism, so that data correlation is reduced, and learning efficiency is improved. And periodically updating the stable learning process of the target network through balanced exploration and utilization of an epsilon-greedy strategy. Finally, the model can adjust heat treatment parameters in real time, the quenching hardness is optimized, and the bearing quality is improved.

Description

Technical Field

[0001] The present invention belongs to the field of bearing manufacturing, and is specifically a method for dynamically optimizing bearing heat treatment parameters based on a deep Q network (DQN) Background Art

[0002] As a key rotating component in a mechanical system, the performance of bearings directly affects the operating efficiency and life of the entire equipment. The heat treatment process of bearings is an important part of improving their performance. Through this process, the mechanical properties of materials, such as hardness, wear resistance and toughness, can be significantly improved. Traditional optimization of bearing heat treatment parameters mainly relies on empirical settings and trial and error methods, which are inefficient and difficult to adapt to changing production conditions and material properties. With the development of Industry 4.0 and intelligent manufacturing, higher requirements are placed on the precise control of bearing heat treatment processes. In this context, people have begun to explore the application of machine learning and artificial intelligence technologies to optimize heat treatment parameters. In existing studies, there are attempts to use traditional machine learning algorithms, such as support vector machines (SVMs) and artificial neural networks (ANNs), for parameter optimization. However, these methods usually require a large amount of labeled data and have poor adaptability in dynamically changing production environments.

[0003] Zhenglei He designed a multi-agent reinforcement learning (MARL) system based on deep Q-network (DQN) to optimize multiple objectives in the textile manufacturing process, such as quality, productivity and cost, to achieve overall performance and product quality improvement. Min Kong designed a new carbon emission reduction engineering method based on deep Q-learning (DQN) algorithm for the energy efficiency scheduling problem of single batch processing machines in semiconductor manufacturing. It optimizes energy consumption and production efficiency through intelligent scheduling strategies, and reduces carbon emissions and operating costs while ensuring that production tasks are completed on time. Dennis Gankin proposed a modular production control method based on multi-agent deep Q-learning (DQN). By assigning an agent to each key link in the production process and allowing these agents to autonomously learn the optimal decision-making strategy through deep Q-learning algorithms, the automation and intelligence of production scheduling are realized, thereby improving production efficiency, reducing energy consumption, and enhancing adaptability to changes in the production environment. It is especially suitable for high-tech industries such as semiconductor manufacturing that require high automation and flexible scheduling.

[0004] Although DQN has achieved certain success in other fields, its application in the optimization of bearing heat treatment parameters is relatively rare. In the prior art, there is no effective method that can use DQN to achieve dynamic optimization of bearing heat treatment parameters to achieve the purpose of improving product quality and reducing energy consumption and cost. Summary of the invention

[0005] The purpose of the present invention is to provide a dynamic optimization method for bearing heat treatment parameters based on deep Q network (DQN), which can dynamically adjust the heat treatment parameters according to real-time data to achieve accurate control of performance indicators such as bearing quenching hardness. Through the present invention, the intelligent level of bearing production can be improved, the production process can be optimized, the cost can be reduced, and ultimately the overall performance and market competitiveness of the product can be improved.

[0006] The technical solution adopted by the present invention is:

[0007] A method for dynamic optimization of bearing heat treatment parameters based on a deep Q network (DQN), comprising the following steps:

[0008] S1. Define the state space during the bearing heat treatment process;

[0009] S2, define the executable action space;

[0010] S3. Construct a deep neural network as an approximation of the Q function;

[0011] S4. Design a reward function to evaluate the effectiveness of the actions taken by the agent.

[0012] S5. Initialize the experience replay buffer D to store tuples of state (s), action (a), reward (r), and new state (s′):

[0013] S6, adopt ε-greedy strategy for action selection, balancing exploration and exploitation;

[0014] S7, during the heat treatment, the agent selects an action based on the current state and executes it, observing the new state (s′) and the reward (r) obtained;

[0015] S8, adopt the experience replay mechanism, by storing the tuple of state (s), action (a), reward (r) and new state (s′), using these historical experiences to reduce data correlation and improve learning efficiency;

[0016] S9, randomly extract a batch of experiences from the experience replay buffer for training the deep neural network;

[0017] S10, setting two identical deep neural networks, one for generating the current Q value (Q network) and the other for generating the target Q value (target network) to stabilize the training process;

[0018] S11, regularly update the target network parameters, copy the parameters of the Q network to the target network at specific time steps to update the parameters of the target network to improve learning efficiency;

[0019] S12. Adjust the hyperparameters of the DQN model, including the learning rate (α), discount factor (γ), exploration rate (ε), and experience replay buffer size;

[0020] S13, simulate the heat treatment process in a virtual environment, use the trained DQN model to guide parameter adjustment, and verify the effectiveness of the model;

[0021] S14, deploy the trained model to the actual heat treatment production line and adjust the parameters in real time to optimize the quenching hardness;

[0022] S15. Based on the performance of the model in the actual production environment, continue to collect data and iteratively optimize the model.

[0023] Furthermore, step S1 is specifically as follows:

[0024] The state space in the bearing heat treatment process refers to all variables that may affect the decision during the heat treatment process. Specifically, it includes:

[0025] Definition: Outside diameter (D) is the outside diameter of the bearing in millimeters (mm).

[0026] Definition: Bore diameter (d) is the inside diameter of the bearing in millimeters (mm).

[0027] Define height (H) as the height of the bearing in millimeters (mm).

[0028] The temperatures of zone 1 to zone 4 (T1, T2, T3, T4) are defined as the temperatures of different zones during the heat treatment process, in degrees Celsius (°C).

[0029] The mesh belt frequency (F) is defined as the moving frequency of the mesh belt, with the unit being Hertz (Hz).

[0030] The number of loading layers (L) is defined as the number of layers of bearings in the furnace.

[0031] The state space is represented by the following formula: State space = {D, d, H, T1, T2, T3, T4, F, L}

[0032] Furthermore, step S2 is specifically as follows:

[0033] The action space refers to all possible actions that the agent can take in the current state. The action space is expressed by the following formula: Action space = {ΔT1, ΔT2, ΔT3, ΔT4, ΔF, ΔL}

[0034] Among them, ΔT1 is the increase or decrease of the temperature of zone 1, ΔT2 is the increase or decrease of the temperature of zone 2, ΔT3 is the increase or decrease of the temperature of zone 3, ΔT4 is the increase or decrease of the temperature of zone 4, ΔF is the increase or decrease of the mesh belt frequency, and ΔL is the increase or decrease of the number of furnace layers.

[0035] Use the temperature control algorithm to adjust the temperature. The formula is: T new =T current +ΔT·sign(Q(s,a)-Q tar get ). Among them, T new is the new temperature setting, T current is the current temperature, ΔT is the temperature adjustment step, and sign is the sign function, which determines whether the temperature increases or decreases based on the difference between the Q value and the target Q value.

[0036] Use the furnace loading layer optimization algorithm to adjust the furnace loading layer number. The formula is: Among them, L new is the number of new furnace layers, L current is the current number of furnace layers, and ΔL is the step size for adjusting the number of layers.

[0037] Use the mesh belt frequency adjustment algorithm to adjust the mesh belt frequency. The formula is: Among them, F new is the new belt frequency, F current is the current belt frequency, ΔF is the frequency adjustment step, and σ is the standard deviation parameter.

[0038] Furthermore, step S3 is specifically as follows:

[0039] A multi-layer perceptron (MLP) structure is used to build a deep neural network. The network layers are constructed by the following steps:

[0040] Construct input layer: Define an input layer to receive the state vector consisting of the physical dimensions of the bearing (outer diameter, inner diameter, height) and heat treatment parameters (temperature of zones 1 to 4, mesh belt frequency, number of furnace layers).

[0041] Building hidden layers: Add two hidden layers after the input layer, each containing 128 neurons. These neurons use the ReLU activation function to introduce nonlinearity to learn the complex relationship between state and action.

[0042] Construct the output layer: Construct the output layer after the last hidden layer. The number of neurons in this layer matches the dimension of the action space and outputs the expected Q value of each action.

[0043] Network training: Use mean squared error (MSE) as the loss function and minimize the loss function through stochastic gradient descent (SGD) or its variants (such as Adam optimizer) to train the network parameters.

[0044] Target network synchronization: Set up a target network with the same structure as the main network, and regularly copy the parameters of the main network to the target network to stabilize the training process.

[0045] Through these steps, the constructed deep neural network can approximate the Q function and provide the expected Q value for each state and action pair. The calculation formula of Q value is: Q tar get =r t+1 +γmax a′ Q(S t+1 ,a′;θ target )

[0046] Use the gradient descent algorithm to optimize the parameters of DQN. The formula is:

[0047] Among them, Q target is the target Q value, r t+1 is the immediate reward obtained at time step t+1, γ is the discount factor, and θ tar get are the parameters of the target network. t+1 are the network parameters at time step t+1, θ t are the network parameters at time step t, α is the learning rate, is the gradient of the loss function L with respect to the parameter θ.

[0048] Furthermore, step S4 is specifically as follows:

[0049] The degree of closeness between the quenching hardness (H_actual) and the preset target hardness (H_target) gives positive rewards or penalties. The reward function is designed as:

[0050] Furthermore, step S5 is specifically as follows:

[0051] Experience = (s, a, r, s′). If the size of D exceeds the preset capacity N, some old experiences are randomly removed. The update formula of the experience playback buffer is: D←D∪{(s t ,a t ,r t ,s t+1 )};

[0052] Furthermore, step S6 is specifically as follows:

[0053] Use the ε-greedy strategy to select actions. This strategy selects a random action with a probability of ε and selects the action with the highest current Q value with a probability of 1-ε. The formula is expressed as:

[0054]

[0055] Further, step S8 is specifically as follows:

[0056] By monitoring the physical dimensions of the bearing (outer diameter, inner diameter, height) and heat treatment parameters (temperature in zones 1 to 4, mesh belt frequency, number of furnace layers) in real time, these parameters are used to construct a description of the current state.

[0057] The DQN model is used to adjust the heat treatment parameters by evaluating possible actions based on the current state to achieve the optimization goal.

[0058] By comparing the actual value of the quenching hardness with the preset target hardness, this comparison result is used to set the reward to evaluate the effectiveness of the actions taken by the intelligent agent.

[0059] The action of the intelligent agent is used to update the state of the environment. By monitoring the changes in the physical dimensions and heat treatment parameters of the bearing, these updated parameters are used to describe the new state.

[0060] Furthermore, step S10 is specifically as follows:

[0061] Provide a relatively stable Q value estimate through the target network to facilitate the calculation of the target y value. The specific steps are as follows:

[0062] Calculate the target y value: Based on the immediate reward r and the next state s', use the target network to calculate the maximum Q' value, that is: y = r + γmax a′ Q′(s′,a′,θ - ). Where γ is the discount factor, which represents the current value of future rewards.

[0063] Calculate the loss function: Use the calculated target y value and the Q value of the current network to calculate the loss function, which is used to update the parameters of the current Q network.

[0064] Define the loss function: L(θ)=E[(y t -Q(s t ,a t ;θ)) 2 ]

[0065] With this approach, we are able to stably update the Q-network parameters to approach the optimal thermal treatment parameter adjustment strategy.

[0066] Further, step S13 is specifically as follows:

[0067] A multi-objective optimization algorithm is used to perform weighted summation of performance indicators such as wear resistance and toughness.

[0068] The specific formula is as follows: Q(s,a)=ω1·Q1(s,a)+ω2·Q2(s,a)+…+ω n Q n (s,a) where Q1(s,a), Q2(s,a), …, Qn (s,a) are the Q values ​​of different performance indicators, ω1, ω2, …, ω n is the corresponding weight.

[0069] Compared with the prior art, the beneficial technical effects of the present invention are:

[0070] 1. The dynamic optimization method of bearing heat treatment parameters based on deep Q network (DQN) proposed in this invention successfully applies deep reinforcement learning technology to the parameter optimization of bearing heat treatment process. It realizes the improvement of product quality and production efficiency by optimizing heat treatment parameters in real time during the bearing production process, while reducing energy consumption and production costs, and ensuring the stability and reliability of the production process.

[0071] 2. The present invention introduces a deep Q network (DQN) model. Through interactive learning with the environment, the intelligent agent can automatically adjust the heat treatment parameters to adapt to the changing production conditions. This method not only reduces the reliance on expert experience and trial and error methods, but also improves the accuracy and response speed of parameter optimization, providing an effective technical means for the dynamic adjustment of bearing heat treatment parameters.

[0072] 3. The DQN model used in the present invention can effectively learn and predict the optimal heat treatment parameters in actual situations with less data, reducing the complexity of model training and computing resource requirements. At the same time, the DQN model has the advantages of high training efficiency, easy adjustment and deployment, and good generalization ability, which facilitates the real-time optimization of bearing heat treatment parameters and ensures the efficient operation and stable performance of the model in the actual production environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] Figure 1 It is a method flow chart of an embodiment of the present invention.

[0074] Figure 2 It is a DQN model architecture diagram of the present invention.

[0075] Figure 3 Schematic diagram of the DQN model reward function of the present invention. DETAILED DESCRIPTION

[0076] like Figure 1 As shown, a dynamic optimization method for bearing heat treatment parameters based on a deep Q network (DQN) includes the following steps:

[0077] S1. Define the state space during the bearing heat treatment process;

[0078] Furthermore, step S1 is specifically as follows:

[0079] The state space in the bearing heat treatment process refers to all variables that may affect the decision during the heat treatment process. Specifically, it includes:

[0080] Definition: Outside diameter (D) is the outside diameter of the bearing in millimeters (mm).

[0081] Definition: Bore diameter (d) is the inside diameter of the bearing in millimeters (mm).

[0082] Define height (H) as the height of the bearing in millimeters (mm).

[0083] The temperatures of zone 1 to zone 4 (T1, T2, T3, T4) are defined as the temperatures of different zones during the heat treatment process, in degrees Celsius (°C).

[0084] The mesh belt frequency (F) is defined as the moving frequency of the mesh belt, with the unit being Hertz (Hz).

[0085] The number of loading layers (L) is defined as the number of layers of bearings in the furnace.

[0086] The state space is represented by the following formula: State space = {D, d, H, T1, T2, T3, T4, F, L}

[0087] S2, define the executable action space;

[0088] Furthermore, step S2 is specifically as follows:

[0089] The action space refers to all possible actions that the agent can take in the current state. The action space is expressed by the following formula: Action space = {ΔT1, ΔT2, ΔT3, ΔT4, ΔF, ΔL}

[0090] Among them, ΔT1 is the increase or decrease of the temperature of zone 1, ΔT2 is the increase or decrease of the temperature of zone 2, ΔT3 is the increase or decrease of the temperature of zone 3, ΔT4 is the increase or decrease of the temperature of zone 4, ΔF is the increase or decrease of the mesh belt frequency, and ΔL is the increase or decrease of the number of furnace layers.

[0091] Use the temperature control algorithm to adjust the temperature. The formula is: T new =T current +ΔT·sign(Q(s,a)-Q target ). Among them, T new is the new temperature setting, T current is the current temperature, ΔT is the temperature adjustment step, and sign is the sign function, which determines whether the temperature increases or decreases based on the difference between the Q value and the target Q value.

[0092] Use the furnace loading layer optimization algorithm to adjust the furnace loading layer number. The formula is: Among them, L new is the number of new furnace layers, L current is the current number of furnace layers, and ΔL is the step size for adjusting the number of layers.

[0093] Use the mesh belt frequency adjustment algorithm to adjust the mesh belt frequency. The formula is: Among them, F new is the new belt frequency, F current is the current belt frequency, ΔF is the frequency adjustment step, and σ is the standard deviation parameter.

[0094] S3. Construct a deep neural network as an approximation of the Q function;

[0095] Furthermore, step S3 is specifically as follows:

[0096] A multi-layer perceptron (MLP) structure is used to build a deep neural network. The network layers are constructed by the following steps:

[0097] Construct input layer: Define an input layer to receive the state vector consisting of the physical dimensions of the bearing (outer diameter, inner diameter, height) and heat treatment parameters (temperature of zones 1 to 4, mesh belt frequency, number of furnace layers).

[0098] Building hidden layers: Add two hidden layers after the input layer, each containing 128 neurons. These neurons use the ReLU activation function to introduce nonlinearity to learn the complex relationship between state and action.

[0099] Construct the output layer: Construct the output layer after the last hidden layer. The number of neurons in this layer matches the dimension of the action space and outputs the expected Q value of each action.

[0100] Network training: Use mean squared error (MSE) as the loss function and minimize the loss function through stochastic gradient descent (SGD) or its variants (such as Adam optimizer) to train the network parameters.

[0101] Target network synchronization: Set up a target network with the same structure as the main network, and regularly copy the parameters of the main network to the target network to stabilize the training process.

[0102] Through these steps, the constructed deep neural network can approximate the Q function and provide the expected Q value for each state and action pair. The calculation formula of Q value is: Q tar get =r t+1 +γmax a′ Q(S t+1 ,a′;θ target )

[0103] Use the gradient descent algorithm to optimize the parameters of DQN. The formula is:

[0104] Among them, Q target is the target Q value, rt+1 is the immediate reward obtained at time step t+1, γ is the discount factor, and θ target are the parameters of the target network. t+1 are the network parameters at time step t+1, θ t are the network parameters at time step t, α is the learning rate, is the gradient of the loss function L with respect to the parameter θ.

[0105] S4. Design a reward function to evaluate the effectiveness of the actions taken by the agent.

[0106] Furthermore, step S4 is specifically as follows:

[0107] The degree of closeness between the quenching hardness (H_actual) and the preset target hardness (H_target) gives positive rewards or penalties. The reward function is designed as:

[0108] S5. Initialize the experience replay buffer D to store tuples of state (s), action (a), reward (r), and new state (s′):

[0109] Furthermore, step S5 is specifically as follows:

[0110] Experience = (s, a, r, s′). If the size of D exceeds the preset capacity N, some old experiences are randomly removed. The update formula of the experience playback buffer is: D←D∪{(s t ,a t ,r t ,s t+1 )};

[0111] S6, adopt ε-greedy strategy for action selection, balancing exploration and exploitation;

[0112] Furthermore, step S6 is specifically as follows:

[0113] Use the ε-greedy strategy to select actions. This strategy selects a random action with a probability of ε and selects the action with the highest current Q value with a probability of 1-ε. The formula is expressed as:

[0114]

[0115] S7. During the heat treatment process, the agent selects actions based on the current state and executes them, observing the new state (s′) and the reward (r) obtained.

[0116] S8. Adopt the experience replay mechanism to use these historical experiences to reduce data correlation and improve learning efficiency by storing tuples of state (s), action (a), reward (r), and new state (s′).

[0117] Further, step S8 is specifically as follows:

[0118] By monitoring the physical dimensions of the bearing (outer diameter, inner diameter, height) and heat treatment parameters (temperature in zones 1 to 4, mesh belt frequency, number of furnace layers) in real time, these parameters are used to construct a description of the current state.

[0119] The DQN model is used to adjust the heat treatment parameters by evaluating possible actions based on the current state to achieve the optimization goal.

[0120] By comparing the actual value of the quenching hardness with the preset target hardness, this comparison result is used to set the reward to evaluate the effectiveness of the actions taken by the intelligent agent.

[0121] The action of the intelligent agent is used to update the state of the environment. By monitoring the changes in the physical dimensions and heat treatment parameters of the bearing, these updated parameters are used to describe the new state.

[0122] S9, randomly extract a batch of experiences from the experience replay buffer for training the deep neural network;

[0123] S10, setting two identical deep neural networks, one for generating the current Q value (Q network) and the other for generating the target Q value (target network) to stabilize the training process;

[0124] Furthermore, step S10 is specifically as follows:

[0125] Provide a relatively stable Q value estimate through the target network to facilitate the calculation of the target y value. The specific steps are as follows:

[0126] Calculate the target y value: Based on the immediate reward r and the next state s', use the target network to calculate the maximum Q' value, that is: y = r + γmax a′ Q′(s′,a′,θ - ). Where γ is the discount factor, which represents the current value of future rewards.

[0127] Calculate the loss function: Use the calculated target y value and the Q value of the current network to calculate the loss function, which is used to update the parameters of the current Q network.

[0128] Define the loss function: L(θ)=E[(y t -Q(s t ,a t ;θ)) 2 ]

[0129] With this approach, we are able to stably update the Q-network parameters to approach the optimal thermal treatment parameter adjustment strategy.

[0130] S11. Regularly update the target network parameters. At specific time steps, copy the parameters of the Q network to the target network to update the parameters of the target network to improve learning efficiency.

[0131] S12. Adjust the hyperparameters of the DQN model, including the learning rate (α), discount factor (γ), exploration rate (ε), and experience replay buffer size;

[0132] S13. Simulate the heat treatment process in a virtual environment, use the trained DQN model to guide parameter adjustment, and verify the effectiveness of the model.

[0133] Further, step S13 is specifically as follows:

[0134] A multi-objective optimization algorithm is used to perform weighted summation of performance indicators such as wear resistance and toughness.

[0135] The specific formula is as follows: Q(s,a)=ω1·Q1(s,a)+ω2·Q2(s,a)+…+ω n Q n (s,a) where Q1(s,a), Q2(s,a), …, Q n (s,a) are the Q values ​​of different performance indicators, ω1, ω2, …, ω n is the corresponding weight.

[0136] S14, deploy the trained model to the actual heat treatment production line and adjust the parameters in real time to optimize the quenching hardness;

[0137] S15. Based on the performance of the model in the actual production environment, continue to collect data and iteratively optimize the model.

[0138] The above is a detailed description of the embodiments of the present invention in conjunction with the accompanying drawings, but the present invention is not limited to the above embodiments. Within the knowledge scope of ordinary technicians in the field, various changes can be made without departing from the purpose of the present invention, which are all within the protection scope of this technology.

Claims

1. A dynamic optimization method for bearing heat treatment parameters based on deep Q network (DQN), characterized in that: The following steps are involved: S1. Define the state space during the bearing heat treatment process; S2, define the executable action space; S3. Construct a deep neural network as an approximation of the Q function; S4. Design a reward function to evaluate the effectiveness of the actions taken by the agent; S5. Initialize the experience replay buffer D to store tuples of state (s), action (a), reward (r), and new state (s′): S6, adopt ε-greedy strategy for action selection, balancing exploration and exploitation; S7, during the heat treatment, the agent selects an action based on the current state and executes it, observing the new state (s′) and the reward (r) obtained; S8, adopt the experience replay mechanism, by storing the tuple of state (s), action (a), reward (r) and new state (s′), using these historical experiences to reduce data correlation and improve learning efficiency; S9, randomly extract a batch of experiences from the experience replay buffer for training the deep neural network; S10, setting two identical deep neural networks, one for generating the current Q value (Q network) and the other for generating the target Q value (target network) to stabilize the training process; S11, regularly update the target network parameters, copy the parameters of the Q network to the target network at specific time steps to update the parameters of the target network to improve learning efficiency; S12. Adjust the hyperparameters of the DQN model, including the learning rate (α), discount factor (γ), exploration rate (ε), and experience replay buffer size; S13, simulate the heat treatment process in a virtual environment, use the trained DQN model to guide parameter adjustment, and verify the effectiveness of the model; S14, deploy the trained model to the actual heat treatment production line and adjust the parameters in real time to optimize the quenching hardness; S15. Based on the performance of the model in the actual production environment, continue to collect data and iteratively optimize the model.

Images