Multi-agent reinforcement learning microwave drilling control method based on physical information neural network assistance

Abstract

The present application relates to a multi-agent reinforcement learning microwave drilling control method based on physical information neural network assistance, the present application fuses physical model constraint and deep reinforcement learning, outputs duty cycle, cooling flow, rock breaking speed and other control actions through multi-agent cooperation, constructs a closed-loop control architecture of "physical process-PINN prediction-multi-agent decision-action execution". The attention mechanism is used to realize the information interaction between agents, the centralized Critic network and the generalized advantage estimation are used to complete the value evaluation and policy optimization, and the physical consistency is integrated into the learning process through PINN auxiliary reward shaping, forming an integrated structure of physical simulation, prediction, control and safety constraint, which can effectively improve the control precision and sample efficiency, enhance the safety and stability of the system, and is suitable for intelligent microwave drilling control under complex thermal coupling conditions.

Description

Technical Field

[0001] This invention relates to the field of industrial process control technology, and in particular to a microwave drilling control method based on physical information neural network-assisted multi-agent reinforcement learning. Background Technology

[0002] High-power microwave (HPM) drilling technology is an advanced rock-breaking technique widely used in deep-earth resource exploration, geothermal energy development, and shale gas extraction. Compared with traditional mechanical drilling, microwave drilling has significant advantages such as higher drilling efficiency, less tool wear, and less disturbance to the formation. However, a high-power microwave drilling system is a complex, multivariable, strongly coupled system, and its control faces the following technical challenges: (1) Multi-physics field strong coupling problem: The thermal field distribution generated by microwave energy absorption, rock crack propagation caused by thermal stress, the interaction between drill bit mechanical wear and thermal damage, and the balance between cooling system efficiency and thermal management are coupled with each other to form a complex thermo-mechanical coupling system.

[0003] (2) The control variables are complex and mutually restrictive: The system involves multiple control variables such as microwave power (15-85kW), rotation speed (60-190rpm), drilling pressure / feed speed (0.8-5.5mm / rev), and coolant flow rate (15-75L / min), and there are complex coupling relationships between the variables.

[0004] (3) Strict safety constraints: The system operation needs to meet multiple safety constraints such as upper temperature limit (≤160°C), upper stress limit (≤90MPa), vibration limit (≤0.32mm / s), and specific absorption rate (≤2.0W / kg).

[0005] (4) Limitations of existing control methods: Traditional PID control is difficult to handle multivariable strongly coupled systems; single reinforcement learning methods have low sample efficiency, lack physical interpretability, and are difficult to meet industrial safety requirements; traditional physical model methods have low computational efficiency and are difficult to calibrate model parameters. Summary of the Invention

[0006] The purpose of this invention is to propose a microwave drilling control method based on physical information neural network-assisted multi-agent reinforcement learning, which integrates physical model constraints and deep reinforcement learning to achieve efficient, safe, and intelligent drilling process control under complex thermo-mechanical coupling conditions.

[0007] To achieve the above objectives, the present invention provides the following solution: A multi-agent reinforcement learning microwave drilling control method based on physical information neural network assistance includes: S1: Construct a physical simulation environment model for microwave drilling and obtain state observation data; wherein, the physical simulation environment model includes: thermal field model, stress field model, vibration model and efficiency model; S2: Input the state observation data into the multi-task physical information neural network model to obtain the predicted values ​​of physical quantities; S3: Based on the predicted values ​​of the physical quantities, construct the PINN-assisted reward shaping mechanism; S4: Construct a multi-agent reinforcement learning system and use the PINN-assisted reward shaping mechanism to control the multi-agent reinforcement learning system to perform reinforcement learning; S5: Construct a safety constraint mechanism, integrate safety thresholds into the reinforcement learning reward function, punish excessive behavior, and guide the agent to avoid safety risks during the reinforcement learning training phase; wherein, the safety thresholds include preset temperature safety thresholds, preset stress safety thresholds, and preset vibration safety thresholds; S6: During the deployment phase, action selection without exploration noise is adopted. When the security prediction probability obtained based on PINN prediction and security factor is lower than the preset threshold, the security protection protocol is triggered and the system automatically switches to conservative control mode.

[0008] Optionally, the thermal field model is: ; The stress field model is as follows: ; The vibration model is as follows: ; The efficiency model is as follows: ; in, The value represents the drilling speed, and the subscript t is the time step index, representing the dynamic change of the corresponding physical quantity over time. t∈{0,1,2,…,N}, where t=0 is the initial time, and t+1 is the next control step at time t. For microwave power, For cooling flow rate, For rotational speed, For temperature, For stress, For vibration, For efficiency, The Gaussian noise term for temperature prediction follows a mean of 0 and a standard deviation of . T follows a normal distribution. The Gaussian noise term for vibration prediction follows a mean of 0 and a standard deviation of . V follows a normal distribution. The Gaussian noise term for stress prediction follows a mean of 0 and a standard deviation of . It follows a normal distribution.

[0009] Optionally, the multi-task physical information neural network model includes: Shared feature extraction backbone network: Each layer is followed by a batch normalization layer (BatchNorm) and a rectified linear unit (ReLU) activation function to extract shared feature representations; Temperature prediction auxiliary head: It adopts a four-layer network structure to provide auxiliary and supervised temperature prediction; Final physical quantity output layer: Employs a fully connected layer to simultaneously output the predicted values ​​of the physical quantities; wherein, the predicted values ​​of the physical quantities include: temperature, stress, efficiency, safety factor, wear, specific absorptivity (SAR), and vibration.

[0010] Optionally, the multi-task physical information neural network model is further provided with: a physical constraint loss function, used to constrain the prediction to conform to physical laws; The physical constraint loss function includes constraints based on the heat conduction equation, Faraday's law, Ampere's law, momentum equation, continuity equation, and stress equation.

[0011] Optionally, constructing a PINN-assisted reward shaping mechanism includes: The predicted values ​​of the physical quantities are incorporated into the reinforcement learning reward function, resulting in the following total reward function: ; in, For the total reward function, for , for , For PINN prediction consistency rewards, To restrain punishment, The weighting coefficient for PINN prediction consistency reward is used to balance the optimization priority of control performance and physical consistency. The value range is [0.05, 0.2], and the preferred value is 0.1.

[0012] Optionally, the multi-agent reinforcement learning system is constructed using a centralized training and distributed execution CTDE paradigm based on the MAPPO algorithm for multi-agent proximal policy optimization. The multi-agent reinforcement learning system includes: State space input module: used to output state vector; the state vector includes: drilling speed, rotation speed, microwave power, cooling flow rate, temperature, stress, vibration, efficiency, wear, and safety factor; the state vector serves as the global state input to each agent and the centralized Critic network; The Actor network is used to set up four parallel agents: a power agent, a cooling agent, a rotation speed agent, and a drilling speed agent, which are responsible for microwave power control, cooling system control, rotation speed control, and drilling speed control, respectively. Multi-head attention information exchange module: used to realize dynamic information exchange between intelligent agents. It adopts the Q, K, V mechanism, which enables each intelligent agent to adaptively pay attention to the information of other intelligent agents according to the current global state, so as to achieve coordinated decision-making. Hyper network Mixer module: used to dynamically generate adaptive mixing weights based on the global state, and to perform weighted fusion of the outputs of multiple agents; Centralized Critic Network: Used for value estimation and advantage calculation during the training phase; Motion execution module: used to map 4-dimensional continuous motion into physical control quantities and output duty cycle, cooling flow rate, rotation speed and drilling speed to the actuator.

[0013] Optionally, each agent includes a state encoder and a policy network Actor: the state encoder encodes local observations, and the policy network Actor outputs the agent's actions; the outputs of all policy network Actors together form a joint action, which is then input into a centralized Critic network.

[0014] Optionally, integrating security constraint mechanisms into the reinforcement learning outcomes includes: When the temperature exceeds the preset temperature safety threshold, a penalty is applied according to the set ratio; when the stress exceeds the preset stress safety threshold, a penalty is applied according to the set ratio; when the vibration exceeds the preset vibration safety threshold, a penalty is applied according to the set ratio.

[0015] The beneficial effects of this invention are as follows: 1. Significantly improved control accuracy: By integrating PINN with prior physical knowledge, the temperature prediction accuracy reaches ±2°C, and the drilling speed control error is reduced to ±3%, which is more than 50% higher than the traditional PID method.

[0016] 2. Significantly improved sample efficiency: By utilizing physical constraints to reduce the exploration space required for reinforcement learning, the number of training episodes required to achieve the target performance is reduced from 380 to 180, resulting in a 53% improvement in sample efficiency.

[0017] 3. Multi-agent collaborative optimization: Dynamic information exchange between agents is achieved through attention communication mechanism to solve the credit allocation problem and realize the coordinated optimization of multiple control variables.

[0018] 4. Safety performance assurance: Integrated safety constraint mechanism reduces the number of temperature over-limit times from 45 to 5, and improves the overall safety score from 0.45 to 0.91.

[0019] 5. Strong generalization ability: Through physically consistent representation learning, it maintains stable control performance under different working conditions (temperature change ±20°C, rock hardness change ±30%). Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is an overall block diagram of the multi-agent reinforcement learning microwave drilling control method based on physical information neural network assisted by an embodiment of the present invention; Figure 2 This is a schematic diagram of the Industrial PINN network structure and physical constraint loss function according to an embodiment of the present invention; Figure 3 This is a schematic diagram of the MAPPO multi-agent cooperative control architecture according to an embodiment of the present invention; Figure 4 This is a schematic diagram of the PINN-assisted reward shaping and security constraint mechanism according to an embodiment of the present invention; Figure 5 This is a schematic diagram of the process of a multi-agent reinforcement learning microwave drilling control method based on physical information neural network assistance according to an embodiment of the present invention. Detailed Implementation

[0022] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0024] like Figure 1 and Figure 5 As shown, this embodiment proposes a multi-agent reinforcement learning microwave drilling control method based on physical information neural network assistance, including: S1: Construct a physical simulation environment model for microwave drilling and obtain state observation data; wherein, the physical simulation environment model includes: thermal field model, stress field model, vibration model and efficiency model; S2: Input the state observation data into the multi-task physical information neural network model to obtain the predicted values ​​of physical quantities; S3: Based on the predicted values ​​of the physical quantities, construct the PINN-assisted reward shaping mechanism; S4: Construct a multi-agent reinforcement learning system and use the PINN-assisted reward shaping mechanism to control the multi-agent reinforcement learning system to perform reinforcement learning; S5: Construct a safety constraint mechanism, integrate safety thresholds such as temperature, stress, and vibration into the reinforcement learning reward function, punish behaviors that exceed the limits, and guide the agent to avoid safety risks during the reinforcement learning training phase. S6: During the deployment phase, action selection without exploration noise is adopted. When the security prediction probability obtained based on PINN prediction and security factor is lower than the preset threshold, the security protection protocol is triggered and the system automatically switches to conservative control mode.

[0025] This embodiment differs from existing technologies in that it uses the PINN-assisted reward shaping mechanism to directly embed the physical model prediction results into the reinforcement learning optimization process, thereby achieving coupled optimization of physical consistency and control policy learning.

[0026] Specifically, in this embodiment, step S1 includes the following steps: S11. Establish a physical simulation environment for microwave drilling. The state space is a 10-dimensional vector, containing drilling speed d, rotational speed ω, microwave power P, cooling flow rate f, temperature T, stress σ, vibration v, efficiency η, wear w, and safety factor s; the action space is a 4-dimensional continuous space, corresponding to the increments of each control variable. For example... Figure 1 As shown, the controlled object in this embodiment is an environmental / industrial process, including microwave heating (energy input), tool-workpiece interaction (dynamics and force transmission between the drill bit and the rock), temperature / stress / electromagnetic field (multi-physics field state), and sensors and actuators (state acquisition and action execution). Figure 5 As shown, the microwave drilling physical simulation environment in this embodiment corresponds to the "initialize system state" step in the flowchart, which includes a temperature dynamic response model, a stress dynamic response model, a vibration dynamic response model, and an efficiency calculation model.

[0027] The microwave drilling physical simulation environment in this embodiment is a numerical simulation platform based on multiphysics coupling. Its core function is to simulate the electromagnetic field distribution, heat conduction, rock fracturing, and thermal stress evolution during microwave heating. The simulation environment employs a discretization method, dividing the continuous spatial domain into a finite number of control volumes. The spatiotemporal distribution of each physical quantity is obtained by solving the governing equations. The time step of the environment is set to 0.1 seconds, and the spatial resolution is set to a 1mm × 1mm × 1mm cubic grid according to the computational accuracy requirements. To ensure simulation efficiency, the environment adopts an adaptive time step strategy: when physical quantities change drastically, the time step is automatically reduced to 0.01 seconds; when the system tends to stabilize, the time step is increased to 0.5 seconds. The simulation environment's input consists of four control variables (microwave power, cooling flow rate, rotational speed, and drilling speed), and its output consists of ten state variables, forming a complete Markov decision process (MDP) framework.

[0028] S12. Define the ranges of each physical quantity: drilling speed d ranges from 0.8 to 5.5 mm / rev, rotational speed ω ranges from 60 to 190 rpm, microwave power P ranges from 15 to 85 kW, cooling flow rate f ranges from 15 to 75 L / min, temperature T ranges from 45 to 170°C, stress σ ranges from 20 to 95 MPa, vibration v ranges from 0.04 to 0.32 mm / s, efficiency η ranges from 0.48 to 0.92, wear w ranges from 0.1 to 0.45, and safety factor s ranges from 0.3 to 4.0.

[0029] The state space is represented by a ten-dimensional continuous vector, and the action space is a four-dimensional continuous space with values ​​ranging from -1 to 1, corresponding to the increments of four control variables: microwave power duty cycle increment, cooling flow rate increment, rotational speed increment, and drilling speed increment. The mapping relationship between actions and actual physical quantities is: microwave power increment... Mapped to Incremental cooling flow Mapped to Speed ​​increment Mapped to Drilling speed increment Mapped to .

[0030] S13. Establish dynamic response models for temperature, stress, vibration, and efficiency.

[0031] The temperature dynamic response model is established based on the first law of thermodynamics and the heat conduction equation, taking into account factors such as microwave heat source input, coolant heat dissipation, heat radiation loss, and rock heat capacity. The mathematical expression of the model is: Where T(t) is the temperature at the current moment, For time step, Input power to the microwave heat source ( , (This represents the microwave-to-thermal energy conversion efficiency, with a value of 0.85). Heat carried away by the coolant ( h is the convective heat transfer coefficient, and A is the heat transfer area. (coolant inlet temperature) For thermal radiation loss ( ρ is the rock density, c is the rock specific heat capacity, and V is the control volume. The model parameters were obtained through experimental data calibration: for granite, ρ = 2700 kg / m³, c = 790 J / m³. Basalt has a density of ρ = 2950 kg / m³ and a concentration of c = 840 J / m³. .

[0032] The stress dynamic response model is based on thermoelasticity theory, considering the effects of thermal expansion, additional stress caused by thermal gradients, and contributions from mechanical loads. The mathematical expression of the model is: Where E is the elastic modulus of the rock (E=70GPa for granite, E=90GPa for basalt), and α is the coefficient of thermal expansion (…). ), Let ν be the temperature change, and ν be Poisson's ratio (ν=0.25). For thermal stress components, The stress components are induced by mechanical loads. Thermal stress is calculated using the discretized form of the Galerkin finite element method, obtaining the stress values ​​at each node through shape function integration.

[0033] The vibration dynamic response model is based on a multi-degree-of-freedom vibration system, treating the drill bit-drill rod-machine body as a series mass-spring-damped system. The mathematical expression of the model is: Where v(t) is the vibration velocity and x(t) is the vibration displacement. The excitation force is denoted by c, which is related to microwave power and rock hardness. c is the damping coefficient, k is the equivalent stiffness, and m is the equivalent mass. The model parameters were determined through modal analysis experiments: the system's natural frequency is... The damping ratio ζ = 0.02.

[0034] The efficiency calculation model combines empirical formulas with physical models: ,in, The baseline efficiency is 0.72. This is the rock hardness coefficient (range 0.6-1.0, the higher the hardness, the lower the value). This is the microwave power matching coefficient (related to the uniformity of SAR specific absorptivity distribution). This is the drill bit condition coefficient (negatively correlated with wear amount w). This represents the maximum permissible wear of the drill bit (value 0.5).

[0035] Furthermore, the multi-task physical information neural network model includes: Shared feature extraction backbone network: Each layer is followed by BatchNorm and ReLU activation functions to extract shared feature representations; Temperature prediction auxiliary head: It adopts a four-layer network structure to provide auxiliary and supervised temperature prediction; Final physical quantity output layer: Employs a fully connected layer to simultaneously output the predicted values ​​of the physical quantities; wherein, the predicted values ​​of the physical quantities include: temperature, stress, efficiency, safety factor, wear, specific absorptivity (SAR), and vibration.

[0036] Furthermore, the multi-task physical information neural network model also includes a physical constraint loss function, used to constrain the prediction to conform to physical laws; The physical constraint loss function includes constraints based on the heat conduction equation, Faraday's law, Ampere's law, momentum equation, continuity equation, and stress equation.

[0037] Specifically, in this embodiment, step S2 is implemented as follows: like Figure 1 As shown, the overall architecture of the microwave drilling intelligent control system in this embodiment aims to achieve "microwave source control parameter decision-making" and consists of the following closed loop components: (i) Environment / Industrial Process: including microwave heating, tool-workpiece interaction, temperature / stress / electromagnetic field, and sensors and actuators, which are the controlled objects; (ii) State Observation: receiving raw data from the environment / industrial process and outputting state observations / state features, including temperature curves, field distribution maps, and process parameters, to provide input for the neural network model; (iii) Neural Network Model: including a physical field prediction network and a strategy network; the physical field prediction network predicts temperature, stress, electromagnetic field, and flow field based on state observations and supports safety assessment; the strategy network learns control strategies and maps states to actions (duty cycle, cooling, rock breaking speed); (iv) Reinforcement Learning Control Loop: the agent uses the strategy network to select actions, which are to adjust the duty cycle, cooling flow rate, and rock breaking speed; the environment returns feedback signals based on efficiency and safety, thereby completing reinforcement learning and strategy updates. The aforementioned closed loop forms a complete process of "state observation → neural network model (physical field prediction + strategy) → action execution → environmental feedback → strategy update", enabling intelligent decision-making for microwave source control parameters.

[0038] like Figure 2 As shown, the Industrial PINN network structure in this embodiment includes four parts: an input layer, a hidden layer network, a physical constraint module, and a loss function, and a residual connection is provided from the input layer to the output layer.

[0039] The input layer is designed as an 8-dimensional vector (input_dim=8), denoted as x, y, z, t, T, E, φ, ψ in the diagram, corresponding to spatial coordinates (x, y, z), time (t), and operating conditions and electromagnetic parameters. The spatial coordinates are used to identify the prediction target at different locations in the physical field, and the time parameters are used to capture the time-varying characteristics of physical quantities. The input vector undergoes standardization preprocessing, normalizing each dimension to the [0,1] interval to eliminate the influence of dimensional differences on training.

[0040] The hidden layer and output layer structure is as follows: Figure 2 As shown: The main network consists of four hidden layers: 8→64→128→128→64→8, with dimensions of 64, 128, 128, and 64 respectively. The output layer is 8-dimensional (output_dim=8). The output quantities are denoted as E, B, ρ, μ, v, w, cx12, and cx13, corresponding to electromagnetic field, density, viscosity, velocity, and multiphysics coupling variables, used for subsequent derivation of key physical quantities such as temperature, stress, efficiency, safety factor, SAR specific absorptivity, vibration, and wear. The network uses the GELU (Gaussian Error Linear Unit) activation function, expressed as follows: , where Φ is the cumulative distribution function of the standard normal distribution; Dropout is set to 0.1 for each hidden layer to enhance generalization. Residual connections are directly connected from the input layer to the output layer, i.e., y=F(x)+x, which alleviates gradient vanishing and improves the training stability of deep networks.

[0041] The physical constraint module substitutes the network predictions into the physical equations to calculate the PDE residuals, such as... Figure 2 As shown in the "Physical Constraints (PDEConstraints)", it includes six core equations: (1) the heat conduction equation ( (2) Faraday's Law (3) Ampere's Law (4) Momentum equation ( (5) Continuity equation ( (6) Stress equation ( The "Network Prediction" function automatically differentiates and calculates the residuals of each equation; the "PDE Residuals" are then input into the loss function.

[0042] loss function as follows Figure 2 As shown in the image, the "Loss Function" consists of four weighted parts: ,in For data fitting loss, This represents the physical constraint loss (obtained from the PDE residuals). For boundary condition loss, This represents the initial conditional loss. The weights in the graph are: By updating network parameters through "gradient backpropagation," Industrial PINN can satisfy physical laws while fitting data, thereby providing predictions of temperature, stress, electromagnetic field, and flow field for control and safety assessment.

[0043] Specifically, step S3 of this embodiment is implemented as follows: The PINN-assisted reward shaping mechanism in this embodiment deeply integrates physical prediction information into the reinforcement learning reward function, which is one of the core innovations of this embodiment. Traditional reinforcement learning methods rely solely on sparse final reward signals, resulting in low sample efficiency. In contrast, this embodiment introduces the physical prediction information of PINN into the reward function, providing the agent with rich intermediate supervision signals and significantly improving learning efficiency.

[0044] like Figure 4 As shown, in the multi-objective reward and safety constraint integrated control framework of this embodiment, the total reward function consists of four parts, the relationship of which is as follows: (Note in the diagram: Total reward = efficiency reward + security reward + PINN consistency reward - constraint penalty). Efficiency reward Safety Rewards PINN Consistency Rewards Positive contributions should be subject to penalties. This comes from the security monitoring (security constraints) module and is a negative contribution.

[0045] Efficiency rewards are calculated based on system efficiency; safety rewards are calculated based on safety factors and safety-related quantities such as temperature and stress. The ideal temperature range for temperature rewards is 80-120°C, and the safety stress threshold for stress rewards is 60MPa. The weighting is 0.3×temperature + 0.3×stress + 0.2×efficiency + 0.2×safety factor, and the final reward is normalized to the interval [-1,1]. The PINN Consistency Reward is provided by Industrial PINN (PINN Prediction) to encourage predictions that are consistent with physical laws; the diagram clearly states that the reward comes from PINN predictions and supports security assessments. Outputted by the safety monitoring (safety constraint) module, when temperature, stress, or vibration exceeds a set threshold, penalties are applied according to a set ratio and summarized into the constraint penalty item.

[0046] Specifically, step S4 of this embodiment is implemented as follows: The multi-agent reinforcement learning system in this embodiment is based on the Multi-Agent Proximal Policy Optimization (MAPPO) algorithm and adopts the CTDE (Centralized Training, Distributed Execution) paradigm. The system sets up four parallel agents: a power agent responsible for microwave power duty cycle control, a cooling agent responsible for coolant flow control, a rotation speed agent responsible for drill bit rotation speed control, and a drilling speed agent responsible for drilling speed control.

[0047] like Figure 3 As shown, the MAPPO multi-agent cooperative control architecture for multi-agent proximal policy optimization in this embodiment adopts the CTDE (centralized training, distributed execution) paradigm, including a state space input module, four parallel agent networks, a multi-head attention information exchange module, a Hyper network Mixer module, a centralized Critic network, and an action execution module.

[0048] The state space input module outputs a 10-dimensional state vector [d,ω,P,f,T,σ,v,η,w,s], which represents drilling speed d, rotational speed ω, microwave power P, cooling flow rate f, temperature T, stress σ, vibration v, efficiency η, wear w, and safety factor s. This vector serves as the global state input for each agent and the centralized Critic.

[0049] Four parallel intelligent agents, such as Figure 3 As shown, the agents are: Power Agent (Actor-p), responsible for microwave power / duty cycle control; Cooling Agent (Actor-f), responsible for cooling flow control; Rotation Agent (Actor-w), responsible for drill bit rotation speed control; and Drilling Speed ​​Agent (Actor-d), responsible for drilling speed control. Each agent contains a state encoder and a policy network (Actor): the state encoder encodes local observations, and the policy network (Actor) outputs the agent's actions. The outputs of all Actors collectively form joint actions, which are then input into a centralized Critic.

[0050] The multi-head attention information exchange module realizes dynamic information exchange between agents (Multi-Head Attention). It adopts the Q, K, V mechanism, which enables each agent to adaptively pay attention to the information of other agents according to the current global state, so as to achieve coordinated decision-making.

[0051] HyperNetwork Mixer module, such as Figure 3 As shown, the mixed weights are dynamically generated based on the global state: ,in This represents the global state. W1 and W2 are used to perform differentiable mixing of the outputs of multiple agents, achieving state-dependent value decomposition and action integration.

[0052] The input to a centralized Critic network (centralized Critic) is the global state. Together with joint actions, the output is Q(s,a) or V(s), used for value estimation and advantage calculation during the training phase; during the execution phase, only the Actors are used, and the Critic does not participate in the decision-making.

[0053] The action execution module maps a 4-dimensional continuous action a∈[-1,1]4 to physical control quantities: (Duty cycle) (Cooling flow rate) (Speed) (Drilling speed), output to the actuator. The actuator employs saturation and limiting to prevent sudden changes in control quantity from impacting the equipment.

[0054] Furthermore, the security constraint mechanisms for integrating reinforcement learning outcomes include: When the temperature exceeds the preset temperature safety threshold, a penalty is applied according to the set ratio; when the stress exceeds the preset stress safety threshold, a penalty is applied according to the set ratio; when the vibration exceeds the preset vibration safety threshold, a penalty is applied according to the set ratio.

[0055] Specifically, in this embodiment, step S5 is implemented as follows: The security constraint mechanism in this embodiment establishes a multi-layered security protection system, which is a key design feature to ensure the safe operation of industrial systems. Figure 4 The terms "security constraint mechanism (security prediction)" and "security monitoring (security constraint)" correspond to each other.

[0056] like Figure 4 As shown, the safety constraint mechanism performs threshold judgments on three key physical quantities: if "Temperature T exceeds the set threshold?", a penalty is applied according to a set ratio (temperature penalty); if "Stress σ exceeds the set threshold?", a penalty is applied according to a set ratio (stress penalty); if "Vibration v exceeds the set threshold?", a penalty is applied according to a set ratio (vibration penalty). These penalties are summarized in the safety monitoring (safety constraint) module and used as... Input the total reward calculation. In this embodiment, a penalty of 0.005 / °C is applied when the temperature exceeds 160°C, a penalty of 0.005 / MPa is applied when the stress exceeds 90MPa, and a penalty of 0.2 / (mm / s) is applied when the vibration exceeds 0.32mm / s. It is noted in the figure that the safety factor target value can be dynamically adjusted according to the optimization target.

[0057] like Figure 4As shown in the control decision logic in the lower right corner, the safety monitoring module further determines whether the "safety prediction probability is lower than the preset threshold?" (This safety prediction probability is obtained based on PINN prediction and the safety factor). If yes, the safety protection protocol is triggered, and the system automatically switches to conservative control mode; if no, normal control output is performed. The diagram also notes that a strategy with no exploratory noise is used during deployment, meaning a deterministic strategy is employed during execution, without adding exploratory noise, to ensure stable and predictable control behavior.

[0058] In this embodiment, the target value for the safety factor is set to 2.0, with an adjustment range of 1.5-4.0. It can be dynamically adjusted according to the optimization objective: when the optimization objective emphasizes efficiency, it can be appropriately reduced; when the optimization objective emphasizes safety, it can be increased to 3.0-4.0. When the safety prediction probability is lower than a preset threshold (e.g., 0.85), the protection protocol is triggered and the system switches to conservative control parameters (e.g., power is reduced to 50%, speed is reduced to 60 rpm, and cooling flow rate is increased to 60 L / min).

[0059] S5 is a closed-loop safety constraint mechanism that runs through the entire reinforcement learning process. Its core functions are divided into two layers: Training phase: In the reinforcement learning process in step 4, safety constraints (temperature, stress, vibration threshold) are incorporated into the reward function to punish behaviors that exceed the limits, guiding the agent to avoid safety risks during the learning phase, rather than only correcting them in the outcome phase; Deployment phase: In conjunction with the S6 security protection protocol, perform real-time security verification on the control actions output in step 4 to ensure that the executed actions meet industrial safety requirements.

[0060] Specifically, in this embodiment, step S6 is implemented as follows: like Figure 5 As shown, the control method in this embodiment is executed according to the following process: First, initialization is performed—network parameters (Actor / Critic network and PINN network weights) are initialized, and system states (temperature, stress, power, cooling flow rate, drilling speed, safety factor, etc.) are initialized; then, the system states are input into the PINN network to predict physical quantities such as temperature / stress and calculate PDE residuals; next, each agent makes independent decisions and outputs actions related to power, cooling, drilling speed, and safety. Before the action is executed, it is determined whether the action meets physical constraints (temperature / stress threshold, safety factor constraints): if not, the process returns to the independent decision-making step of each agent to regenerate the action; if it meets the constraints, the actions are merged and applied to the system to obtain the next system state and calculate the global reward; then, the MAPPO training is optimized according to the multi-agent proximal strategy to update the network parameters of all agents; the above process is cyclically fed back to form a complete training and execution closed loop.

[0061] During the deployment phase, action selection without exploration noise is adopted, combined with a safety check mechanism to ensure the safe operation of the system. The deployment process is as follows: (1) Obtain the current system status from the sensor; (2) Input the status into Industrial PINN to predict physical quantities; (3) Calculate the safety margin of each physical quantity; (4) If the safety margin is lower than the threshold, trigger the protection protocol; (5) Otherwise, input the status into MAPPO to generate control actions; (6) Send the actions to the actuator; (7) Return to step (1) and repeat the process.

[0062] A pinn_marl_history historical database was established to record timestamp sequences, predicted temperature / stress / power value sequences, actual observation value sequences, reward value sequences, and action sequences, for system performance monitoring, offline analysis, and model tuning. The database is implemented using SQLite, retains a maximum of 1000 historical records, and supports visualization of prediction errors and control performance curves. The database table structure includes: timestamp, state (10-dimensional state vector, JSON format), prediction (7-dimensional prediction vector), observation (7-dimensional observation vector), reward (scalar reward value), and action (4-dimensional action vector).

[0063] The offline analysis functions include: (1) prediction error statistics, calculating the root mean square error (RMSE) and mean absolute error (MAE) of each physical quantity; (2) control performance evaluation, analyzing the mean and variance of key indicators such as drilling speed and efficiency; (3) anomaly detection, identifying temperature and stress over-limit events and counting their frequency; (4) strategy analysis, visualizing action distribution and reward curves, and supporting strategy playback and debugging.

[0064] The simulation experiment for this embodiment is as follows: To verify the effectiveness of this embodiment, six mainstream methods were selected for comparison, including IPPO, VDN, QMIX, MAPPO, MADDPG, and MASAC. In the experiment, all algorithms used the same environment settings and experimental parameters.

[0065] Table 1 Comparison of reward values ​​for different methods (with PINN assistance vs. without PINN) Note: Higher reward values ​​indicate better performance. Experimental data are derived from the actual training results of this embodiment, with 500 episodes × 100 steps trained and the average of 3 random seeds taken.

[0066] As shown in Table 1, the Industrial PINN auxiliary mechanism used in this embodiment significantly improves the MAPPO algorithm. After adding Industrial PINN assistance, the reward value of the MAPPO algorithm increased from 97.6 to 148.1, an improvement of 51.7%. Meanwhile, regarding safety constraints, none of the algorithms exhibited excessive temperature, stress, or vibration limits during training, demonstrating good safety performance.

[0067] The experimental results above show that this embodiment can effectively improve the performance of multi-agent reinforcement learning through the PINN-assisted reward shaping mechanism, and has application value in the field of microwave drilling control.

[0068] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made to the technical solutions of the present invention by those skilled in the art without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A microwave drilling control method based on physical information neural network-assisted multi-agent reinforcement learning, characterized in that, include: S1. Construct a physical simulation environment model for microwave drilling and obtain state observation data; wherein, the physical simulation environment model includes: thermal field model, stress field model, vibration model and efficiency model; S2. Input the state observation data into the multi-task physical information neural network model to obtain the predicted values ​​of physical quantities; S3. Based on the predicted values ​​of the physical quantities, construct the PINN-assisted reward shaping mechanism; S4. Construct a multi-agent reinforcement learning system and use the PINN assisted reward shaping mechanism to control the multi-agent reinforcement learning system to perform reinforcement learning. S5. Construct a safety constraint mechanism, integrate safety thresholds into the reinforcement learning reward function, punish excessive behavior, and guide the agent to avoid safety risks during the reinforcement learning training phase; wherein, the safety thresholds include preset temperature safety thresholds, preset stress safety thresholds, and preset vibration safety thresholds; S6. During the deployment phase, action selection without exploration noise is adopted. When the security prediction probability obtained based on PINN prediction and security factor is lower than the preset threshold, the security protection protocol is triggered and the system automatically switches to conservative control mode.

2. The microwave drilling control method based on physical information neural network assisted by claim 1, characterized in that, The thermal field model is as follows: ； The stress field model is as follows: ； The vibration model is as follows: ； The efficiency model is as follows: ； in, The value represents the drilling speed, and the subscript t is the time step index, representing the dynamic change of the corresponding physical quantity over time. t∈{0,1,2,…,N}, and t+1 is the next control step at time t. For microwave power, For cooling flow rate, For rotational speed, For temperature, For stress, For vibration, For efficiency, The Gaussian noise term for temperature prediction follows a mean of 0 and a standard deviation of . T follows a normal distribution. The Gaussian noise term for vibration prediction follows a mean of 0 and a standard deviation of . V follows a normal distribution. The Gaussian noise term for stress prediction follows a mean of 0 and a standard deviation of . It follows a normal distribution.

3. The microwave drilling control method based on physical information neural network assisted by multi-agent reinforcement learning according to claim 1, characterized in that, The multi-task physical information neural network model includes: Shared feature extraction backbone network: Each layer is followed by a batch normalization layer and a modified linear unit activation function to extract shared feature representations; Temperature prediction auxiliary head: It adopts a four-layer network structure to provide auxiliary and supervised temperature prediction; Final physical quantity output layer: Employs a fully connected layer to simultaneously output the predicted values ​​of the physical quantities; wherein, the predicted values ​​of the physical quantities include: temperature, stress, efficiency, safety factor, wear, specific absorptivity (SAR), and vibration.

4. The microwave drilling control method based on physical information neural network assisted by multi-agent reinforcement learning according to claim 1, characterized in that, The multi-task physical information neural network model also includes a physical constraint loss function, which is used to constrain the prediction to conform to physical laws. The physical constraint loss function includes constraints based on the heat conduction equation, Faraday's law, Ampere's law, momentum equation, continuity equation, and stress equation.

5. The microwave drilling control method based on physical information neural network assisted by claim 1, characterized in that, The PINN assisted reward shaping mechanism includes: The predicted values ​​of the physical quantities are incorporated into the reinforcement learning reward function, resulting in the following total reward function: ； in, For the total reward function, for , for , For PINN prediction consistency rewards, To restrain punishment, Weighting coefficients for predicting consistency rewards for PINN.

6. The microwave drilling control method based on physical information neural network assisted by multi-agent reinforcement learning according to claim 1, characterized in that, The multi-agent reinforcement learning system is constructed using the centralized training and distributed execution CTDE paradigm based on the MAPPO algorithm for multi-agent proximal policy optimization. The multi-agent reinforcement learning system includes: State space input module: used to output state vector; the state vector includes: drilling speed, rotation speed, microwave power, cooling flow rate, temperature, stress, vibration, efficiency, wear, and safety factor; the state vector serves as the global state input to each agent and the centralized Critic network; The Actor network is used to set up four parallel agents: a power agent, a cooling agent, a rotation speed agent, and a drilling speed agent, which are responsible for microwave power control, cooling system control, rotation speed control, and drilling speed control, respectively. Multi-head attention information exchange module: used to realize dynamic information exchange between intelligent agents. It adopts the Q, K, V mechanism, which enables each intelligent agent to adaptively pay attention to the information of other intelligent agents according to the current global state, so as to achieve coordinated decision-making. Hyper network Mixer module: used to dynamically generate adaptive mixing weights based on the global state, and to perform weighted fusion of the outputs of multiple agents; Centralized Critic Network: Used for value estimation and advantage calculation during the training phase; Motion execution module: used to map 4-dimensional continuous motion into physical control quantities and output duty cycle, cooling flow rate, rotation speed and drilling speed to the actuator.

7. The microwave drilling control method based on physical information neural network assisted by claim 6, characterized in that, Each agent contains a state encoder and a policy network Actor: the state encoder encodes local observations, and the policy network Actor outputs the agent's actions; the outputs of all policy network Actors together form a joint action, which is then input into a centralized Critic network.

8. The microwave drilling control method based on physical information neural network assisted by claim 1, characterized in that, Integrating security constraints on reinforcement learning outcomes includes: When the temperature exceeds the preset temperature safety threshold, a penalty is applied according to the set ratio; when the stress exceeds the preset stress safety threshold, a penalty is applied according to the set ratio; when the vibration exceeds the preset vibration safety threshold, a penalty is applied according to the set ratio.

9. The microwave drilling control method based on physical information neural network assisted by multi-agent reinforcement learning according to claim 1, characterized in that, The security prediction probability is calculated according to the following formula: ； in: These are the predicted values ​​for temperature, stress, and vibration, respectively. These are the safety thresholds corresponding to temperature, stress, and vibration, respectively. w 1. w 2. w 3 is the weighting coefficient.

Images