A robot thermal constraint safety model driven reinforcement learning method and system

Abstract

This application discloses a reinforcement learning method and system driven by a robot thermal constraint safety model, applicable to the field of robot thermal management technology. The method includes: constructing a thermophysical model object for the robot; acquiring the robot's operating condition data; combining the thermophysical model object and the operating condition data to determine the temperature estimates and uncertainty metrics for various parts of the robot, thereby determining the temperature safety boundaries for each part of the robot and the quantified value of the historical overheating cumulative effect. After obtaining the above system data, a safety model-driven reinforcement learning algorithm is used to determine the original control commands, which are then mapped to a feasible region that satisfies the temperature safety boundaries, dew point constraints, and preset physical limits, thus obtaining safety control commands. Finally, the robot is cooled based on these safety control commands. In this way, by introducing multiple constraints and finally using a safety model-driven reinforcement learning algorithm to determine the control commands, the accuracy and rationality of the system's cooling regulation are improved.

Description

Technical Field

[0001] This application relates to the field of robot thermal management technology, and in particular to a reinforcement learning method and system driven by a robot thermal constraint safety model. Background Technology

[0002] As humanoid robots continue to operate for longer periods in complex environments, their joints and drive systems face the challenge of a sudden surge in thermal load under conditions of high-dynamic motion, frequent posture switching, and multiple contact scenarios.

[0003] Existing thermal management technologies primarily rely on rule-based or proportional-integral-derivative (PI-DE) control cooling strategies, enhancing heat transfer capacity through structural improvements such as evaporation, spraying, microchannels, and fins. This approach typically depends on temperature thresholds to trigger simple on / off control, setting cooling intensity through empirical parameters. The signal path is generally "temperature sensor → controller → cooling execution unit." While this approach can achieve basic thermal protection under certain operating conditions, it suffers from limited system cooling regulation and unreasonable energy consumption distribution in complex scenarios involving high dynamics, strong disturbances, and multiple concurrent tasks.

[0004] Therefore, how to improve the accuracy and rationality of system cooling regulation is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] To address the aforementioned issues, this application provides a reinforcement learning method and system driven by a robot thermal constraint safety model. By introducing multiple constraints, the system finally utilizes a safety model-driven reinforcement learning algorithm to determine control commands, thereby improving the accuracy and rationality of the system's cooling regulation.

[0006] In a first aspect, embodiments of this application provide a reinforcement learning method driven by a robot thermal constraint safety model, including: Construct a thermophysical model object for the robot; Obtain the operating condition data of various parts of the robot; By combining the thermophysical model object and the working condition data, the estimated temperature values ​​and uncertainty measures of various parts of the robot are determined; Based on the operating data and the uncertainty measure, the temperature safety boundary of each part of the robot and the quantified value of the historical over-temperature cumulative effect are determined. Based on the operating condition data, the temperature estimate, the uncertainty measure, and the quantization value, the original control command is determined using a safety model-driven reinforcement learning algorithm. The original control command is mapped to a feasible domain that satisfies the temperature safety boundary, dew point constraint, and preset physical limit, and a safety control command corresponding to the original control command is obtained. The robot is cooled based on the safety control command.

[0007] Optionally, the object for constructing the robot's thermophysical model includes: Determine the materials used to compose each part of the robot; By combining a lumped parameter thermal network, a discretized model of the thermal state of each part of the robot is constructed based on the material properties of the constituent materials. The expression for the thermal state discretization model is: ; In the formula This is a predicted temperature value. This is the current temperature value. For the first matrix, For the second matrix, For the third matrix, The current power loss, For the current control action, To model the perturbation.

[0008] Optionally, the reinforcement learning method further includes: Obtain the phase current, mechanical angular velocity, and winding temperature-related resistance of various parts of the robot; By combining the motor and transmission loss model, the power loss of each part of the robot is determined based on the phase current, the mechanical angular velocity, and the winding temperature-related resistance.

[0009] Optionally, the motor and transmission loss model is as follows: ; In the formula To reduce power loss, For phase current, For winding temperature-dependent resistance, For mechanical angular velocity, The first calibration coefficient, This is the second calibration coefficient. This is the third calibration coefficient.

[0010] Optionally, determining the temperature estimates and uncertainty measures for various parts of the robot by combining the thermophysical model object and the operating condition data includes: Based on the thermophysical model object and the operating condition data, the predicted temperature values ​​of various parts of the robot are determined; By combining the prediction-update relationship expression in the integrated Kalman filter algorithm, the temperature estimates and uncertainty measures of various parts of the robot are determined using the operating data and the predicted temperature values.

[0011] Optionally, the prediction-update relationship expression is: ; In the formula Let k be the posterior state estimate at time k. Let k be the prior state estimate at time k. For Kalman gain, These are actual measured values. To measure the predicted value.

[0012] Optionally, the expression for the temperature safety boundary is: ; In the formula This is the temperature safety boundary for the i-th location. Let i be the maximum allowable temperature for the i-th part. For safety reasons, Let be the standard deviation of the temperature at the i-th location; The quantitative expression for the historical overheating cumulative effect is as follows: ; In the formula The accumulated thermal damage index from time 0 to k+1. The accumulated thermal damage index from time 0 to k. Let be the measured temperature of the i-th part at time k. Let i be the maximum allowable temperature for the i-th part. For time step.

[0013] Optionally, the step of determining the original control command using a safety model-driven reinforcement learning algorithm based on the operating condition data, the temperature estimate, the uncertainty measure, and the quantization value includes: Determine the lifespan risk of each part of the robot; Based on the operating condition data, the temperature estimate, the uncertainty measure, the quantification value, and the life risk, the original control command is determined using a strategy learning function; The immediate reward corresponding to the policy learning function is: ; In the formula For immediate returns, Let be the measured temperature of the i-th part at time k. Let be the reference temperature for the i-th part. The value represents the accumulated thermal damage index from time 0 to k. To the maximum capacity of the cooling system, For cooling energy consumption, For reference energy consumption, For life risk, As the first weight parameter, This is the second weighting parameter. As the third weighting parameter, This is the fourth weighting parameter.

[0014] Optionally, the dew point temperature corresponding to the dew point constraint is determined based on the Magnus formula; The expression for the Magnus formula is: ; In the formula For humidity-related intermediate variables, For ambient temperature, Let be the relative humidity, 'a' be the first coefficient, and 'b' be the second coefficient. This refers to the dew point temperature. The value of 'a' is 17.62, and the value of 'b' is 243.12℃. .

[0015] Secondly, embodiments of this application provide a reinforcement learning system driven by a robot thermal constraint safety model, the reinforcement learning system comprising: The thermal network modeling module is used to construct the thermal physics model object of the robot; A multi-source distributed sensing module is used to acquire working condition data of various parts of the robot; The state estimation and uncertainty fusion module is used to combine the thermophysical model object and the working condition data to determine the temperature estimate and uncertainty measure of each part of the robot. The safety boundary tightening and over-limit integration module is used to determine the temperature safety boundary of each part of the robot, as well as the quantified value of the historical over-temperature cumulative effect, based on the working condition data and the uncertainty metric. The reinforcement learning decision module is used to determine the original control command based on the operating condition data, the temperature estimate, the uncertainty measure, and the quantization value, using a safety model-driven reinforcement learning algorithm. The safety projection module is used to map the original control command to a feasible domain that satisfies the temperature safety boundary, dew point constraint and preset physical limit, and to obtain a safety control command corresponding to the original control command. A multi-channel cooling execution module is used to cool the robot based on the safety control command.

[0016] As can be seen from the above technical solutions, compared with the prior art, this application has the following advantages: This application provides a reinforcement learning method driven by a robot thermal constraint safety model. First, a thermophysical model of the robot is constructed, and operating condition data for various parts of the robot are acquired. Then, the temperature estimates and uncertainty measures for each part of the robot are determined by combining the thermophysical model and the operating condition data. Based on the operating condition data and uncertainty measures, the temperature safety boundaries for each part of the robot, as well as the quantified value of the historical overheating cumulative effect, are determined. Next, based on the operating condition data, temperature estimates, uncertainty measures, and quantified values, a safety model-driven reinforcement learning algorithm is used to determine the original control commands. These original control commands are then mapped to a feasible region that satisfies the temperature safety boundaries, dew point constraints, and preset physical limits, resulting in safety control commands corresponding to the original control commands. Finally, the robot is cooled based on the safety control commands. Thus, by introducing multiple constraints and finally using a safety model-driven reinforcement learning algorithm to determine the control commands, the accuracy and rationality of the system's cooling regulation are improved. Attached Figure Description

[0017] Figure 1 A flowchart illustrating a reinforcement learning method driven by a robot thermal constraint safety model, provided as an embodiment of this application; Figure 2 This is a schematic diagram of a reinforcement learning system driven by a robot thermal constraint safety model, provided as an embodiment of this application. Detailed Implementation

[0018] As mentioned earlier, existing thermal management technologies suffer from limited system cooling regulation and unreasonable energy consumption distribution. Specifically, existing thermal management technologies are mainly based on rule-based or proportional-integral-derivative (PI-DE) control cooling strategies, enhancing heat transfer capacity through structural improvements (such as evaporation, spraying, microchannels, fins, etc.). This method typically relies on temperature thresholds to trigger simple on / off control, setting cooling intensity through empirical parameters. The signal path is generally "temperature sensor → controller → cooling execution unit." While this can achieve basic thermal protection under certain operating conditions, it suffers from limited system cooling regulation and unreasonable energy consumption distribution in complex scenarios involving high dynamics, strong disturbances, and multiple concurrent tasks.

[0019] To address the aforementioned issues, this application provides a reinforcement learning method driven by a robot thermal constraint safety model, comprising: first, constructing a thermophysical model object of the robot and acquiring operating condition data for various parts of the robot; then, combining the thermophysical model object and the operating condition data to determine the temperature estimates and uncertainty measures for each part of the robot, and determining the temperature safety boundaries for each part of the robot, as well as the quantified value of the historical overheating cumulative effect, based on the operating condition data and the uncertainty measures; next, based on the operating condition data, temperature estimates, uncertainty measures, and quantified values, using a safety model-driven reinforcement learning algorithm to determine the original control commands, and mapping the original control commands to a feasible region that satisfies the temperature safety boundaries, dew point constraints, and preset physical limits, thereby obtaining safety control commands corresponding to the original control commands; finally, cooling the robot based on the safety control commands.

[0020] Thus, by introducing multiple constraints and finally using a safety model to drive a reinforcement learning algorithm to determine control commands, the accuracy and rationality of the system's cooling regulation are improved.

[0021] It should be noted that the reinforcement learning method and system driven by a robot thermal constraint safety model provided in this application can be applied to the field of robot thermal management technology. The above are merely examples and do not limit the application field of the reinforcement learning method and system driven by a robot thermal constraint safety model provided in this application.

[0022] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0023] Figure 1 This is a flowchart illustrating a reinforcement learning method driven by a robot thermal constraint safety model, provided as an embodiment of this application. (Combined with...) Figure 1 As shown in the embodiments of this application, a reinforcement learning method driven by a robot thermal constraint safety model may include: S101: Construct the robot's thermophysical model object.

[0024] In practical applications, the thermal conduction and convection coupling between the entire robot body (including windings, iron core, gears / bearings, housing, cooling medium and environment) is first modeled. The resulting thermophysical model object is used to provide physical priors for subsequent temperature prediction of the robot.

[0025] Furthermore, since there are different ways to construct thermophysical model objects, this application embodiment can describe one possible construction method.

[0026] In one case, the object for constructing the thermophysical model of the robot includes: Determine the materials used to compose each part of the robot; By combining a lumped parameter thermal network, a discretized model of the thermal state of each part of the robot is constructed based on the material properties of the constituent materials. The expression for the thermal state discretization model is: ; In the formula This is a predicted temperature value. This is the current temperature value. For the first matrix, For the second matrix, For the third matrix, The current power loss, For the current control action, To model the perturbation.

[0027] In practical applications, a discretized model of a lumped-parameter thermal network is introduced. First, the material data for each part of the robot is determined. Then, combined with parameters such as material properties, a thermophysical model object is constructed. This allows for the forward-looking prediction of how current control actions will affect future temperatures. Specifically, the thermophysical model object is represented as a discretized thermal state model, the expression of which is as follows: ; In the formula This refers to the predicted temperature (i.e., the future temperature). This is the current temperature value. Based on the heat capacity of the material Thermal conductivity matrix and sampling period The zero-order preserved discretization is obtained. Specifically, For the first matrix, For the second matrix, This is the third matrix. The current power loss, For the current control action, For modeling perturbations. Additionally, this application provides commonly used material thermal conductivity values: aluminum approximately 205 W / m·K, copper approximately 385 W / m·K, and steel approximately 16 W / m·K; convective heat transfer coefficient... Natural convection: 5–15 W / m²·K; forced air cooling: 20–120 W / m²·K; spray / boiling: 500–3000 W / m²·K.

[0028] Furthermore, since the methods for determining power loss are not entirely the same, this application embodiment can describe one possible method of determination.

[0029] In one instance, the reinforcement learning method further includes: Obtain the phase current, mechanical angular velocity, and winding temperature-related resistance of various parts of the robot; By combining the motor and transmission loss model, the power loss of each part of the robot is determined based on the phase current, the mechanical angular velocity, and the winding temperature-related resistance.

[0030] Furthermore, the motor and transmission loss model is as follows: ; In the formula To reduce power loss, For phase current, For winding temperature-dependent resistance, For mechanical angular velocity, The first calibration coefficient, This is the second calibration coefficient. This is the third calibration coefficient.

[0031] In practical applications, the discretized thermal state model considers current heating and mechanical losses. This embodiment introduces a motor and transmission loss model to provide accurate thermal load input for thermal prediction. Specifically, the motor and transmission loss model is as follows: ; In the formula To reduce power loss, For phase current, For winding temperature-dependent resistance, This refers to the mechanical angular velocity. , as well as Calibrated by steady-state and step experiments, generally in – These are on the order of magnitude and can be denoted as the first calibration coefficient, the second calibration coefficient, and the third calibration coefficient, respectively. The phase current and mechanical angular velocity can be acquired in real time by a multi-source distributed sensing module. Satisfy the following expression: ; In the formula , The resistance is 25°C. The winding temperature is also acquired in real time by a multi-source distributed sensing module.

[0032] S102: Obtain the working condition data of each part of the robot.

[0033] In practical applications, after obtaining the predicted state (predicted temperature value) based on the thermal state discretization model, it is necessary to achieve multi-sensor data fusion and system state estimation through integrated Kalman filtering. Before this, the operating condition data of various parts of the robot must first be acquired. Specifically, the multi-source distributed sensing module can collect data such as temperature, flow rate, valve position, pump speed, current, voltage, ambient temperature and humidity, and wind speed of various parts of the robot in real time. These data are then sent to a high-precision analog-to-digital converter (ADC) via an isolated front end. The digital signals are synchronously uploaded to the upper control platform via a fieldbus and used as input for the operating condition data of various parts of the robot.

[0034] S103: Combine the thermophysical model object and the working condition data to determine the temperature estimate and uncertainty measure of each part of the robot.

[0035] In practical applications, the thermophysical model object can make forward-looking state (temperature) predictions by combining the robot's working condition data. In order to improve the effective management of uncertainties such as environmental disturbances and sensor noise, this application embodiment introduces uncertainty estimation. By adopting an uncertainty estimation algorithm, the temperature prediction value is fused with multi-source observation values ​​(such as thermistors and infrared thermography) to determine the optimal temperature estimate (temperature estimate) and uncertainty quantification result (uncertainty measure) for each part of the robot.

[0036] Furthermore, since the methods for determining temperature estimates and uncertainty measures are not entirely the same, this application embodiment can describe one possible determination method.

[0037] In one scenario, determining the temperature estimates and uncertainty metrics for various parts of the robot by combining the thermophysical model object and the operating condition data includes: Based on the thermophysical model object and the operating condition data, the predicted temperature values ​​of various parts of the robot are determined; By combining the prediction-update relationship expression in the integrated Kalman filter algorithm, the temperature estimates and uncertainty measures of various parts of the robot are determined using the operating data and the predicted temperature values.

[0038] Furthermore, the prediction-update relationship expression is as follows: ; In the formula Let k be the posterior state estimate at time k. Let k be the prior state estimate at time k. For Kalman gain, These are actual measured values. To measure the predicted value.

[0039] In practical applications, a lumped-parameter thermal network is used as the physical prior model, and an integrated Kalman filter algorithm is employed to fuse multi-source sensor data to quantify uncertainty. Specifically, combining the real-time operating condition data collected by the multi-source distributed sensing module, the temperature prediction value obtained using the thermal state discretization model is used as the measurement prediction value and substituted into the prediction-update relationship expression in the integrated Kalman filter algorithm to determine the optimal temperature estimate (temperature estimate) and the estimation error covariance (uncertainty measure) for each part of the robot. The prediction-update relationship expression is as follows: ; In the formula Let k be the posterior state estimate at time k. Let k be the prior state estimate at time k. For Kalman gain, These are actual measured values. To measure the predicted value. Furthermore, ,in The cross-covariance matrix, Let R be the autocovariance matrix, where R is the observation noise covariance (typically 0.01–0.1°C² for patch thermal imaging and 0.1–1.0°C² for a single pixel in end infrared thermal imaging), and Q is the process noise, which reflects the operating condition disturbance and is typically taken as 10. - ³~10 - The normalized order of magnitude of ¹. Thus, the embodiments of this application introduce physical priors using lumped parameter thermal networks and uncertainty quantification using the Kalman filter algorithm, suppressing model prediction drift caused by multiple robot postures and multiple contacts, and making the strategy highly robust to changes in wind speed, environmental temperature and humidity through domain random training.

[0040] S104: Determine the temperature safety boundary of each part of the robot and the quantified value of the historical over-temperature cumulative effect based on the working condition data and the uncertainty measure.

[0041] Furthermore, the expression for the temperature safety boundary is: ; In the formula This is the temperature safety boundary for the i-th location. Let i be the maximum allowable temperature for the i-th part. For safety reasons, Let be the standard deviation of the temperature at the i-th location; The quantitative expression for the historical overheating cumulative effect is as follows: ; In the formula The accumulated thermal damage index from time 0 to k+1. The accumulated thermal damage index from time 0 to k. Let be the measured temperature of the i-th part at time k. Let i be the maximum allowable temperature for the i-th part. For time step.

[0042] In practical applications, this embodiment introduces temperature constraints and over-limit integral constraints. First, the upper limit of temperature safety is dynamically adjusted based on the uncertainty measurement results. Then, after obtaining stable and reliable temperature data, it is substituted into the over-limit integral to integrate the temperature amplitude and time exceeding the upper limit, thus preventing excessively high cumulative thermal damage indicators and potential damage to the robot. In this way, by dynamically tightening the upper limit of temperature and the integral constraints of thermal damage indicators online, thermal safety management is transformed from "reactive protection after the fact" to "proactive prediction and cumulative control during the event," ensuring that the temperature at critical nodes remains within a controlled range. Specifically, the expression for the temperature safety boundary, driven by uncertainty, is as follows: ; The estimated uncertainty is quantified into a controllable safety margin. This allows the system to automatically adopt a more conservative strategy when things are uncertain, thus achieving adaptive safety. In the formula, This is the temperature safety boundary for the i-th location. This represents the maximum allowable temperature for the i-th location. Generally, the winding... =155℃, casing Between 85℃ and 95℃. For safety margin, it is usually set to 2-3, corresponding to 95%-99% confidence coverage. Let be the temperature standard deviation of the i-th location output by the Kalman filter algorithm described above. Further, after the multi-source distributed sensing module obtains stable and reliable temperature data, it is substituted into the time-over-limit integral constraint (Γ) to integrate the over-temperature intensity and duration, thereby quantifying the cumulative effect of historical over-temperature. The quantified value of this historical over-temperature cumulative effect can be expressed as: ; In the formula The accumulated thermal damage index from time 0 to k+1. The accumulated thermal damage index from time 0 to k. Let be the measured temperature of the i-th part at time k. Let i be the maximum allowable temperature for the i-th part. Let Γ be the time step. Thus, by introducing the time-overlimit integral (Γ) and dynamically tightened safety boundaries as constraints, the cooling command is ultimately generated through a reinforcement learning algorithm driven by a safety model, improving the accuracy and rationality of subsequent system cooling adjustments.

[0043] S105: Based on the operating condition data, the temperature estimate, the uncertainty measure, and the quantization value, the original control command is determined using a safety model-driven reinforcement learning algorithm.

[0044] In practical applications, this application embodiment introduces a safety model-driven reinforcement learning algorithm, which takes the above-mentioned system data (operating condition data, temperature estimate, uncertainty measure and quantification value, wherein the operating condition data may specifically be historical temperature, environmental parameters and cooling channel status, etc.) as input, as well as gait and task summary. Finally, the model outputs a "hybrid action" that includes discrete cooling mode selection (such as "air cooling + liquid cooling") and continuous intensity setting (such as fan speed, pump speed, TEC current), and uses it as the original control command.

[0045] Furthermore, since the methods for determining the original control commands are not entirely the same, this application embodiment can describe one possible determination method.

[0046] In one scenario, determining the original control command using a safety model-driven reinforcement learning algorithm based on the operating condition data, the temperature estimate, the uncertainty measure, and the quantized value includes: Determine the lifespan risk of each part of the robot; Based on the operating condition data, the temperature estimate, the uncertainty measure, the quantification value, and the life risk, the original control command is determined using a strategy learning function; The immediate reward corresponding to the policy learning function is: ; In the formula For immediate returns, Let be the measured temperature of the i-th part at time k. Let be the reference temperature for the i-th part. The value represents the accumulated thermal damage index from time 0 to k. To the maximum capacity of the cooling system, For cooling energy consumption, For reference energy consumption, For life risk, As the first weight parameter, This is the second weighting parameter. As the third weighting parameter, This is the fourth weighting parameter.

[0047] In practical applications, this embodiment further introduces a policy learning function and a lifetime risk function for guidance and optimization of reinforcement learning. Specifically, the lifetime risk function can be approximated by Arrhenius, and its expression is as follows: ; In the formula, For life risk, This is the absolute temperature (measured temperature). For reference absolute temperature, This reflects the empirical rule that "lifespan is halved for every 10°C increase." Furthermore, measured values ​​can be replaced with estimated values, i.e., the measured temperature can be replaced with an estimated temperature. This is because, in many cases, physical sensors cannot be directly placed inside the hardware (such as inside battery cells or the core of a chip). Thus, using easily measurable parameters such as current, voltage, and load, the core temperature is "estimated" using corresponding algorithms. This "estimated value" is then substituted into the Arrhenius equation to calculate the "lifespan risk," and finally fed back to "policy learning" to form a closed-loop technology. Further, combined with the policy learning function, the system data obtained above (such as operating condition data, the estimated temperature, the uncertainty measure, the quantified value, and the lifespan risk, etc.) are input into the function, and the immediate reward of policy learning is determined. Specifically, the immediate reward corresponding to the policy learning function is as follows: ; In the formula, For immediate returns, Let be the measured temperature of the i-th part at time k. Let be the reference temperature for the i-th part. The value represents the accumulated thermal damage index from time 0 to k. The maximum capacity of the cooling system (set according to the life curve, typically 10²–10²). 4 °C·s), For cooling energy consumption, For reference energy consumption, For life risk, As the first weight parameter, This is the second weighting parameter. As the third weighting parameter, This is the fourth weighting parameter. Furthermore, The value of is between 0.01 and 10, inclusive. It can be understood that the policy learning function provided in this application embodiment is a multi-objective reward function, which simultaneously penalizes overheating, cumulative thermal shock, high energy consumption, and high lifespan risk. By optimizing this reward function, the reinforcement learning algorithm can automatically learn to find the optimal balance between ensuring safety (temperature and lifespan) and saving energy consumption.

[0048] S106: Map the original control command to a feasible domain that satisfies the temperature safety boundary, dew point constraint, and preset physical limit, and obtain a safety control command corresponding to the original control command.

[0049] Furthermore, the dew point temperature corresponding to the dew point constraint is determined based on the Magnus formula; The expression for the Magnus formula is: ; In the formula For humidity-related intermediate variables, For ambient temperature, Let be the relative humidity, 'a' be the first coefficient, and 'b' be the second coefficient. This refers to the dew point temperature. The value of 'a' is 17.62, and the value of 'b' is 243.12℃. .

[0050] In practical applications, the proposed embodiment introduces a safety layer. Before issuing control commands, a small-scale convex quadratic programming approach is used to project the feasible region to obtain a more optimized safety control command. Specifically, the original action corresponding to the original control command output by the strategy can be projected to the nearest feasible point that satisfies all constraints (including the tightened temperature safety boundary, the surface temperature must be higher than the dew point temperature, and the actuator command must be within its physical limits), thereby obtaining the safety control command for issuance to each actuator. The constraints corresponding to the small-scale convex quadratic programming are as follows: ; In the formula One-step prediction or first-order linearization from lumped parameter thermal networks get. The surface temperature required for spraying access. M represents the dew point temperature, with M set to 3℃-5℃. U represents the actuator command. This is the lower limit of the actuator specification. This represents the upper limit of the actuator specification. Furthermore, the dew point temperature can be obtained using the Magnus formula, which is expressed as follows: ; In the formula For humidity-related intermediate variables, For ambient temperature, Let be the relative humidity, 'a' be the first coefficient, and 'b' be the second coefficient. These are dew point temperatures. Value a is 17.62℃, and value b is 243.12℃. Additionally, the preset physical limits are the actual actuator boundaries, which can be given according to the actual hardware specifications. For example, pump / fan 0–100% command, proportional valve 0–100% opening, single nozzle 25–80 mL / min, thermoelectric cooler (TEC) 2–6A. Furthermore, slope limits can also be added. To avoid resonance and water hammer, this application embodiment employs a hybrid action strategy and a safety projection linkage mechanism to achieve coordinated scheduling of cooling modes and intensity. This allows cooling resources to be precisely allocated based on spatial hotspot distribution and temporal heat load rhythm, and uses dew point constraints to unify the trade-offs between energy consumption, response speed, and safety risks.

[0051] S107: Cool the robot based on the safety control command.

[0052] In practical applications, the projected safety control commands are sent to various actuators (such as drive air-cooled fans, liquid-cooled pumps, spray valves, TECs, etc.) via a real-time bus. The actuators then cool the robot based on these safety control commands. Furthermore, after the commands are sent, the actuator status is read back in real-time via the fieldbus for status estimation and control in the next cycle, forming a process of "perception—estimation—tightening—decision—safe execution—feedback".

[0053] Furthermore, if the robot cannot deploy spraying and TEC, the reinforcement learning system provided in this application embodiment can still operate under dual-channel air cooling and liquid cooling, with the policy network reducing the discrete mode dimension while retaining continuous amplitude scheduling. Similarly, if end-of-line infrared thermal imaging (IR) cannot be deployed, a multi-point contact thermal sensing and observation variance model can be used instead, still constructing temperature tightening boundaries. If computing power is limited, the projection of the safety layer can be simplified to piecewise linear limiting and priority arbitration, supplemented by event-triggered conservative scene set switching to maintain a safety closed loop on low-computing-power platforms. In addition, for applications requiring stronger task-thermal coupling description, a "learning-optimization integration" approach can be used to replace reinforcement learning strategies. For example, a temporal coding network can predict the cooling command sequence, and the commands can be constrained and projected through a differentiable optimization layer, thereby obtaining an executable solution that satisfies boundary and dew point constraints without introducing iterative solutions.

[0054] In summary, this application provides a reinforcement learning method driven by a robot thermal constraint safety model. First, a thermophysical model of the robot is constructed, and operating condition data for various parts of the robot are acquired. Then, the temperature estimates and uncertainty measures for each part of the robot are determined by combining the thermophysical model and the operating condition data. Based on the operating condition data and uncertainty measures, the temperature safety boundaries for each part of the robot, as well as the quantified value of the historical overheating cumulative effect, are determined. Next, based on the operating condition data, temperature estimates, uncertainty measures, and quantified values, a safety model-driven reinforcement learning algorithm is used to determine the original control commands. These original control commands are then mapped to a feasible region that satisfies the temperature safety boundaries, dew point constraints, and preset physical limits, resulting in safety control commands corresponding to the original control commands. Finally, the robot is cooled based on the safety control commands. Thus, by introducing multiple constraints and finally using a safety model-driven reinforcement learning algorithm to determine the control commands, the accuracy and rationality of the system's cooling regulation are improved.

[0055] Figure 2 This is a schematic diagram of a reinforcement learning system driven by a robot thermal constraint safety model, provided as an embodiment of this application. (Combined with...) Figure 2 As shown, the reinforcement learning system 200 driven by the robot's thermal constraint safety model includes: Thermal network modeling module 201 is used to construct the thermal physics model object of the robot; The multi-source distributed sensing module 202 is used to acquire working condition data of various parts of the robot; The state estimation and uncertainty fusion module 203 is used to combine the thermophysical model object and the working condition data to determine the temperature estimate and uncertainty measure of each part of the robot. The safety boundary tightening and over-limit integration module 204 is used to determine the temperature safety boundary of each part of the robot, as well as the quantified value of the historical over-temperature cumulative effect, based on the working condition data and the uncertainty measure. The reinforcement learning decision module 205 is used to determine the original control command based on the operating condition data, the temperature estimate, the uncertainty measure, and the quantization value, using a safety model-driven reinforcement learning algorithm. The safety projection module 206 is used to map the original control command to a feasible domain that satisfies the temperature safety boundary, dew point constraint and preset physical limit, and to obtain a safety control command corresponding to the original control command. The multi-channel cooling execution module 207 is used to cool the robot based on the safety control command.

[0056] Furthermore, in the reinforcement learning system driven by the robot thermal constraint safety model provided in this application embodiment, each node aligns its clock based on the Precision Time Protocol (PTP) to ensure the time consistency of sampling, estimation, decision-making and execution. The data flow and control commands between all modules achieve high-frequency, low-latency closed-loop control through priority queues and periodic scheduling mechanisms.

[0057] Furthermore, the expression for the temperature safety boundary is: ; In the formula This is the temperature safety boundary for the i-th location. Let i be the maximum allowable temperature for the i-th part. For safety reasons, Let be the standard deviation of the temperature at the i-th location; The quantitative expression for the historical overheating cumulative effect is as follows: ; In the formula The accumulated thermal damage index from time 0 to k+1. The accumulated thermal damage index from time 0 to k. Let be the measured temperature of the i-th part at time k. Let i be the maximum allowable temperature for the i-th part. For time step.

[0058] The dew point temperature corresponding to the dew point constraint is determined based on the Magnus formula; The expression for the Magnus formula is: ; In the formula For humidity-related intermediate variables, For ambient temperature, Let be the relative humidity, 'a' be the first coefficient, and 'b' be the second coefficient. This refers to the dew point temperature. The value of 'a' is 17.62, and the value of 'b' is 243.12℃. .

[0059] As one implementation method, regarding how to construct the robot's thermal physics model object, the aforementioned thermal network modeling module 201 is specifically used for: Determine the materials used to compose each part of the robot; By combining a lumped parameter thermal network, a discretized model of the thermal state of each part of the robot is constructed based on the material properties of the constituent materials. The expression for the thermal state discretization model is: ; In the formula This is a predicted temperature value. This is the current temperature value. For the first matrix, For the second matrix, For the third matrix, The current power loss, For the current control action, To model the perturbation.

[0060] As one implementation method, regarding how to determine the power loss of various parts of the robot, the aforementioned thermal network modeling module 201 is further used for: Obtain the phase current, mechanical angular velocity, and winding temperature-related resistance of various parts of the robot; By combining the motor and transmission loss model, the power loss of each part of the robot is determined based on the phase current, the mechanical angular velocity, and the winding temperature-related resistance.

[0061] Furthermore, the motor and transmission loss model is as follows: ; In the formula To reduce power loss, For phase current, For winding temperature-dependent resistance, For mechanical angular velocity, The first calibration coefficient, This is the second calibration coefficient. This is the third calibration coefficient.

[0062] As one implementation method, regarding how to determine the temperature estimates and uncertainty measures for various parts of the robot, the aforementioned state estimation and uncertainty fusion module 203 is specifically used for: Based on the thermophysical model object and the operating condition data, the predicted temperature values ​​of various parts of the robot are determined; By combining the prediction-update relationship expression in the integrated Kalman filter algorithm, the temperature estimates and uncertainty measures of various parts of the robot are determined using the operating data and the predicted temperature values.

[0063] Furthermore, the prediction-update relationship expression is as follows: ; In the formula Let k be the posterior state estimate at time k. Let k be the prior state estimate at time k. For Kalman gain, These are actual measured values. To measure the predicted value.

[0064] As one implementation method, regarding how to determine the original control command, the reinforcement learning decision module 205 is specifically used for: Determine the lifespan risk of each part of the robot; Based on the operating condition data, the temperature estimate, the uncertainty measure, the quantification value, and the life risk, the original control command is determined using a strategy learning function; The immediate reward corresponding to the policy learning function is: ; In the formula For immediate returns, Let be the measured temperature of the i-th part at time k. Let be the reference temperature for the i-th part. The value represents the accumulated thermal damage index from time 0 to k. To the maximum capacity of the cooling system, For cooling energy consumption, For reference energy consumption, For life risk, As the first weight parameter, This is the second weighting parameter. As the third weighting parameter, This is the fourth weighting parameter.

[0065] In summary, this application first constructs a thermophysical model of the robot and acquires operating condition data for various parts of the robot. Then, combining the thermophysical model and operating condition data, it determines the estimated temperature and uncertainty measure for each part of the robot. Based on the operating condition data and uncertainty measure, it determines the temperature safety boundary for each part of the robot and the quantified value of the historical overheating cumulative effect. Next, based on the operating condition data, temperature estimates, uncertainty measure, and quantified value, it uses a safety model-driven reinforcement learning algorithm to determine the original control commands. These original control commands are then mapped to a feasible region that satisfies the temperature safety boundary, dew point constraints, and preset physical limits, resulting in a safety control command corresponding to the original control command. Finally, the robot is cooled based on the safety control command. Thus, by introducing multiple constraints and finally using a safety model-driven reinforcement learning algorithm to determine the control commands, the accuracy and rationality of the system's cooling regulation are improved.

[0066] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A reinforcement learning method driven by a robot thermal constraint safety model, characterized in that, The reinforcement learning method includes: Construct a thermophysical model object for the robot; Obtain the operating condition data of various parts of the robot; By combining the thermophysical model object and the working condition data, the estimated temperature values ​​and uncertainty measures of various parts of the robot are determined; Based on the operating data and the uncertainty measure, the temperature safety boundary of each part of the robot and the quantified value of the historical over-temperature cumulative effect are determined. Based on the operating condition data, the temperature estimate, the uncertainty measure, and the quantization value, the original control command is determined using a safety model-driven reinforcement learning algorithm. The original control command is mapped to a feasible domain that satisfies the temperature safety boundary, dew point constraint, and preset physical limit, and a safety control command corresponding to the original control command is obtained. The robot is cooled based on the safety control command.

2. The reinforcement learning method according to claim 1, characterized in that, The thermophysical model object for constructing the robot includes: Determine the materials used to compose each part of the robot; By combining a lumped parameter thermal network, a discretized model of the thermal state of each part of the robot is constructed based on the material properties of the constituent materials. The expression for the thermal state discretization model is: ； In the formula This is a predicted temperature value. This is the current temperature value. For the first matrix, For the second matrix, For the third matrix, For the current power loss, For the current control action, To model the perturbation.

3. The reinforcement learning method according to claim 1, characterized in that, The reinforcement learning method further includes: Obtain the phase current, mechanical angular velocity, and winding temperature-related resistance of various parts of the robot; By combining the motor and transmission loss model, the power loss of each part of the robot is determined based on the phase current, the mechanical angular velocity, and the winding temperature-related resistance.

4. The reinforcement learning method according to claim 3, characterized in that, The motor and transmission loss model is as follows: ； In the formula To reduce power loss, For phase current, For winding temperature-dependent resistance, For mechanical angular velocity, The first calibration coefficient, This is the second calibration coefficient. This is the third calibration coefficient.

5. The reinforcement learning method according to claim 1, characterized in that, The process of determining the temperature estimates and uncertainty measures for various parts of the robot by combining the thermophysical model object and the operating condition data includes: Based on the thermophysical model object and the operating condition data, the predicted temperature values ​​of various parts of the robot are determined; By combining the prediction-update relationship expression in the integrated Kalman filter algorithm, the temperature estimates and uncertainty measures of various parts of the robot are determined using the operating data and the predicted temperature values.

6. The reinforcement learning method according to claim 5, characterized in that, The prediction-update relationship expression is as follows: ； In the formula Let k be the posterior state estimate at time k. Let k be the prior state estimate at time k. For Kalman gain, These are actual measured values. To measure the predicted value.

7. The reinforcement learning method according to claim 1, characterized in that, The expression for the temperature safety boundary is: ； In the formula This is the temperature safety boundary for the i-th location. The maximum allowable temperature for the i-th part is... For safety reasons, Let be the standard deviation of the temperature at the i-th location; The quantitative expression for the historical overheating cumulative effect is as follows: ； In the formula The accumulated thermal damage index from time 0 to k+1. The accumulated thermal damage index from time 0 to k. Let be the measured temperature of the i-th part at time k. The maximum allowable temperature for the i-th part is... For time step.

8. The reinforcement learning method according to claim 1, characterized in that, The process of determining the original control command using a safety model-driven reinforcement learning algorithm based on the operating condition data, the temperature estimate, the uncertainty measure, and the quantized value includes: Determine the lifespan risk of each part of the robot; Based on the operating condition data, the temperature estimate, the uncertainty measure, the quantification value, and the life risk, the original control command is determined using a strategy learning function; The immediate reward corresponding to the policy learning function is: ； In the formula For immediate returns, Let be the measured temperature of the i-th part at time k. Let be the reference temperature for the i-th part. The value represents the accumulated thermal damage index from time 0 to k. To the maximum capacity of the cooling system, For cooling energy consumption, For reference energy consumption, For life risk, As the first weight parameter, This is the second weighting parameter. As the third weighting parameter, This is the fourth weighting parameter.

9. The reinforcement learning method according to claim 1, characterized in that, The dew point temperature corresponding to the dew point constraint is determined based on the Magnus formula; The expression for the Magnus formula is: ； In the formula For humidity-related intermediate variables, For ambient temperature, Let be the relative humidity, 'a' be the first coefficient, and 'b' be the second coefficient. This refers to the dew point temperature. The value of 'a' is 17.62, and the value of 'b' is 243.12℃. .

10. A reinforcement learning system driven by a robot thermal constraint safety model, characterized in that, The reinforcement learning system includes: The thermal network modeling module is used to construct the thermal physics model object of the robot; A multi-source distributed sensing module is used to acquire working condition data of various parts of the robot; The state estimation and uncertainty fusion module is used to combine the thermophysical model object and the working condition data to determine the temperature estimate and uncertainty measure of each part of the robot. The safety boundary tightening and over-limit integration module is used to determine the temperature safety boundary of each part of the robot, as well as the quantified value of the historical over-temperature cumulative effect, based on the working condition data and the uncertainty metric. The reinforcement learning decision module is used to determine the original control command based on the operating condition data, the temperature estimate, the uncertainty measure, and the quantization value, using a safety model-driven reinforcement learning algorithm. The safety projection module is used to map the original control command to a feasible domain that satisfies the temperature safety boundary, dew point constraint and preset physical limit, and to obtain a safety control command corresponding to the original control command. A multi-channel cooling execution module is used to cool the robot based on the safety control command.

Images