Nuclear power simulation system parameter intelligent debugging method based on reinforcement learning

Abstract

The invention belongs to the technical field of simulation system parameter intelligent debugging, aims to solve the problems that a traditional optimization method based on engineering experience is slow in convergence, prone to falling into local optimum and poor in dynamic environment adaptability, and discloses a nuclear power simulation system parameter intelligent debugging method based on reinforcement learning. An MDP used for describing the parameter optimization process of the nuclear power simulation system is constructed, the state space of the MDP comprises parameter vectors subjected to physical correlation grouping and normalization processing, a current system loss value and a loss change trend, an action decision is output through a strategy network, collaborative optimization is carried out through a double-value network and a maximum entropy regularization mechanism, exploration and utilization are balanced, and the optimization efficiency of the nuclear power simulation system is improved. Training the intelligent agent by using historical simulation data or real-time interaction data, and deploying the intelligent agent for automatic parameter debugging and optimization of the nuclear power simulation system after training is completed. According to the method, efficient autonomous optimization of simulation system parameters is realized, and the defects of a traditional heuristic algorithm in the aspects of convergence, stability and adaptive capacity can be remarkably relieved.

Description

Technical Field

[0001] This application belongs to the field of intelligent parameter debugging technology for simulation systems, and in particular relates to an intelligent parameter debugging method for nuclear power simulation systems based on reinforcement learning. Background Technology

[0002] Nuclear power simulation systems are core tools for nuclear power engineering design, safety analysis, and operational verification. The accuracy and reliability of these simulations directly affect the safety, economy, and operational efficiency of nuclear power plants. To achieve high-fidelity simulations, precise adjustments are required to a large number of key parameters in the system. These parameters often exhibit high dimensionality, strong coupling, and nonlinearity; their optimal combination is crucial to ensuring that the simulation system closely approximates the real physical processes.

[0003] Existing parameter tuning methods heavily rely on the experience and knowledge of domain experts and widely employ traditional optimization algorithms, such as heuristic search strategies like Differential Evolution (DE) and Particle Swarm Optimization (PSO). While these methods perform reasonably well on low-dimensional, static problems, their inherent limitations significantly restrict tuning efficiency and effectiveness when dealing with complex, high-dimensional dynamic systems like nuclear power plant simulations.

[0004] First, the performance of traditional heuristic algorithms is highly dependent on their control parameters (such as the mutation factor F and crossover rate CR of DE, and the inertia weight and learning factor of PSO). Inappropriate parameter settings can easily cause the optimization process to prematurely converge to a local optimum, making it difficult to obtain the global optimum. To achieve feasible results, a large amount of manual trial and error and parameter tuning is required, resulting in low automation and insufficient versatility. In addition, these algorithms have low global exploration efficiency and slow convergence speed in high-dimensional parameter spaces. When dealing with the nearly 100-dimensional parameter scale of nuclear power simulation, they often require a very large population and a large number of iterations, resulting in huge computational overhead and making it difficult to meet the debugging efficiency requirements of actual engineering.

[0005] Secondly, traditional methods lack the ability to cope with dynamic environments. The operating conditions and simulation objectives of nuclear power systems often have significant time-varying characteristics, while existing algorithms lack online learning and adaptive mechanisms. Once the operating conditions or objectives change, the optimization process needs to be re-initialized, and historical commissioning experience cannot be inherited and utilized, making it difficult to apply to dynamic and ever-changing engineering environments. Summary of the Invention

[0006] The purpose of this application is to provide an intelligent parameter adjustment method for nuclear power simulation systems based on reinforcement learning, which solves the problems of slow convergence, easy getting trapped in local optima, and poor adaptability to dynamic environments in traditional optimization methods based on engineering experience.

[0007] To achieve the above objectives, this application provides the following technical solution:

[0008] A method for intelligent parameter tuning of a nuclear power simulation system based on reinforcement learning, comprising:

[0009] S1: Construct an MDP to describe the parameter optimization process of a nuclear power simulation system. Its state space includes parameter vectors that have been grouped and normalized by physical correlation, the current system loss value and the trend of loss change. The action space is defined as the continuous adjustment amount of each group of parameters. The reward function integrates trend reward and loss reward to guide the optimization direction.

[0010] S2: The SAC algorithm is used as the agent. The action decision is output through the policy network and optimized in a coordinated manner through the dual value network and the maximum entropy regularization mechanism to balance exploration and exploitation.

[0011] S3: Train the agent using historical simulation data or real-time interactive data, and deploy it for automated parameter debugging and optimization of the nuclear power simulation system after training.

[0012] As one feasible approach, in S1, the state space is:

[0013]

[0014]

[0015] In the formula, G t L is the parameter vector grouped and normalized according to physical correlation. t This represents the overall loss value output by the simulation system at the current moment. This is a loss trend indicator, with values ​​{-1, 0, +1}.

[0016] As one feasible approach, in S1, the action space is:

[0017] a t =[Δx1,Δx2,…,Δx 53 ] ∈[-1,1] 53

[0018] In the formula, Δx i For the adjustments made by the agent to the i-th parameter, and through Convert the action value into an actual parameter adjustment amount.

[0019] As an feasible approach, in S1, the reward function is:

[0020] r t =β1·r trend +β2·r loss

[0021] β1+β2=1

[0022] In the formula, rtrend Based on the trend of loss change Trend reward, r loss Based on the current loss value L t The loss reward is β1, which is the weight of the trend reward, and β2 is the weight of the loss reward.

[0023] As an implementable approach, trend reward r trend for:

[0024]

[0025] In the formula, This is a loss trend indicator, with values ​​{-1, 0, +1}.

[0026] As an feasible approach, loss reward r loss for:

[0027]

[0028] In the formula, L max With L min For historical loss extremes, L t This represents the current simulation system loss value.

[0029] As an feasible approach, in S2, the SAC agent uses two independent dual-value networks to alleviate the Q-value overestimation problem, and employs a target value network and a soft update mechanism to stabilize the training process.

[0030] As an feasible approach, in S2, the SAC agent optimizes the policy by maximizing the weighted sum of expected reward and policy entropy, as shown in the following formula:

[0031]

[0032] In the formula, α is the automatically adjusted entropy regularization coefficient, and s t As the initial state, a t For the action selected based on the current strategy, r(s) t ,a t ) indicates that in state s t Take action a t The reward obtained, H(π(·|s) t )) represents the policy π in state s t Entropy below.

[0033] As an feasible approach, S3 employs vectorized environment technology to run multiple environment instances in parallel; an experience replay pool is set up to store interaction data, the capacity and sampling batch size of which are adjusted accordingly based on the number of parallel environments and the data generation speed; the network update uses the Adam optimizer, and the learning rate and soft update coefficient are set; the action range of the agent output is constrained within a reasonable range by a scaling factor; and the weights of trend reward and loss reward in the reward function are configured as needed.

[0034] As an feasible approach, in S3, the formula for the Loss index of the simulation model evaluating parameter optimization is:

[0035]

[0036] In the formula, N is the dimension of the parameter, M represents different initial parameters, and x j Let j be the value of the j-th parameter in the parameter vector. ω is the j-th parameter value in the target parameter vector output by the simulation model. j This represents the weight of the parameter.

[0037] Using different initial parameters, we simulated a scenario with varying environmental conditions in the parameter optimization of a nuclear power system. We statistically analyzed the average loss and average time results from multiple optimizations. Variance was used to calculate the instability of two factors, as follows:

[0038]

[0039] In the formula, loss i The loss obtained by optimization under the i-th set of initial parameters. The average value of the loss obtained by optimizing each parameter, where M represents different initial parameters;

[0040] The variance of time consumption is calculated as follows:

[0041]

[0042] In the formula, time i This represents the time consumption under the i-th set of initial parameters. This represents the average time consumed for optimizing each parameter, where M represents different initial parameters.

[0043] Compared with existing technologies, the intelligent parameter adjustment method for nuclear power simulation systems based on reinforcement learning provided in this application has the following advantages:

[0044] This application transforms the parameter tuning problem into an MDP problem and employs the SAC algorithm to achieve efficient and stable search in a high-dimensional continuous action space. The agent can autonomously learn optimization strategies, is insensitive to changes in control parameters, and significantly reduces its dependence on external experience. Its online learning capability enables it to quickly adapt to dynamically changing simulation environments, improving the automation level of the tuning process and its engineering application value.

[0045] This method not only effectively reduces the reliance of traditional methods on control parameters and expert experience, achieving efficient and autonomous optimization of simulation system parameters, but also significantly alleviates the shortcomings of traditional heuristic algorithms in terms of convergence, stability, and adaptability. This has significant engineering application value for improving the confidence level of nuclear power simulation, enhancing the reliability of design verification, shortening the commissioning cycle, and reducing development and operation costs.

[0046] This method enables rapid, stable, and automated debugging of numerous parameters in nuclear power simulation systems, thereby optimizing system performance, improving the consistency between the simulation model and real physical processes, and providing crucial technical support for the safety and economy of nuclear power systems. This has profound significance for promoting the development of digital design, intelligent operation and maintenance, and advanced reactor technologies in nuclear power.

[0047] Furthermore, this method also has significant reference value and promising prospects for parameter optimization in other complex industrial simulation systems, such as chemical process simulation, aerospace vehicle control, and energy system scheduling. Attached Figure Description

[0048] To more clearly illustrate the technical solution of this application, the accompanying drawings used in the technical description will be briefly introduced below.

[0049] Figure 1 A flowchart illustrating the intelligent parameter adjustment method for a nuclear power simulation system based on reinforcement learning, provided in an embodiment of this application.

[0050] Figure 2 This is a flowchart of the SAC agent training process provided in an embodiment of this application.

[0051] Figure 3 A schematic diagram illustrating the processing steps for intelligent parameter debugging of a nuclear power simulation system provided in this application embodiment;

[0052] Figure 4 A model architecture diagram for intelligent parameter debugging of a nuclear power simulation system provided in this application embodiment. Detailed Implementation

[0053] The following detailed description provides further details on specific implementation methods.

[0054] Example 1

[0055] like Figures 1 to 4 As shown, this embodiment provides an intelligent parameter adjustment method for a nuclear power simulation system based on reinforcement learning, including:

[0056] S1: Construct a Markov decision process (MDP) to describe the parameter optimization process of a nuclear power simulation system. Its state space includes parameter vectors that have been grouped and normalized by physical correlation, the current system loss value and the trend of loss change. The action space is defined as the continuous adjustment of each group of parameters. The reward function integrates trend reward and loss reward to guide the optimization direction.

[0057] S2: The agent is constructed using the Soft Actor-Critic (SAC) algorithm. The SAC algorithm includes a policy network (Actor), a dual-value network (Critic), and a target value network. The policy network outputs continuous actions that interact with the nuclear power simulation environment. The dual-value network independently estimates the state-action value and takes the minimum value in the target calculation to suppress value overestimation. The target value network periodically synchronizes the main network parameters through a soft update mechanism to stabilize the training process. At the same time, an entropy regularization mechanism and a learnable temperature coefficient are introduced to adjust the policy exploration intensity, thereby optimizing the cumulative reward and policy diversity. The SAC algorithm achieves an adaptive balance between exploration and exploitation through the above mechanisms and ensures the stability of the training process.

[0058] S3: Train the agent using historical simulation data or real-time interactive data, and deploy it for automated parameter debugging and optimization of the nuclear power simulation system after training.

[0059] The intelligent parameter tuning method for nuclear power simulation systems based on reinforcement learning provided in this embodiment is mainly used for automated optimization and dynamic tuning of high-dimensional complex parameters. This method includes three core mechanisms: establishing a mapping relationship between the tuning problem and reinforcement learning by constructing a Markov Decision Process (MDP); achieving a balance between exploration and exploitation using the maximum entropy soft actor-critic (SAC) algorithm; and designing a composite reward mechanism to simultaneously guide short-term optimization trends and long-term stability.

[0060] Specifically, the state space is first defined, including parameter vectors grouped and normalized by physical correlation, the current system loss value, and the trend of loss change; the action space is defined as the continuous adjustment amount for each parameter group; and the reward function is the weighted sum of trend reward and loss reward. Subsequently, using historical simulation data or real-time interactive data, the SAC agent is trained in the constructed MDP environment. During training, mechanisms such as experience replay, dual-value network (Critic network), target value network, and automatic entropy adjustment are employed to improve stability and convergence efficiency. After training, the trained agent is connected to the nuclear power simulation system to read the system state in real time and output parameter adjustment actions, achieving closed-loop automatic optimization. Test results under multiple sets of different initial parameters demonstrate that the reinforcement learning-based intelligent parameter tuning method for nuclear power simulation systems can effectively solve the problems of slow convergence, poor stability, and insufficient environmental adaptability of traditional methods in high-dimensional parameter optimization.

[0061] Specifically, S1 includes: based on the optimization objective of the nuclear power simulation system, constructing a Markov decision process model to describe the parameter adjustment process. Its state space includes a multi-dimensional parameter vector grouped and normalized to a unified dimension after physical correlation analysis, the comprehensive loss value output by the simulation system at the current moment, and a discrete indicator representing the trend of loss change; the action space is defined as a continuous multi-dimensional vector, where each dimension corresponds to the adjustment range of a specific parameter group; the reward function is a hybrid reward signal, which is composed of a trend reward term and a loss reward term weighted linearly by weight coefficients, used to guide the agent to learn in the direction of optimization.

[0062] In this embodiment, based on the ultimate goal of optimizing the parameters of the nuclear power simulation system, a Markov Decision Process (MDP) environment is constructed to describe this process. The ultimate goal is to minimize the comprehensive deviation between the simulation model output and the expected target, and its objective function is defined as:

[0063]

[0064] In the formula, N is the dimension of the parameter to be optimized; x i The value of the i-th parameter predicted by the agent; ω represents the target value of the i-th parameter output by the simulation model. i This represents the weight assigned to this parameter.

[0065] At the same time, the safety constraints for parameter adjustment must be met:

[0066]

[0067] In the formula, Δx i This represents the adjustment made by the agent to the i-th parameter.

[0068] Specifically, the three core elements of a Markov decision process (MDP) include the state space, the action space, and the reward function.

[0069] The state space is constructed as a composite vector, which can be formalized using the following formula:

[0070]

[0071] In the formula, G t ∈R 53 This process, which involves grouping and normalizing multidimensional parameter vectors to a uniform dimension after physical correlation analysis, significantly reduces the complexity and training difficulty of the original high-dimensional parameter space; L t This is the comprehensive loss value output by the simulation system at the current moment, used to quantitatively evaluate the current performance of the system; It is a discrete indicator that characterizes the trend of loss change. It processes the difference between the loss values ​​at adjacent time points through a sign function to indicate whether the performance is in an improving, deteriorating or stable state.

[0072] The action space is defined as a continuous multidimensional vector, where each dimension corresponds to the adjustment magnitude for a specific set of parameters, and is formalized by the following formula:

[0073] a t =[Δx1,Δx2,...,Δx 53 ]∈[-1,1] 53

[0074] The action value a t The output needs to be mapped to the actual parameter adjustment amount by a linear scaling factor β. To ensure that the agent's exploratory behavior is constrained within engineering safety and reasonable limits, it is formalized using the following formula:

[0075]

[0076] Secondly, the parameters need to be continuously adjusted to optimize the optimal solution and avoid getting trapped in local optima, as shown below:

[0077]

[0078] In the formula, G t The parameter vector is grouped and normalized according to physical relevance, and the action value is... This represents the actual parameter adjustment amount.

[0079] The reward function is designed as a hybrid reward signal, consisting of a trend reward term and a loss reward term, weighted linearly summed with weighted coefficients. The reward function design is shown below:

[0080] r t=β1·r trend +β2·r loss β1+β2=1

[0081] In the formula, r trend Based on the trend of loss change Trend reward, r loss Based on the current loss value L t The loss reward, β1 and β2 represent the weights of the trend reward and loss reward, respectively. Generally, for r loss Such unstable reward values ​​should be given smaller weight values.

[0082] Based on state s t The trend of loss change in the value provides a trend reward r trend Its definition is as follows:

[0083]

[0084] In the formula, This is a loss trend indicator, with values ​​{-1, 0, +1}.

[0085] r loss The loss reward is used to quantify the optimization effect of the agent's adjustments at the current moment; the current simulation model loss value L is... t The smaller the value, the greater the reward. After normalization, the current loss is mapped to the [-1, 1] interval, defined as follows:

[0086]

[0087] In the formula, L max With L min The historical loss extremes are set as the loss L0 of the initial parameters and the minimum loss value 0 of the simulation system in this scenario, respectively. t This represents the current simulation system loss value.

[0088] Specifically, the parameter intelligent debugging system uses the aforementioned Markov decision process as its interaction framework and the soft actor-critic (SAC) algorithm as its core driver. Its architecture includes a policy network (Actor network), a dual-value network (Critic network), a target value network, and an entropy regularization adaptive module.

[0089] The policy network receives system state input and outputs continuous adjustment actions for each parameter group through random policy functions. The action generation adopts reparameterization technology to introduce controllable randomness to enhance the exploration capability.

[0090] The dual-value network is deployed in parallel, and the state-action value is estimated independently. The smaller Q value is selected as the optimization target in the value update, which effectively suppresses the overestimation of value.

[0091] The target value network periodically synchronizes the main network parameters through a soft update mechanism, i.e., according to the formula. Progressive updates, where τ is the soft update coefficient, provide a stable target signal for training.

[0092] The entropy regularization adaptive module explores intensity by dynamically adjusting the strategy through a learnable temperature coefficient, thereby maximizing cumulative reward and strategy diversity simultaneously during the optimization process.

[0093] In this embodiment, the agent continuously interacts with the simulation environment to collect state transition samples and stores them in an experience replay pool. During training, based on randomly sampled batch data, it sequentially updates the two Q-value functions, policy function, and entropy coefficients in the value network (Critic network), ultimately achieving efficient learning and stable convergence of the parameter optimization policy in a high-dimensional continuous space. This system effectively balances the contradiction between exploration and exploitation through a maximum entropy reinforcement learning framework, significantly improving the convergence speed and robustness of parameter tuning in complex simulation environments.

[0094] In this embodiment, the core architecture of the intelligent parameter tuning strategy is built upon the soft actor-critic (SAC) algorithm, and its system structure is as follows: Figure 4 As shown, the interaction between the intelligent agent and the simulation environment, as well as the internal learning mechanism, are fully demonstrated. This architecture includes core components such as a nuclear power simulation environment, a policy network (Actor network), a dual-value network (Critic network), a target value network, and an experience replay pool.

[0095] First, the intelligent agent continuously interacts with the nuclear power simulation environment: the environment will transmit the system state s t (Including grouped normalized parameter vectors, comprehensive loss value, and trend indicator) Input agent; policy network (Actor network) generates parameters and adjusts action a based on state information. t This feedback is then sent to the environment for execution; the environment subsequently returns the reward r after execution. t With the new state s t+1 This forms a complete state transition sample (s,a,r,s').

[0096] Subsequently, these interaction samples are stored in the experience replay pool. During training, the agent updates three core components sequentially based on randomly sampled batch data: the Q-value function of the dual-value network (Critic network) is updated by minimizing the Bellman error; the parameters of the policy network (Actor network) are updated by maximizing the expected reward and entropy; and the entropy regularization coefficient is adaptively adjusted to balance the intensity of exploration and exploitation.

[0097] Finally, a soft update mechanism is employed to synchronize the target value network parameters, ensuring the stability of the training process. Through this gradual optimization approach, efficient learning and stable convergence of the parameter optimization strategy in a high-dimensional continuous space are achieved, forming a complete closed-loop optimization system from environmental state perception to parameter adjustment decisions, and then to experience learning and model updates. This architecture, through a maximum entropy reinforcement learning framework, effectively addresses the challenges faced by nuclear power simulation systems in high-dimensional parameter space debugging, such as slow convergence speed and susceptibility to local optima, significantly improving the automation level and optimization efficiency of parameter debugging.

[0098] Specifically, in S2, the intelligent parameter adjustment strategy for the nuclear power simulation system adopts the soft actor-critic (SAC) algorithm based on the maximum entropy reinforcement learning framework as the core intelligent agent architecture. This intelligent agent consists of five deep neural network modules: one policy network (Actor network), two independent value networks (Critic networks), and two corresponding target value networks. The network modules achieve collaborative optimization through a carefully designed interaction mechanism.

[0099] The policy network employs a stochastic policy model based on reparameterization techniques. Its network structure is a three-layer structure with a hidden layer containing 256 neurons. After receiving the system state input, the network outputs the mean and variance parameters of a Gaussian distribution. The final action output is obtained by sampling using reparameterization techniques, enabling randomized exploration of parameter adjustments. This design ensures both exploration efficiency and the smoothness and continuity of the action output.

[0100] The value network employs a dual-value network independent evaluation architecture. Both networks have the same three-layer fully connected structure but are initialized with different parameters. Each value network independently evaluates the value function of the state-action pair and takes the smaller of the two values ​​when calculating the target value. This design effectively suppresses the problem of overestimation of the Q-value and improves the algorithm's stability. The target value network periodically synchronizes the main network parameters through a soft update mechanism, according to the formula... The parameters are updated, where τ is the soft update coefficient, which is usually set to 0.005. This design ensures the stability of the training objective.

[0101] The interaction between the agent and the simulation environment is based on a carefully designed Markov Decision Process (MDP). The state space is defined as a composite vector. In the formula, G t The 53-dimensional parameter vector, after physical correlation grouping and normalization, is obtained by physically grouping and min-max normalizing the original 94-dimensional parameters; L t The comprehensive loss value of the simulation system at the current moment is calculated using the objective function, as shown in the following formula:

[0102]

[0103] As an indicator of loss trend, the sign function sign(L) is used. t -L t-1 The value is calculated and takes the form {-1, 0, +1}, representing a decrease in loss, no change, and an increase in loss, respectively.

[0104] The action space is defined as a 53-dimensional continuous vector a t ∈[-1, 1] 53 Each dimension represents the normalization adjustment magnitude for the corresponding parameter set. This is achieved through a linear mapping. The action value is converted into the actual parameter adjustment amount, where β is the scaling factor, set to 0.001 to ensure the safety of parameter adjustment.

[0105] The reward function uses a composite design, and the formula is as follows:

[0106] r t =β1·r trend +β2·r loss

[0107] In the formula, β1 and β2 are weighting coefficients, set to 0.75 and 0.25 respectively, and r trend Based on the trend of loss change Directional rewards, r loss Based on the current loss value L t Quantitative rewards.

[0108] Trend Rewards trend Directional incentives are provided based on the trend of loss changes, as shown in the following formula:

[0109]

[0110] In the formula, This is a loss trend indicator, with values ​​{-1, 0, +1}.

[0111] Loss reward r loss This is used to quantify the optimization effect of the agent's adjustments at the current moment, and after normalization, maps the current loss to the interval [-1, 1]. The formula is as follows:

[0112]

[0113] In the formula, L max With L min For historical loss extremes, L t This represents the current simulation system loss value.

[0114] In this embodiment, the agent is optimized using a maximum entropy reinforcement learning framework, whose objective function simultaneously maximizes the cumulative reward and the policy entropy value, mathematically expressed as:

[0115]

[0116] In the formula, α is the automatically adjusted entropy regularization coefficient, and s t As the initial state, a t For the action selected based on the current strategy, r(s) t ,a t ) indicates that in state s t Take action a t The reward obtained, H(π(·|s) t )) indicates that policy F is in state s t Entropy below.

[0117] The following key technologies were employed during training: First, vectorized environment technology was used to run 16 environment instances in parallel, significantly improving data acquisition efficiency; second, an experience replay pool with a capacity of 4M was set to store interaction data, and 4096 experiences were randomly sampled as a batch each time an update was performed; then, the Adam optimizer was used to update the network parameters, with the learning rate set to 5e-3; finally, the temperature parameter α was dynamically adjusted through an automatic entropy adjustment mechanism to balance the intensity of exploration and exploitation.

[0118] In this embodiment, the intelligent parameter tuning system constructs an automated optimization system adapted to high-dimensional complex parameter spaces by introducing an intelligent decision-making mechanism based on a maximum entropy reinforcement learning framework. The system uses Markov Decision Process (MDP) to formally model the parameter tuning problem, and through a carefully designed state space representation, it compresses the 94-dimensional original parameters into 53-dimensional feature vectors based on physical relevance, significantly reducing the complexity of the learning task. Simultaneously, it innovatively designs a composite reward function that integrates trend rewards and loss rewards, focusing on both immediate optimization effects and long-term performance improvement, providing multi-timescale optimization guidance for the agent. The intelligent parameter tuning method for nuclear power simulation systems uses the Soft Actor-Critic (SAC) algorithm as the core optimization engine. Through the collaborative optimization of the policy network and the dual-value network, combined with maximum entropy regularization and adaptive exploration mechanisms, it effectively solves the exploration-exploitation trade-off problem in high-dimensional parameter spaces. This architecture supports efficient exploration by the agent in continuous parameter spaces, while ensuring the stability of the learning process through mechanisms such as experience replay and soft updates of the target value network. This method significantly improves upon traditional optimization algorithms in terms of convergence speed, optimization accuracy, and environmental adaptability, effectively addressing the parameter tuning requirements under multiple operating conditions and objectives in nuclear power simulation systems. This system provides an efficient, reliable, and highly generalizable intelligent solution for solving high-dimensional parameter optimization problems in complex industrial simulation environments, suitable for adaptive parameter tuning tasks in various nuclear power simulation scenarios.

[0119] Specifically, in S3, the agent is trained using historical simulation data or real-time interactive data. After training, it is deployed for automated parameter tuning and optimization in the nuclear power simulation system. In the implementation process, vectorized environment technology is used to run multiple environment instances in parallel to accelerate data acquisition; an experience replay pool is set up to store interactive data, with its capacity and sampling batch size adjusted according to the number of parallel environments and the data generation speed; the network update uses the Adam optimizer, with appropriate learning rate and soft update coefficients set; the action amplitude of the agent's output is constrained within a reasonable range by a scaling factor to ensure the safety of parameter adjustment; the weights of trend reward and loss reward in the reward function are configured as needed to effectively balance optimization direction and steady-state performance. After the agent has been sufficiently trained and stabilized, the strategy is converted into deterministic output, which can then be integrated into the nuclear power simulation system to achieve online, automatic closed-loop optimization of parameters. The specific training flowchart is as follows... Figure 2 As shown.

[0120] A trained intelligent agent is deployed in a nuclear power simulation system to achieve automated parameter tuning and optimization. The processing steps for intelligent parameter tuning in a nuclear power simulation system are as follows: Figure 3 As shown.

[0121] Specifically, the formula for the Loss index of the simulation model for evaluating parameter optimization is as follows:

[0122]

[0123] In the formula, N is the dimension of the parameter, M represents different initial parameters, and x j Let j be the value of the j-th parameter in the parameter vector. ω is the j-th parameter value in the target parameter vector output by the simulation model. j This represents the weight assigned to this parameter.

[0124] Different initial parameters were used to simulate a variable environment in nuclear power system parameter optimization, and the average loss and average time results of multiple optimizations were statistically analyzed. Furthermore, variance was used to calculate the instability of both factors, aiming to obtain a more stable optimization algorithm. The formula is shown below:

[0125]

[0126] This formula calculates the variance between the loss obtained under each set of initial parameters and the mean of the loss samples, where loss i The loss is the result of optimization under the i-th set of initial parameters. The average loss is calculated for each optimized parameter, where M represents different initial parameters. Similarly, the variance of time consumption is calculated using the following formula:

[0127]

[0128] In the formula, time i This represents the time consumption under the i-th set of initial parameters. This represents the average time consumed for optimizing each parameter, where M represents different initial parameters.

[0129] In summary, this embodiment employs a reinforcement learning-based intelligent parameter tuning strategy for nuclear power simulation systems. It utilizes Markov decision processes to model high-dimensional parameter optimization problems, capturing the coupling relationships between parameters and the dynamic characteristics of the system. A state space encompassing grouped normalized parameter vectors, real-time loss values, and loss change trends comprehensively represents the system state. Adjustment suggestions for parameter groups are output via a continuous action space, and a reward function integrating trend and absolute performance indicators guides the optimization direction, thereby achieving comprehensive and robust autonomous parameter search and optimization. Furthermore, this strategy leverages the soft actor-critic (SAC) algorithm framework, enhances exploration capabilities through a maximum entropy learning mechanism, and mitigates the overestimation problem of the value function through a dual-value network structure and a target value network, improving the stability and convergence efficiency of strategy training. The proposed intelligent tuning mechanism can stably achieve efficient exploration in high-dimensional parameter spaces, online adaptive optimization in dynamic environments, and safe tuning under multi-objective constraints. It significantly improves the efficiency and robustness of parameter tuning in nuclear power simulation systems, making it suitable for parameter optimization needs under various simulation verification and operational conditions, particularly for complex engineering tuning scenarios characterized by high dimensionality, strong nonlinearity, multiple constraints, and dynamic time-varying characteristics.

[0130] This method not only effectively reduces the reliance of traditional methods on control parameters and expert experience, achieving efficient and autonomous optimization of simulation system parameters, but also significantly alleviates the shortcomings of traditional heuristic algorithms in terms of convergence, stability, and adaptability. This has significant engineering application value for improving the confidence level of nuclear power simulation, enhancing the reliability of design verification, shortening the commissioning cycle, and reducing development and operation costs.

[0131] This method enables rapid, stable, and automated debugging of numerous parameters in nuclear power simulation systems, thereby optimizing system performance, improving the consistency between the simulation model and real physical processes, and providing crucial technical support for the safety and economy of nuclear power systems. This has profound significance for promoting the development of digital design, intelligent operation and maintenance, and advanced reactor technologies in nuclear power.

[0132] Furthermore, this method also has significant reference value and promising prospects for parameter optimization in other complex industrial simulation systems, such as chemical process simulation, aerospace vehicle control, and energy system scheduling.

[0133] Example 2

[0134] This embodiment provides an intelligent debugging device for nuclear power simulation system parameters based on reinforcement learning, including a memory and a processor. The memory stores a computer program, and when the computer program is executed by the processor, it implements the various steps of the intelligent debugging method for nuclear power simulation system parameters based on reinforcement learning in Embodiment 1.

[0135] The device provided in this embodiment is used to implement a method for intelligent debugging of nuclear power simulation system parameters based on reinforcement learning. Therefore, this device also fully possesses the technical effects of the intelligent debugging method for nuclear power simulation system parameters based on reinforcement learning, such as high-dimensional parameter adaptive optimization, dynamic environment robust debugging, and high convergence stability. This device can implement the technical solutions described in the above method embodiments. The specific implementation principle of this device can be found in the corresponding content in the above method embodiments, which will not be repeated here.

[0136] The above description is only a specific embodiment of this application, but the protection scope of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the protection scope of this application.

Claims

1. A method for intelligent parameter adjustment of a nuclear power simulation system based on reinforcement learning, characterized in that, include: S1: Construct an MDP to describe the parameter optimization process of a nuclear power simulation system. Its state space includes parameter vectors that have been grouped and normalized by physical correlation, the current system loss value and the trend of loss change. The action space is defined as the continuous adjustment amount of each group of parameters. The reward function integrates trend reward and loss reward to guide the optimization direction. S2: The SAC algorithm is used as the agent. The action decision is output through the policy network and optimized in a coordinated manner through the dual value network and the maximum entropy regularization mechanism to balance exploration and exploitation. S3: Train the agent using historical simulation data or real-time interactive data, and deploy it for automated parameter debugging and optimization of the nuclear power simulation system after training.

2. The intelligent parameter adjustment method for nuclear power simulation system based on reinforcement learning according to claim 1, characterized in that, In S1, the state space is: In the formula, G t L is the parameter vector grouped and normalized according to physical correlation. t This represents the overall loss value output by the simulation system at the current moment. This is a loss trend indicator, with values ​​{-1, 0, +1}.

3. The intelligent parameter adjustment method for nuclear power simulation system based on reinforcement learning according to claim 1, characterized in that, In S1, the action space is: a t =[Δx1, Δx2,..., Δx 53 ]∈[-1,1] 53 In the formula, Δx i For the adjustments made by the agent to the i-th parameter, and through Convert the action value into an actual parameter adjustment amount.

4. The intelligent parameter adjustment method for nuclear power simulation system based on reinforcement learning according to claim 1, characterized in that, In S1, the reward function is: r t ＝β1·r trend +β2·r loss β1+β2＝1 In the formula, r trend Based on the trend of loss change Trend reward, r loss Based on the current loss value L t The loss reward is β1, which is the weight of the trend reward, and β2 is the weight of the loss reward.

5. The intelligent parameter adjustment method for nuclear power simulation system based on reinforcement learning according to claim 4, characterized in that, Trend Rewards trend for: In the formula, This is a loss trend indicator, with values ​​{-1, 0, +1}.

6. The intelligent parameter adjustment method for nuclear power simulation system based on reinforcement learning according to claim 4, characterized in that, Loss reward r loss for: In the formula, L max With L min For historical loss extremes, L t This represents the loss value at the current moment.

7. The intelligent parameter adjustment method for nuclear power simulation system based on reinforcement learning according to claim 1, characterized in that, In S2, the SAC agent uses two independent dual-value networks and employs a target value network and a soft update mechanism to stabilize the training process.

8. The intelligent parameter adjustment method for nuclear power simulation system based on reinforcement learning according to claim 1, characterized in that, In S2, the SAC agent optimizes its policy by maximizing the weighted sum of expected reward and policy entropy, as shown in the following formula: In the formula, α is the automatically adjusted entropy regularization coefficient, and s t As the initial state, a t For the action selected based on the current strategy, r(s) t ,a t ) indicates that in state s t Take action a t The reward obtained, H(π(·|s) t )) represents the policy π in state s t Entropy below.

9. The intelligent parameter adjustment method for nuclear power simulation system based on reinforcement learning according to claim 1, characterized in that, In S3, vectorized environment technology is used to run multiple environment instances in parallel; an experience replay pool is set up to store interaction data, and its capacity and sampling batch size are adjusted accordingly based on the number of parallel environments and the data generation speed; the network update uses the Adam optimizer, and the learning rate and soft update coefficient are set; the action range of the agent output is constrained within a reasonable range by a scaling factor; the weights of trend reward and loss reward in the reward function are configured as needed.

10. The intelligent parameter adjustment method for nuclear power simulation system based on reinforcement learning according to claim 1, characterized in that, In S3, the formula for the Loss index of the simulation model used to evaluate parameter optimization is: In the formula, N is the dimension of the parameter, M represents different initial parameters, and x j Let j be the value of the j-th parameter in the parameter vector. ω is the j-th parameter value in the target parameter vector output by the simulation model. j This represents the weight of the parameter. Using different initial parameters, we simulated a scenario with varying environmental conditions in the parameter optimization of a nuclear power system. We statistically analyzed the average loss and average time results from multiple optimizations. Variance was used to calculate the instability of two factors, as follows: In the formula, loss i The loss obtained by optimization under the i-th set of initial parameters. The average value of the loss obtained by optimizing each parameter, where M represents different initial parameters; The variance of time consumption is calculated as follows: In the formula, time i This represents the time consumption under the i-th set of initial parameters. This represents the average time consumed for optimizing each parameter, where M represents different initial parameters.

Images