A reactor automatic startup method based on deep reinforcement learning
The reactor automatic startup algorithm was designed through deep reinforcement learning, and UDP communication was used to realize online data interaction between the reactor simulation model and the automatic startup algorithm, which solved the problem of the complex and time-consuming reactor startup process and improved the level of automation and safety.
Patent Information
- Application Number
- CN202310034820.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-01-10
- Publication Date
- 2025-09-30
- Estimated Expiration
- 2043-01-10
AI Technical Summary
The existing reactor startup process is complex and time-consuming, which is prone to human-induced accidents. Existing sequential control technology makes it difficult to effectively achieve automated control.
Deep reinforcement learning is used to design the reactor automatic startup algorithm. Online data interaction between the reactor simulation model and the automatic startup algorithm is realized through UDP communication, and the algorithm parameters are trained to achieve automatic control.
The automation level of the reactor startup process is improved, the workload of the unit operators is relieved, the probability of misoperation is reduced, and the safety of the startup process is improved.
Smart Images
Figure CN116153545B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of reactor startup control, and in particular to a reactor automatic startup method based on deep reinforcement learning. Background Art
[0002] The reactor startup process involves safely bringing the reactor from subcriticality to criticality and establishing a stable chain fission reaction. This process is primarily performed manually by operators. Compared to full-power operation, the startup process involves numerous systems, is complex, and takes a long time. This makes it prone to human-induced accidents, which pose a serious threat to reactor safety. The use of automated reactor startup technology is extremely important in reducing the operator's workload, lowering the probability of operational errors, and improving the safety of the reactor startup process.
[0003] Currently, research on automated reactor startup almost exclusively utilizes sequential control technology. Sequential control is the most fundamental control method in the field of automatic control. It is used to transform complex manual operations into automated, sequential operations according to a specific logic. Its advantage lies in adhering to the procedures and logic of manual operations, thereby improving the automation level of the process. However, sequential control technology may be insufficient in the field of reactor startup because a reactor is a complex nonlinear model, and translating the numerous operational tasks involved in the startup process into accurate sequential logic is difficult. Summary of the Invention
[0004] The purpose of the present invention is to provide a reactor automatic startup method based on deep reinforcement learning. The reactor automatic control algorithm is designed using the deep reinforcement learning principle, UDP communication is used as the communication medium between the control algorithm and the reactor real-time simulation model to realize data interaction between the two, and the algorithm parameters are trained through online training to realize automatic control of the reactor startup process.
[0005] To achieve the above object, the present invention provides the following solutions:
[0006] A method for automatically starting a reactor based on deep reinforcement learning, comprising:
[0007] Design an automatic stack startup algorithm based on deep learning principles;
[0008] Establish a joint simulation platform;
[0009] Through the joint simulation platform, online data interaction between the reactor simulation model and the automatic start-up algorithm is achieved, the automatic start-up algorithm is trained, and the automatic start-up is performed based on the trained automatic start-up algorithm.
[0010] Preferably, designing the automatic stack start algorithm includes:
[0011] The state space and action space of the automatic startup algorithm are determined according to the reactor type, a reward function is designed according to the existing startup operation regulations of the reactor, and the reactor safety state variables are selected to complete the design of the automatic startup algorithm.
[0012] Preferably, the reactor safety state variables include: the rod position of each group of control rods, the reactor cycle, thermal parameters and the reactor coolant inlet and outlet temperatures; wherein, during the startup phase, for a reactor type in which the primary and secondary circuit thermal parameters remain unchanged, the safety state variables do not select the thermal parameters.
[0013] Preferably, the reward function is:
[0014]
[0015] Among them, R is the total reward for reaching the next state after the automatic start-up algorithm takes action, r1 is the reward for reaching the expected critical rod position; r2 is the reward obtained by the change of the reactor cycle; r3 is the reward obtained by the automatic start-up algorithm taking action, T is the reactor cycle, b = cycle warning value / (cycle limit value - cycle warning value).
[0016] Preferably, establishing the joint simulation platform includes:
[0017] Develop UDP communication of the reactor model and UDP communication of the automatic startup algorithm, connect the UDP communication of the reactor model to the reactor simulation model, select the algorithm framework according to the action space of the reactor, and perform UDP communication packet loss data processing to complete the establishment of the joint simulation platform.
[0018] Preferably, the processing of the UDP communication packet loss data includes: filtering the packet loss data by adding a command.
[0019] Preferably, realizing online data interaction between the reactor simulation model and the automatic startup algorithm includes:
[0020] The time interval for each data interaction of the UDP communication, the port and IP address for sending and receiving data of the UDP communication are set, and it is determined that the automatic start-up algorithm and the reactor real-time simulation model directly interact with data.
[0021] Preferably, training the automatic stack start algorithm includes:
[0022] The reactor real-time simulation model and the automatic start-up algorithm are run, and the automatic start-up algorithm is trained by data interaction between the two. After each training session is completed, the algorithm parameters are saved. When the next training session starts, the saved parameters are directly called instead of randomly generated parameters for training.
[0023] Preferably, during the training of the automatic startup algorithm, if the automatic startup algorithm is controlled to perform a certain action that causes certain safety limits of the reactor to be lower than the specified value, the automatic startup algorithm will punish the action to ensure that the algorithm learns the safety strategy of the startup process.
[0024] The beneficial effects of the present invention are:
[0025] The present invention applies currently advanced artificial intelligence algorithms to the reactor startup stage, which can improve the automation level of the reactor startup process; converts manual operations into automatic operations, liberating the unit operators to a certain extent, allowing the unit operators to devote more energy to the status monitoring of the reactor system, thereby improving the safety of the startup process; in the present invention, deep reinforcement learning can gradually learn the optimal control strategy for the entire operation process through continuous interaction with the reactor simulation model, determine the reasonable control rod lifting height and waiting time, and improve the efficiency of the startup operation. BRIEF DESCRIPTION OF THE DRAWINGS
[0026] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0027] Figure 1 This is a flow chart of a method for automatically starting a reactor based on deep reinforcement learning according to an embodiment of the present invention;
[0028] Figure 2 Schematic diagram of the automatic reactor startup process based on deep reinforcement learning in an embodiment of the present invention. DETAILED DESCRIPTION
[0029] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.
[0030] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.
[0031] (1) Reactor state space selection: For different reactor types, it is necessary to select variables that can represent the core safety state during the reactor startup process as the state space. The selected data are mainly some physical and thermal parameters, such as control rod positions, neutron counts outside the reactor, coolant temperature, etc.
[0032] (2) Action space selection: The choice of action space depends on the rod-lifting action space of the reactor model. If the rod-lifting action of the reactor model is continuous, then the algorithm action space selection must be continuous. If the rod-lifting action space of the reactor model is discrete, then the algorithm action space must also be discrete.
[0033] (3) Reward function design: To achieve the goal of automatic reactor startup, a reasonable reward function needs to be designed. The reward function should encourage the control rod to move upward, and punish actions that may cause accidents.
[0034] (4) Joint simulation platform, using UDP network communication as the data exchange medium, and embedding UDP communication into the reactor real-time simulation model and the automatic start-up model development joint experimental platform to achieve online data interaction between the two;
[0035] (5) Algorithm training: Repeated automatic start-up experiments are carried out on the joint experimental platform to complete the training of the automatic start-up model.
[0036] like Figure 1-2 The present invention provides a method for automatically starting a reactor based on deep reinforcement learning, which specifically includes:
[0037] (1) The design process of the automatic start-up model. The automatic start-up model needs to be designed according to the specific reactor type. This is because different types of reactors have different start-up operation strategies, and the control rod groups and control rod action spaces arranged inside the core are also different. Therefore, when designing the control algorithm, some major variables will change with different reactor types. When designing the automatic start-up model, pay attention to the following points:
[0038] a. When designing the automatic startup model, try to follow the existing reactor startup operation strategy;
[0039] b. In selecting the reactor state space, variables that accurately represent the reactor's safe state must be selected. These variables include physical thermal parameters, such as the positions of each set of control rods, reactor cycle, and reactor coolant inlet and outlet temperatures. For reactors where the primary and secondary circuit thermal parameters remain unchanged during startup, thermal parameters do not need to be selected as state variables.
[0040] c. Reward function design. In deep reinforcement learning, the reward function plays an extremely important role because it guides the intelligent agent to learn the optimal strategy. If the reward is a sparse reward, it will affect the convergence speed of the algorithm. On the issue of automatic reactor startup, it is necessary to ensure the absolute safety of the reactor startup process, and each safety parameter is controlled within the prescribed range. If the safety limit specified in the startup regulations is exceeded, the action needs to be punished. At the same time, the reward function also needs to guide the control rod to rise upward to avoid excessive stays at positions where the safety margin of the core is very high. In summary, the present invention recommends the piecewise reward function shown in equation (1);
[0041]
[0042] Among them, R is the total reward for reaching the next state after the automatic startup algorithm takes action, r1 is the reward for reaching the expected critical rod position; r2 is the reward obtained by the change of the reactor cycle. If the reactor cycle is outside the limit specified in the startup regulations, a negative reward is given; r3 is the reward obtained by the automatic startup algorithm for taking action. If the rod lifting action is performed, a positive reward is obtained, and if the control rod is stopped, the reward is 0. The purpose of setting r3 is to guide the control rod to rise upward and avoid the control rod from staying too long in the area with a large safety margin in the core; T is the reactor cycle, b = cycle warning value / (cycle limit value - cycle warning value). Setting a and b is to make the reward r2 change linearly within the range from the cycle limit value to the cycle warning value.
[0043] d. Establish a joint simulation platform. In order to realize the online data interaction between the automatic start-up model and the reactor simulation model, the present invention uses UDP communication as the data interaction medium to develop a joint simulation platform between the automatic start-up model and the reactor simulation model. When establishing the simulation platform, two sections of UDP communication code need to be developed separately, namely: UDP communication of the reactor model and UDP communication of the automatic start-up model. The UDP communication of the reactor model needs to be connected to the reactor simulation model, and the UDP communication of the automatic start-up model can be directly written into the algorithm. It should be noted that if a parallel computing function is added to the UDP communication when developing the joint simulation platform, the present invention recommends using the A3C or A2C framework in deep reinforcement learning to design the automatic start-up model. If the UDP communication does not have a parallel computing function, then the algorithm framework is selected according to the action space of the reactor. If it is a discrete action space, the DDQN framework is recommended. If it is a continuous action space, the DDPG framework is recommended.
[0044] e. Handling packet loss in UDP communication. If the UDP receiving end takes too long to process or the sent data packets are too large, this can result in packet loss. Therefore, when developing a co-simulation platform, additional packet loss preprocessing is required in UDP communication. Experiment with this without packet loss processing to determine what data types are generated after packet loss. Then, add a simple if command to filter out this lost data.
[0045] (2) Algorithm training process. After the automatic start-up model is designed, it is necessary to train the automatic start-up model through online data interaction with the reactor simulation model in the joint experimental platform. When training the algorithm, the following points should be noted:
[0046] a. Determine the data exchange interval. Before training begins, first set the UDP communication exchange interval. To ensure real-time data exchange, this interval should be short. If the data exchange interval is too short, the reactor simulation model will perform multiple actions under the same conditions, potentially introducing excessive reactivity at once when approaching critical rod positions. This invention recommends a data exchange interval between 0.5 and 1.0 seconds.
[0047] b. Data transmission and reception. When the reactor real-time simulation model sends data to the automatic control algorithm, it generates a 1×n array. The first few data points in the array store parameters that represent the reactor's safe state. The remaining data points representing the reactor's state are all set to 0. This array is encoded and sent to the automatic control algorithm. When the automatic control algorithm receives the data, it only needs to decode the data representing the reactor's safety. After receiving the reactor state data, the automatic control algorithm calculates the action that the reactor should take under that state. The automatic control algorithm then sends the action to the reactor simulation model using the same method as the reactor simulation model to send data. This method achieves real-time data exchange. To speed up training, all reactor state data should preferably be normalized first.
[0048] c. Determining the Initial and Final States. Before training begins, the initial and final states for each training session must be determined. Since this invention targets the automatic startup of a reactor from a hot shutdown state to a critical state, the hot shutdown state is selected as the initial state, and the final state is either a reactor emergency shutdown or the control rods reaching the predicted critical position.
[0049] d. Training of the automatic start-up model. During algorithm training, the reactor simulation model and the automatic control algorithm need to be started at the same time. Algorithm training is a long process. Therefore, after each round of training, the automatic control algorithm needs to save the current algorithm parameters to facilitate direct call in the next round of training. However, the initial parameters of deep reinforcement learning are randomly generated, so when developing the automatic start-up model, two program files should be developed. The main algorithm framework of the two program files is the same. The only difference is that the algorithm parameters of the first program file are randomly generated, and the algorithm parameters of the second program file are called from the parameters trained in the previous round. The first program file is only used in the first round of training, and the second program file is used for all subsequent training.
[0050] e. Verify the effectiveness of automatic startup. After the algorithm has been trained a certain number of times, if the rewards gradually stabilize and the reactor is able to automatically start, remove the ε-greedy strategy and training phase and run an automatic startup experiment using only the trained results. If the automatic startup is successful, training is complete. If not, re-run the experiment with the ε-greedy strategy.
[0051] UDP network communication is used as a data exchange medium to realize online data interaction between the reactor simulation model and the automatic startup model. The automatic startup model is trained by online data interaction with the real-time reactor simulation model. The details are as follows:
[0052] a. Determine the state space and action space of the automatic startup model based on the reactor type;
[0053] b. Design a reward function based on the reactor's existing startup regulations;
[0054] c. Select an appropriate deep reinforcement learning framework based on the type of action space (discrete action space or continuous action space) of the real-time reactor simulation model;
[0055] d. Develop a UDP communication program. This communication program consists of two parts: the UDP communication program for the automatic control algorithm and the UDP communication program for the reactor real-time simulation model. The UDP communication programs for the automatic control algorithm and the reactor real-time simulation model must be developed using their respective development languages. The UDP communication program for the reactor real-time simulation model must be incorporated into the reactor simulation model, while the UDP communication for the automatic startup model can be written directly into the automatic startup model.
[0056] e. Set the time interval for each data exchange of UDP communication, the ports and IP addresses for sending and receiving data of the two UDP communications, and ensure that the automatic start-up model and the reactor real-time simulation model directly exchange data, and the set time can meet the requirements of online training;
[0057] f. Run the real-time reactor simulation model and the automatic startup model, training the algorithm through data exchange between them. After each training session, save the algorithm parameters. When the next training session begins, directly use the saved parameters instead of randomly generating them.
[0058] After the automatic startup model is trained, it can automatically determine the control rod lift height and dwell time each time, without the need for manual settings in advance. The details are as follows:
[0059] During the algorithm training phase, variables representing the reactor's safety status are normalized and fed into the automatic startup model during each data interaction interval to train the model. After receiving the reactor status data, the automatic startup model takes action and sends this action to the reactor real-time simulation model.
[0060] b. Through repeated training, the automatic startup model will gradually learn the optimal control strategy for the reactor startup phase, and understand under which conditions the control rods should be raised and under which conditions the control rods should be kept.
[0061] c. The automatic startup model does not directly output the dwell time after each control rod lift. Instead, it indirectly determines the dwell time by determining the number of consecutive dwell maneuvers within a certain time period. Assuming the interval between data exchanges is Δt and the control rod has made n consecutive dwell maneuvers, the dwell time after the last control rod lift is nΔt.
[0062] d. If the reactor simulation model's rod raising motion space is continuous, the control rod lifting height can be directly determined using the automatic control algorithm. If the motion space is discrete, the control rod lifting height is also determined by determining the dwell time.
[0063] Applying deep reinforcement learning algorithms to reactor startup improves the automation level of nuclear power plants and frees up operator personnel. Specific features include:
[0064] a. Use deep reinforcement learning in artificial intelligence to design an automatic control algorithm to convert manual operations during the reactor startup and criticality process into automatic operations;
[0065] b. After training, deep reinforcement learning can make correct control actions based on the reactor status. Compared with sequential control, it does not need to convert the operations during the startup process into accurate sequential logic, which improves the automation level of the reactor to a certain extent.
[0066] c. The startup operation process does not require manual control by the crew operators, which reduces the operating pressure of the crew operators;
[0067] d. The crew operators can focus more on monitoring the reactor status, thus improving the safety of the operation process.
[0068] The main advantages of the present invention are:
[0069] (1) Applying current advanced artificial intelligence algorithms to the reactor startup stage can improve the automation level of the reactor startup process;
[0070] (2) Converting manual operation to automatic operation frees up the crew operators to a certain extent, allowing them to focus more on monitoring the status of the reactor system and improving the safety of the startup process;
[0071] (3) Through continuous interaction with the reactor simulation model, deep reinforcement learning can gradually learn the optimal control strategy for the entire operation process, determine the reasonable control rod lifting height and waiting time, and improve the efficiency of the startup operation.
[0072] The embodiments described above are merely descriptions of preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by persons skilled in the art should fall within the scope of protection defined by the claims of the present invention.
Claims
1. A method for automatic reactor startup based on deep reinforcement learning, characterized in that: include: Design an automatic stack startup algorithm based on deep learning principles; Establish a joint simulation platform; Through the joint simulation platform, online data interaction between the reactor simulation model and the automatic start-up algorithm is realized, the automatic start-up algorithm is trained, and the automatic start-up is performed based on the trained automatic start-up algorithm; Designing the automatic stack start algorithm includes: Determining the state space and action space of the automatic startup algorithm according to the reactor type, designing a reward function according to the existing startup operation regulations of the reactor, selecting the reactor safety state variables, and completing the design of the automatic startup algorithm; The reactor safety state variables include: the rod positions of each group of control rods, the reactor cycle, thermal parameters, and the reactor coolant inlet and outlet temperatures; wherein, during the startup phase, for a reactor type with unchanged primary and secondary circuit thermal parameters, the safety state variables do not select the thermal parameters; The reward function is: Among them, R is the total reward for reaching the next state after the automatic start-up algorithm takes action, r1 is the reward for reaching the expected critical rod position; r2 is the reward obtained by the change of the reactor cycle; r3 is the reward obtained by the automatic start-up algorithm taking action, T is the reactor cycle, b = cycle warning value / (cycle limit value - cycle warning value); Establishing the joint simulation platform includes: Develop UDP communication of the reactor model and UDP communication of the automatic startup algorithm, connect the UDP communication of the reactor model to the reactor simulation model, select the algorithm framework according to the action space of the reactor, and perform UDP communication packet loss data processing to complete the establishment of the joint simulation platform.
2. The reactor automatic startup method based on deep reinforcement learning according to claim 1, characterized in that: The UDP communication packet loss data processing includes: filtering the packet loss data by adding a command.
3. The reactor automatic startup method based on deep reinforcement learning according to claim 1, characterized in that: Implementing online data interaction between the reactor simulation model and the automatic startup algorithm includes: The time interval for each data interaction of the UDP communication, the port and IP address for sending and receiving data of the UDP communication are set, and it is determined that the automatic start-up algorithm and the reactor real-time simulation model directly interact with data.
4. The reactor automatic startup method based on deep reinforcement learning according to claim 3, characterized in that: Training the automatic stack start algorithm includes: The reactor real-time simulation model and the automatic start-up algorithm are run, and the automatic start-up algorithm is trained by data interaction between the two. After each training session is completed, the algorithm parameters are saved. When the next training session starts, the saved parameters are directly called instead of randomly generated parameters for training.
5. The method for automatic reactor startup based on deep reinforcement learning according to claim 4, characterized in that: During the training of the automatic startup algorithm, if the automatic startup algorithm is controlled to perform a certain action that causes certain safety limits of the reactor to fall below the specified value, the automatic startup algorithm will penalize the action to ensure that the algorithm learns the safety strategy of the startup process.