Method and apparatus for generating fuel management schemes based on deep reinforcement learning

Abstract

This invention discloses a method and apparatus for generating fuel management schemes based on deep reinforcement learning. The method includes: for any fuel management scheme, using nuclear design software to solve for the safety and economic parameters of the fuel management scheme; establishing reward and penalty criteria for evaluating the fuel management scheme; modifying the fuel management scheme based on weights using a deep reinforcement learning network to obtain a new fuel management scheme; determining the reward and penalty values ​​of the original fuel management scheme and the new fuel management scheme, and comparing the two reward and penalty values; determining whether the fuel management scheme meets the safety and economic requirements, and if not, continuing to train the deep reinforcement learning network. This method and apparatus quantify the effectiveness of the fuel management scheme output by the reinforcement learning network using reward and penalty values, which facilitates improving the decision-making ability of the reinforcement learning network and can more quickly obtain fuel management schemes that meet the safety and economic requirements of the reactor core.

Description

Technical Field

[0001] This invention belongs to the field of nuclear reactor design, specifically relating to a method and apparatus for generating fuel management schemes based on deep reinforcement learning. Background Technology

[0002] The reactor core is the region where fission nuclear reactions occur, releasing fission energy. The core is composed of multiple fuel assemblies arranged in a specific manner. To ensure reactor safety, the core must meet targets such as power distribution, burnup limits, moderator temperature coefficient, and boom failure limits. Simultaneously, to meet economic requirements, a longer cycle length is needed to generate more heat. Whether the core meets both safety and economic requirements is determined by the characteristics of each fuel assembly and the characteristics of surrounding assemblies. Therefore, the design of the characteristics and arrangement of each fuel assembly in the core—that is, nuclear reactor fuel management—is crucial for achieving both core safety and economic efficiency.

[0003] The search for fuel management schemes in nuclear power plants is a typical combinatorial optimization problem. Its nonlinear, discontinuous, and multi-peak characteristics make the optimization problem complex and difficult to handle. The calculation of each fuel management scheme (including fuel consumption calculation) requires a certain amount of computation time. If we use the exhaustive method to calculate all fuel management schemes and then select the optimal scheme, we can intuitively obtain the global optimal solution. However, as the problem scale increases, the exhaustive method and other exact algorithms are not feasible in terms of computational cost and time.

[0004] Currently, approximation methods are commonly used to find near-optimal solutions to this type of problem. Common combinatorial optimization algorithms include genetic algorithms, ant colony optimization, and simulated annealing. Inspired by Darwin's theory of evolution, genetic algorithms are heuristic search algorithms that borrow from the process of biological evolution. They encode possible solutions to a problem as a chromosome, representing an individual. The algorithm initializes by randomly generating a subset of individuals, then evaluates each individual according to a given fitness function, assigning a fitness value. Individuals with high fitness are selected to generate the next generation; those with low fitness are eliminated, while those with high fitness serve as ancestors, undergoing crossover and mutation to produce new individuals. Genetic algorithms do not require differentiation or constraints on the continuity of functions; they employ a probabilistic optimization method, automatically acquiring and guiding the search space without the need for definite rules, adaptively adjusting the search direction, and evolving gradually towards the optimal solution or a local optimum.

[0005] Ant colony optimization (ACO) is derived from the foraging behavior of ants in nature. Ants can find the shortest path from a food source to their nest without any prompts, and can adaptively search for new optimal paths after environmental changes (such as the addition of obstacles to the original path). Ants communicate with each other by leaving pheromones during their journeys, and use the pheromone concentration feedback to find the optimal solution in the solution space. ACO is autonomous, with each individual ant searching independently and searching in parallel from multiple solution spaces, exhibiting strong global search capabilities. Furthermore, the pheromone concentration and optimization direction of the interaction between ants form a positive feedback process.

[0006] The simulated annealing algorithm draws inspiration from the annealing process of solids. When a solid is at a high temperature, its internal energy is relatively large, and the particles within it are in a state of rapid, disordered motion. As the temperature gradually decreases, the internal energy of the solid decreases, and the particles gradually become more ordered. Eventually, the solid reaches a state of normal temperature, where the internal energy is minimized, and the particles are at their most stable. The simulated annealing algorithm decomposes the problem into three parts: the solution space, the objective function, and the initial solution. By continuously iterating through new solutions and accepting inferior solutions with a certain probability to escape local optima, it eventually approaches the global optimum.

[0007] In summary, traditional optimization algorithms have the following problems:

[0008] 1. Traditional optimization algorithms are an iterative optimization process. As the problem size increases, they still suffer from low computational efficiency. In addition, for more complex environments, traditional optimization algorithms are often unable to model, or the modeling cost is very high.

[0009] 2. Traditional optimization algorithms are uncertain optimizations, tending to seek a set of relatively good solutions and then seek even better solutions on this basis. When dealing with combinatorial optimization problems, they often get caught up in the problem of multiple peak values.

[0010] 3. Traditional optimization algorithms cannot solve dynamic programming problems, such as the game of Go. After a decision is made, the environment changes because of the action taken, and the previous decision made for the specific environment becomes meaningless. It is necessary to iterate and solve the problem again, which has a large computational cost.

[0011] Existing patent CN114203317A discloses a method for loading spent fuel assemblies in a pressurized water reactor core. The method includes a pressurized water reactor core that is a first-cycle core, which is loaded with new fuel assemblies and spent fuel assemblies. The spent fuel assemblies are arranged at the periphery and sub-periphery of the first-cycle core. The spent fuel assemblies are arranged inside the first-cycle core and are distributed in coordination with the new fuel assemblies according to the flattened core power distribution.

[0012] Existing patent CN105139899A discloses a core loading method for the first cycle of a pressurized water reactor nuclear power plant. The first cycle core uses 4 to 6 fuel assemblies with different enrichment levels. The fuel assemblies with the highest enrichment level are placed on the outermost ring of the core, while the different fuel assemblies with lower enrichment levels are placed inside the core in a cross-arrangement. At the same time, borosilicate glass solid combustible poison rods and integrated gadolinium-loaded fuel rods are used to suppress core reactivity and control power distribution. Borosilicate glass solid combustible poison rods are loaded into some fuel assemblies, and integrated gadolinium-loaded fuel rods are loaded into other fuel assemblies.

[0013] The two patents mentioned above employ low-leakage loading and high-leakage loading modes, respectively. They rely on engineers' experience to arrange spent fuel assemblies with new fuel assemblies, resulting in relatively simple fuel management methods and lower economic efficiency and safety of the fuel management solutions. Summary of the Invention

[0014] The purpose of this invention is to address the problems of current fuel management relying heavily on engineer experience and the poor efficiency of traditional methods in automatically searching for refueling schemes. This invention proposes a fuel management scheme generation method and apparatus based on deep reinforcement learning, which can replace manual labor in terms of efficiency and scheme effectiveness.

[0015] To achieve the above objectives, this invention proposes a method for generating fuel management schemes based on deep reinforcement learning, the method comprising:

[0016] S1. For any fuel management scheme, the safety and economic parameters of the fuel management scheme are obtained by solving the nuclear design software.

[0017] S2, Establish reward and penalty standards for evaluating fuel management programs;

[0018] S3, using a deep reinforcement learning network to modify the fuel management scheme based on weights to obtain a new fuel management scheme;

[0019] S4. Based on the safety parameters, economic parameters, and reward and punishment criteria, determine the reward and punishment value of the fuel management scheme and the reward and punishment value of the new fuel management scheme, and determine whether the reward and punishment value of the new fuel management scheme is higher than the reward and punishment value of the fuel management scheme. If not, proceed to step S5; if yes, proceed to step S6.

[0020] S5, update the weights using the gradient descent method until the reward / penalty value of the fuel management scheme after updating the weights is greater than the reward / penalty value of the fuel management scheme, and then return to step S3;

[0021] S6. Further determine whether the optimized fuel management scheme meets the safety and economic requirements. If it does not meet the requirements, return to step S3. If it does meet the requirements, proceed to step S7.

[0022] S7. Test the adaptability of the deep reinforcement learning network with different fuel management schemes. If the adaptability requirements are met, the process ends; otherwise, return to step S3.

[0023] Further, the deep reinforcement learning network is trained, including:

[0024] The deep reinforcement learning network is trained using the DQN algorithm, wherein the DQN algorithm is expressed by Equation 1.

[0025] Formula 1: Among them, s t For the state of the fuel assembly, a t For the actions performed on the fuel assembly, r t Here, θ represents the reward / penalty value, and θ is a parameter of the main network in the deep reinforcement learning network. These are the parameters of the target network in the deep reinforcement learning network.

[0026] Furthermore, the deep reinforcement learning network is trained using the DQN algorithm, including:

[0027] Gradient descent is used to update the parameters θ, thereby reducing the error between the main network and the target network. Here, the parameters θ of the main network are updated in each iteration, and the parameters of the target network are updated accordingly. It is updated for N iterations.

[0028] Furthermore, the safety parameters include core power distribution, burnup limits, moderator temperature coefficient, and spring rod accident limits, while the economic parameters include the number of new fuel assemblies and cycle length.

[0029] Furthermore, the fuel management scheme is modified based on weights using a deep reinforcement learning network, including:

[0030] The weights of the state space and action space of the fuel management scheme in the deep reinforcement learning network are adjusted. The state space includes the enrichment degree of new fuel components, gadolinium content, and number of new fuel components. The action space includes the number of gadolinium rods, the position and angle of new fuel components, and the position and angle of old fuel components.

[0031] To achieve the above objectives, the present invention also proposes an apparatus for generating fuel management schemes based on deep reinforcement learning, comprising:

[0032] The acquisition module is used to solve for the safety and economic parameters of any fuel management scheme using nuclear design software.

[0033] Establish a module to create reward and penalty standards for evaluating fuel management programs;

[0034] The modification module is used to modify the fuel management scheme based on weights using a deep reinforcement learning network to obtain a new fuel management scheme;

[0035] The determination module is used to determine the reward and penalty values ​​of the fuel management scheme and the new fuel management scheme based on the safety parameters, economic parameters and reward and penalty standards, and to compare the reward and penalty values ​​of the two.

[0036] The training module is used to update the weights using gradient descent when the reward / penalty value of the new fuel management scheme is lower than the reward / penalty value of the fuel management scheme, until the reward / penalty value of the fuel management scheme after updating the weights is greater than the reward / penalty value of the fuel management scheme.

[0037] The judgment module is used to determine whether the optimized fuel management scheme meets the safety and economic requirements;

[0038] The testing module is used to test the adaptability of the deep reinforcement learning network with different fuel management schemes when the optimized fuel management scheme meets the safety and economy requirements.

[0039] Furthermore, the training module is used for:

[0040] The deep reinforcement learning network is trained using the DQN algorithm, wherein the DQN algorithm is expressed by Equation 1.

[0041] Formula 1: Among them, s t For the state of the fuel assembly, a t For the actions performed on the fuel assembly, r t Here, θ represents the reward / penalty value, and θ is a parameter of the main network in the deep reinforcement learning network. These are the parameters of the target network in the deep reinforcement learning network.

[0042] Furthermore, the training module is used for:

[0043] Gradient descent is used to update the parameters θ, thereby reducing the error between the main network and the target network. Here, the parameters θ of the main network are updated in each iteration, and the parameters of the target network are updated accordingly. It is updated for N iterations.

[0044] Furthermore, the safety parameters include core power distribution, burnup limits, moderator temperature coefficient, and spring rod accident limits, while the economic parameters include the number of new fuel assemblies and cycle length.

[0045] Furthermore, the modification module is used for:

[0046] The weights of the state space and action space of the fuel management scheme in the deep reinforcement learning network are adjusted. The state space includes the enrichment degree of new fuel components, gadolinium content, and number of new fuel components. The action space includes the number of gadolinium rods, the position and angle of new fuel components, and the position and angle of old fuel components.

[0047] Based on the above technical solution, the present invention has at least the following beneficial effects:

[0048] 1. This invention establishes a comprehensive evaluation standard for fuel management schemes, combines safety and economic parameters obtained from nuclear design software to obtain reward and penalty values ​​for fuel management schemes, and uses the comparison results of reward and penalty values ​​to judge the effectiveness of fuel management schemes output by reinforcement learning networks. This facilitates the improvement of the decision-making ability of reinforcement learning networks and can obtain fuel management schemes that meet the safety and economic requirements of reactor cores more quickly.

[0049] 2. This invention utilizes fuel management adjustment experience to train a deep reinforcement learning network. The agent obtained through deep reinforcement learning interacts autonomously with the environment and generates fuel management solutions, reducing reliance on engineers. At the same time, it utilizes the theoretical optimal solution of reinforcement learning decision-making to make the generated fuel management solutions better.

[0050] 3. This invention combines deep reinforcement learning algorithms to generate fuel management schemes. By utilizing the processing capabilities of deep learning neural networks for high-dimensional data, it effectively solves the problems of low efficiency and high computational complexity of traditional methods in automatic refueling schemes.

[0051] 4. This invention combines deep reinforcement learning algorithms to generate fuel management schemes. It utilizes the characteristics of deep reinforcement learning, namely "offline learning and online decision-making", to store past training samples for subsequent training, thus achieving offline training. After the model has completed training, given a problem is input, and the trained model can directly output the solution to the problem, enabling real-time solution. Attached Figure Description

[0052] The accompanying drawings, which form part of this specification, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings:

[0053] Figure 1 This is a flowchart of a method for generating a fuel management scheme based on deep reinforcement learning, according to an embodiment of the present invention.

[0054] Figure 2 This is a graph showing the variation of Fdh for different schemes formed by a deep reinforcement learning network during the scheme search process, according to a specific embodiment of the present invention.

[0055] Figure 3 This is a schematic diagram of a fuel management scheme based on deep reinforcement learning, according to a specific embodiment of the present invention.

[0056] Figure 4 This is a distribution diagram of Fdh in the core quarter quadrant of a fuel management scheme based on deep reinforcement learning, according to a specific embodiment of the present invention.

[0057] Figure 5 This is a flowchart illustrating the specific operation of the DQN algorithm used in this invention.

[0058] Figure 6 This is a 1 / 8 symmetrical coding diagram of a reactor core fuel assembly according to a specific embodiment of the present invention;

[0059] Figure 7 This is a schematic diagram of a fuel management scheme generation device based on deep reinforcement learning according to an embodiment of the present invention. Detailed Implementation

[0060] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0061] The present invention will be further described in detail below with reference to specific embodiments, which should not be construed as limiting the scope of protection claimed by the present invention.

[0062] Example

[0063] According to one aspect of the present invention, a method for generating fuel management schemes based on deep reinforcement learning is proposed.

[0064] Figure 1 The flowchart illustrates a method for generating a fuel management scheme based on deep reinforcement learning according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0065] S1. For any fuel management scheme, the safety and economic parameters of the fuel management scheme are obtained by solving the nuclear design software.

[0066] This step requires establishing an interface between the reinforcement learning network and the nuclear design software, using the nuclear design software to read the core loading scheme generated by the reinforcement learning network. The "any fuel management scheme" refers to any fuel management scheme selected from existing fuel management schemes.

[0067] In one embodiment of the present invention, any fuel management scheme is input into nuclear design software, and the safety and economic parameters of the fuel management scheme are obtained through solving, and used for the evaluation of the fuel management scheme. Specifically, the safety parameters include core power distribution, burnup limit, moderator temperature coefficient, and boom accident limit, and the economic parameters include the number of new fuel assemblies and cycle length.

[0068] S2, Establish reward and penalty standards for evaluating fuel management programs.

[0069] Specifically, it involves establishing a comprehensive evaluation standard for safety and economic parameters. The evaluation standard established in this embodiment is as follows:

[0070] For safety parameters, if the calculated safety parameters do not meet the limits, the total reward / penalty value of the fuel management scheme is 0; provided that the safety parameter limits are met, the further the deviation from the limits, the higher the score of a single parameter; for economic parameters, the impact on economics is positively correlated with the score.

[0071] It should be understood that the order in which steps S2 and S1 are executed is not restricted in this embodiment.

[0072] S3. Using a deep reinforcement learning network, the fuel management scheme is modified based on the weights to obtain a new fuel management scheme.

[0073] In this step, the fuel management scheme is modified primarily by updating the weights corresponding to the parameters in the state space and action space. The state space includes the enrichment degree, gadolinium content, and number of new fuel assemblies, while the action space includes the number of gadolinium rods, the position and angle of the new fuel assemblies, and the position and angle of the old fuel assemblies.

[0074] In this embodiment, initial weight values ​​are set for the parameters contained in the state space and action space. If the weight of an action (parameter) is large, there is a higher probability that the action will be selected; if the weight of an action is small, there is a lower probability that the action will be selected. Specific actions may include: swapping the positions of fuel assemblies, replacing spent fuel assemblies with new fuel assemblies, replacing new fuel assemblies with spent fuel assemblies, rotating the angle of spent fuel assemblies, modifying the enrichment of new fuel assemblies, and modifying the number of gadolinium rods in new fuel assemblies.

[0075] Specifically, the reinforcement learning network identifies components based on their keff (effective neutron multiplication factor). Using the keff of each component in its four quadrants as a benchmark, the network arranges and combines components, mapping their positions to the positions of each component in the reactor core. This, combined with the weights in the reinforcement learning network, yields a modified core fuel management scheme. As the reinforcement learning network iterates, the weights within it change, thus altering the positions of components with the same keff within the reactor core.

[0076] S4. Based on the safety parameters, economic parameters, and reward / penalty criteria, determine the reward / penalty value of the fuel management scheme and the reward / penalty value of the new fuel management scheme, and determine whether the reward / penalty value of the new fuel management scheme is higher than the reward / penalty value of the fuel management scheme.

[0077] Based on the safety and economic parameters obtained in step S1 and the reward and penalty standards established in step S2, the reward and penalty values ​​of the initial input fuel management scheme can be determined.

[0078] Similarly, the new fuel management scheme obtained in step S3 is input into the nuclear design software for solving, that is, step S1 is executed to obtain the safety parameters and economic parameters of the new fuel management scheme, and the reward and penalty values ​​of the new fuel management scheme are determined according to the reward and penalty standards established in step S2.

[0079] Based on the reward and penalty values ​​of the initial input fuel management scheme and the new fuel management scheme obtained from the above two operations, it is determined whether the reward and penalty value of the new fuel management scheme is higher than the reward and penalty value of the initial input fuel management scheme.

[0080] If, after assessment, the reward / penalty value of the new fuel management scheme is lower than the reward / penalty value of the initially input fuel management scheme, then step S5 is executed; if the reward / penalty value of the new fuel management scheme is higher than the reward / penalty value of the initially input fuel management scheme, then step S6 is executed.

[0081] S5, update the weights using the gradient descent method until the reward / penalty value of the fuel management scheme after updating the weights is greater than the reward / penalty value of the fuel management scheme, and then return to step S3.

[0082] Specifically, when the reward / penalty value of the new fuel management scheme is lower than the reward / penalty value of the initially input fuel management scheme, this embodiment uses gradient descent to update the initial weight value and returns to step S3. Based on the new weights, the fuel management scheme is modified to obtain a new fuel management scheme. Then, step S4 is executed, comparing the reward / penalty value of the new fuel management scheme with the reward / penalty value of the initial fuel management scheme. If the reward / penalty value of the new fuel management scheme is higher than the reward / penalty value of the initial fuel management scheme, step S6 can be executed; otherwise, step S5 continues until the reward / penalty value of the fuel management scheme after updating the weights is greater than the reward / penalty value of the initial fuel management scheme. Figure 2 The figure shows the variation of Fdh for different schemes formed by the deep reinforcement learning network during the scheme search process, where the horizontal axis represents the number of searches.

[0083] S6, further determine whether the optimized fuel management scheme meets the safety and economic requirements.

[0084] If step S4 shows that the reward / penalty value of the new fuel management scheme is higher than the reward / penalty value of the original fuel management scheme, then it can be further determined whether the optimized new fuel management scheme meets the safety and economic requirements.

[0085] If step S6 determines that the optimized new fuel management scheme meets the safety and economic requirements, then step S7 is executed; otherwise, if step S6 determines that the optimized new fuel management scheme does not meet the safety and economic requirements, then the process returns to step S3 to continue training the deep reinforcement learning network.

[0086] S7. Test the adaptability of the deep reinforcement learning network using different fuel management schemes.

[0087] If the optimized new fuel management scheme obtained after step S6 meets the safety and economic requirements, then multiple different initial fuel management schemes are used as test data to test the adaptability of the deep reinforcement learning network.

[0088] If the deep reinforcement learning network meets the adaptability requirements after testing, it indicates that a deep reinforcement learning network that meets the requirements for generating a fuel management scheme has been obtained, and the process ends; if the test result does not meet the adaptability requirements, it is necessary to return to step S3 and continue training the deep reinforcement learning network.

[0089] like Figure 3 The fuel management scheme based on a deep reinforcement learning network after deep learning, as shown in the specific embodiment, meets the limit requirements for power distribution, moderator temperature coefficient, cycle length, and fuel consumption. Figure 4This is a specific embodiment of the fuel management scheme based on deep reinforcement learning, showing the Fdh distribution within the quarter quadrant of the reactor core.

[0090] The training process of the deep reinforcement learning network in this invention will be described in detail below.

[0091] Traditional Q-learning algorithms use a Q-table to store state-action pairs (s, a). The assembly and combination problem to be solved in this invention is a high-dimensional, discrete optimization problem. In this embodiment, the DQN algorithm is specifically used to train a deep reinforcement learning network. Combining the deep reinforcement learning network's ability to process high-dimensional data with the theoretical optimal solution of reinforcement learning decision-making, the combinatorial optimization problem can be solved better.

[0092] Specifically, the DQN algorithm has two parallel networks: the network used for training is called the main network, and the network used for prediction is called the target network. The goal of training is to use gradient descent to update the parameters θ so as to minimize the error between the main network and the target network.

[0093] The error definition of the DQN algorithm is shown in Formula 1.

[0094] Formula 1: Among them, s t For the state of the fuel assembly, a t For the actions performed on the fuel assembly, r t Here, θ represents the reward / penalty value, and θ is a parameter of the main network in the deep reinforcement learning network. These are the parameters of the target network in a deep reinforcement learning network, Q(s) t ,a t ;θ) is the predicted value of the neural network, It is the update target of the temporal difference algorithm.

[0095] Furthermore, this invention uses the expert's material-changing action selection process as a training set input into a deep reinforcement learning network, and during parameter initialization, ensures that the parameters of the main network and the target network are identical, i.e. During training, the parameters θ of the main network are updated in each iteration, while the parameters of the target network... The update is performed in N iterations, meaning it is updated every N steps. The update method is to directly copy the parameters θ of the main network.

[0096] Figure 5 The specific computational flow of the DQN algorithm is shown in the figure, such as Figure 5 As shown, before the algorithm starts, an initial state s of a fuel component needs to be randomly obtained from the fuel component environment.

[0097] Secondly, select an action 'a' that maximizes the value of Q(s,a;θ) through the main network.

[0098] The selected action 'a' is input into the fuel assembly environment, and the agent obtained by deep reinforcement learning executes this action 'a' to generate a preliminary fuel management plan.

[0099] After executing action a, the fuel component environment solves the initially formed fuel management scheme and returns a reward / penalty value r and the next state a', thus obtaining a data sequence (s, a, r, s'), which is stored in the experience replay pool. Then, the next state s' is regarded as the current state s, and the next action a' to be executed is selected according to the reward / penalty value r. The above steps are repeated until the experience replay pool is full.

[0100] Once the experience replay pool is full, the main network and target network in the DQN algorithm are updated. To ensure that the input data conforms to independent and identically distributed distributions and to break the correlation between data during network updates, random sampling is performed from the experience replay pool. The sampled fuel component states and executed actions (s) are then used to update the data. t ,a t The input is fed into the main network to calculate Q(s) in the current state. t ,a t The value of θ) is also obtained. Simultaneously, the next state s of the sampled fuel assembly is determined. t+1 Input the target network, and select a' through the target network such that The maximum value is Q(s) calculated by combining the results of the main network and the target network. t ,a t ;θ) and the reward / penalty value r obtained from sampling t The DQN error value is calculated according to Formula 1.

[0101] Based on the calculated DQN error, the main network parameters θ are updated using gradient descent.

[0102] Repeat the above steps, updating the main network parameters θ in each iteration. Gradient descent is used to update θ, reducing the error between the main network and the target network. Every N steps, the main network parameters θ are copied to the target network parameters. Training is complete when the error between the main network and the target network converges.

[0103] In one specific embodiment, a reactor core contains 121 fuel assemblies, placed in 121 locations to form a fuel management scheme, considering that the reactor core has 1 / 8 symmetry ( Figure 6The diagram shows a 1 / 8 symmetric coding diagram of the reactor core fuel assemblies. Therefore, only the positions of 21 fuel assemblies need to be determined, meaning that coding optimization needs to be performed on these 21 positions, while the positions of the remaining assemblies are determined. Since the burnup in each quarter quadrant of each spent fuel assembly is different, to obtain a better fuel management scheme, each spent fuel assembly except the central assembly needs to be rotated. This means there are 84 selectable fuel assemblies, and 21 positions need to be determined.

[0104] in, Figure 6 Position 1 represents the location of the central component, and position 21 represents the location of the spent fuel assembly or a new assembly. First, the system initializes a state as input. This initialization state can be encoded using a 21×3 array, where the first column represents the position number, the second column represents the name of the selected assembly, and the third column represents the assembly's rotation angle. Next, Q(s) is selected through the main network. t ,a t The action a corresponding to the maximum value of θ) t The system will calculate the reward / penalty value r based on the current optimization plan. t Action a t Represented by a 21×2 array, the first column is the name of the fuel component, and the second column is the corresponding rotation angle.

[0105] This can be understood as follows: the agent obtained by deep reinforcement learning automatically adjusts its components according to the above strategy. Each time, two components are swapped and the rotation angle is determined. If it is the central component, no rotation is required. Then the system calculates the reward and penalty value according to the current optimization scheme and enters the next state.

[0106] In addition, this invention incorporates a greedy algorithm to balance the exploration and utilization of the agent. At the beginning of model training, a large exploration factor is set to allow the model to explore more actions, that is, to have more actions to choose from in the process of forming a fuel management plan, which makes it easier to find the optimal fuel management plan. After the model training is completed, the exploration factor is reduced, so that the agent's choice of refueling actions tends to be more stable and more dependent on the strategies that the model has learned.

[0107] To achieve the above objectives, the present invention also proposes a fuel management scheme generation device based on deep reinforcement learning.

[0108] Figure 7 The diagram shows a schematic of a fuel management scheme generation device based on deep reinforcement learning according to an embodiment of the present invention. Figure 7 As shown, the device includes: an acquisition module 71, an establishment module 72, a modification module 73, a determination module 74, a training module 75, a judgment module 76, and a testing module 77, which are described in detail below:

[0109] The acquisition module 71 is used to obtain the safety and economic parameters of any fuel management scheme by using nuclear design software.

[0110] Specifically, the safety parameters include core power distribution, burnup limits, moderator temperature coefficient, and spring rod accident limits, while the economic parameters include the number of new fuel assemblies and cycle length.

[0111] Module 72 is established to create reward and penalty standards for evaluating fuel management programs.

[0112] Modify module 73 to modify the fuel management scheme based on weights using a deep reinforcement learning network to obtain a new fuel management scheme.

[0113] The modification module 73 is specifically used for:

[0114] The weights of the state space and action space of the fuel management scheme in the deep reinforcement learning network are adjusted. The state space includes the enrichment degree of new fuel components, gadolinium content, and number of new fuel components. The action space includes the number of gadolinium rods, the position and angle of new fuel components, and the position and angle of old fuel components.

[0115] The determination module 74 is used to determine the reward and penalty values ​​of the fuel management scheme and the new fuel management scheme based on the safety parameters, economic parameters and reward and penalty standards, and to compare the reward and penalty values ​​of the two.

[0116] Training module 75 is used to train the deep reinforcement learning network when the optimized fuel management scheme does not meet the safety and economic requirements, until the optimized fuel management scheme meets the safety and economic requirements.

[0117] The training module 75 is specifically used for:

[0118] The deep reinforcement learning network is trained using the DQN algorithm, wherein the DQN algorithm is expressed by Equation 1.

[0119] Formula 1: Among them, s t For the state of the fuel assembly, a t For the actions performed on the fuel assembly, r t Here, θ represents the reward / penalty value, and θ is a parameter of the main network in the deep reinforcement learning network. These are the parameters of the target network in the deep reinforcement learning network.

[0120] The training module 75 is specifically used for:

[0121] Gradient descent is used to update the parameters θ, thereby reducing the error between the main network and the target network. Here, the parameters θ of the main network are updated in each iteration, and the parameters of the target network are updated accordingly. It is updated for N iterations.

[0122] The judgment module 76 is used to determine whether the optimized fuel management scheme meets the safety and economic requirements.

[0123] The testing module 77 is used to test the adaptability of the deep reinforcement learning network with different fuel management schemes when the optimized fuel management scheme meets the safety and economy requirements.

[0124] It should be understood that the embodiment of the fuel management scheme generation device based on deep reinforcement learning is consistent with the embodiment of the corresponding fuel management scheme generation method based on deep reinforcement learning, so it will not be described again in this embodiment.

[0125] In summary, as can be seen from the above description, the above embodiments of the method and apparatus for generating fuel management schemes based on deep reinforcement learning achieve the following technical effects:

[0126] 1. This invention establishes a comprehensive evaluation standard for fuel management schemes, combines safety and economic parameters obtained from nuclear design software to obtain reward and penalty values ​​for fuel management schemes, and uses the comparison results of reward and penalty values ​​to judge the effectiveness of fuel management schemes output by reinforcement learning networks. This facilitates the improvement of the decision-making ability of reinforcement learning networks and can obtain fuel management schemes that meet the safety and economic requirements of reactor cores more quickly.

[0127] 2. This invention utilizes fuel management adjustment experience to train a deep reinforcement learning network. The agent obtained through deep reinforcement learning interacts autonomously with the environment and generates fuel management solutions, reducing reliance on engineers. At the same time, it utilizes the theoretical optimal solution of reinforcement learning decision-making to make the generated fuel management solutions better.

[0128] 3. This invention combines deep reinforcement learning algorithms to generate fuel management schemes. By utilizing the processing capabilities of deep learning neural networks for high-dimensional data, it effectively solves the problems of low efficiency and high computational complexity of traditional methods in automatic refueling schemes.

[0129] 4. This invention combines deep reinforcement learning algorithms to generate fuel management schemes. It utilizes the characteristics of deep reinforcement learning, namely "offline learning and online decision-making", to store past training samples for subsequent training, thus achieving offline training. After the model has completed training, given a problem is input, and the trained model can directly output the solution to the problem, enabling real-time solution.

[0130] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0131] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus or device (such as a computer-based system, a processor-included system or other system that can fetch and execute instructions from, an instruction execution system, apparatus or device).

[0132] It should be understood that various parts of the present invention can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system.

[0133] It should be noted that, in the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0134] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for generating fuel management schemes based on deep reinforcement learning, characterized in that, include: S1. For any fuel management scheme, the safety parameters and economic parameters of the fuel management scheme are obtained by solving the nuclear design software. The safety parameters include core power distribution, burnup limit, moderator temperature coefficient, and boom accident limit. The economic parameters include the number of new fuel assemblies and cycle length. S2, Establish reward and penalty standards for evaluating fuel management schemes, including: establishing comprehensive evaluation standards for the safety parameters and the economic parameters; S3, using a deep reinforcement learning network to modify the fuel management scheme based on weights to obtain a new fuel management scheme, including: adjusting the weights of the state space and action space of the fuel management scheme in the deep reinforcement learning network, wherein the state space includes the enrichment degree of the new fuel component, the gadolinium content, and the number of new fuel components, and the action space includes the number of gadolinium rods, the position and angle of the new fuel component, and the position and angle of the old fuel component; S4. Based on the safety parameters, economic parameters, and reward and punishment criteria, determine the reward and punishment value of the fuel management scheme and the reward and punishment value of the new fuel management scheme, and determine whether the reward and punishment value of the new fuel management scheme is higher than the reward and punishment value of the fuel management scheme. If not, proceed to step S5; if yes, proceed to step S6. S5, update the weights using the gradient descent method until the reward / penalty value of the fuel management scheme after updating the weights is greater than the reward / penalty value of the fuel management scheme, and then return to step S3; S6. Further determine whether the new fuel management scheme meets the safety and economic requirements. If it does not meet the requirements, return to step S3. If it does meet the requirements, proceed to step S7. S7. Test the adaptability of the deep reinforcement learning network with different fuel management schemes. If the adaptability requirements are met, the process ends; otherwise, return to step S3.

2. The method according to claim 1, characterized in that, Training the deep reinforcement learning network includes: The deep reinforcement learning network is trained using the DQN algorithm, wherein the DQN algorithm is expressed by Equation 1. Formula 1: ,in, The state of the fuel assembly. For the actions performed on the fuel assembly, As a reward and punishment value, These are the parameters of the main network in the deep reinforcement learning network. These are the parameters of the target network in the deep reinforcement learning network.

3. The method according to claim 2, characterized in that, Training the deep reinforcement learning network using the DQN algorithm includes: Update parameters using gradient descent This reduces the error between the main network and the target network, wherein the parameters of the main network... For each iteration update, the parameters of the target network are... It is updated for N iterations.

4. A device for generating fuel management schemes based on deep reinforcement learning, characterized in that, include: The acquisition module is used to solve for any fuel management scheme using nuclear design software to obtain the safety parameters and economic parameters of the fuel management scheme. The safety parameters include core power distribution, burnup limit, moderator temperature coefficient, and boom accident limit. The economic parameters include the number of new fuel assemblies and cycle length. A module is established to create reward and penalty standards for evaluating fuel management schemes, including: establishing comprehensive evaluation standards for the safety parameters and the economic parameters; The modification module is used to modify the fuel management scheme based on weights using a deep reinforcement learning network to obtain a new fuel management scheme. The modification module includes: adjusting the weights of the state space and action space of the fuel management scheme in the deep reinforcement learning network. The state space includes the enrichment degree of the new fuel component, the gadolinium content, and the number of new fuel components. The action space includes the number of gadolinium rods, the position and angle of the new fuel component, and the position and angle of the old fuel component. The determination module is used to determine the reward and penalty values ​​of the fuel management scheme and the new fuel management scheme based on the safety parameters, economic parameters and reward and penalty standards, and to compare the reward and penalty values ​​of the two. The training module is used to update the weights using gradient descent when the reward / penalty value of the new fuel management scheme is lower than the reward / penalty value of the fuel management scheme, until the reward / penalty value of the fuel management scheme after updating the weights is greater than the reward / penalty value of the fuel management scheme. The judgment module is used to determine whether the new fuel management scheme meets the safety and economic requirements; The testing module is used to test the adaptability of the deep reinforcement learning network with different fuel management schemes while ensuring that the new fuel management scheme meets the safety and economy requirements.

5. The apparatus according to claim 4, characterized in that, The training module is used for: The deep reinforcement learning network is trained using the DQN algorithm, wherein the DQN algorithm is expressed by Equation 1. Formula 1: ,in, The state of the fuel assembly. For the actions performed on the fuel assembly, As a reward and punishment value, These are the parameters of the main network in the deep reinforcement learning network. These are the parameters of the target network in the deep reinforcement learning network.

6. The apparatus according to claim 5, characterized in that, The training module is used for: Update parameters using gradient descent This reduces the error between the main network and the target network, wherein the parameters of the main network... For each iteration update, the parameters of the target network are... It is updated for N iterations.

Images