A deep reinforcement learning geothermal scheduling method for a geothermal station

By improving the sugarcane rat optimization algorithm and the safety action shielding mechanism, the deep reinforcement learning model of the geothermal station was optimized, which solved the problems of slow convergence and safety of the geothermal station and achieved stable and efficient operation of the system.

CN123117624APending Publication Date: 2026-11-17SINOPEC GREEN ENERGY GEOTHERMAL DEV CO LTD +2
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611263380.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-08-19
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

In existing technologies, deep reinforcement learning algorithms for geothermal stations converge slowly, are prone to getting stuck in local optima, and lack security guarantees, leading to unstable equipment operation and safety hazards.

Method used

An improved sugarcane rat optimization algorithm is used for global search. The network parameters are optimized by combining chaotic initialization and gray wolf cooperation mechanism. A safety action shielding mechanism is introduced, and the network parameters are fine-tuned by gradient descent method. A multi-objective reward function is constructed to ensure safety and energy efficiency.

Benefits of technology

The algorithm's global search capability and convergence accuracy have been improved, safe out-of-bounds behavior has been avoided, and the system's safety, stability and energy efficiency have been maximized.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123117624A_ABST
    Figure CN123117624A_ABST
Patent Text Reader

Abstract

This invention discloses a deep reinforcement learning-based geothermal scheduling method for geothermal stations, relating to the field of intelligent geothermal energy control technology. The method includes: constructing a geothermal scheduling model using a deep reinforcement learning algorithm; mapping the initial network parameters of the geothermal scheduling model to individual position vectors, and using an improved sugarcane mouse optimization algorithm to perform a global search in the parameter space to find the optimal initial network parameters; initializing the geothermal scheduling model using the optimal initial parameters, introducing a safety action shielding mechanism, performing interactive learning in a geothermal station simulation environment, and fine-tuning the network parameters using gradient descent to obtain the optimal geothermal scheduling model; deploying the optimal geothermal scheduling model to the field control system of the geothermal station, collecting real-time operating data of the geothermal station, performing geothermal scheduling, and outputting optimal control commands. This invention solves the problems of slow convergence, susceptibility to local optima, and lack of safety guarantees during the training process in existing technologies.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of intelligent control technology for geothermal energy, and in particular to a deep reinforcement learning geothermal scheduling method for geothermal stations. Background Technology

[0002] Geothermal energy, as a clean and stable renewable energy source, is widely used in heating, power generation, and other fields. The operation and scheduling of geothermal stations involves the coordinated control of various equipment, including submersible pumps, reinjection pumps, circulation pumps, and valves in production wells. Due to the nonlinear, large time-lag, strong coupling, and time-varying characteristics of geothermal systems, traditional PID-based control or rule-based expert systems often struggle to maintain efficient operation across the entire operating range, making it difficult to achieve optimal energy efficiency.

[0003] In recent years, deep reinforcement learning has shown great potential in solving complex decision-making problems. However, applying it to geothermal station scheduling faces two major challenges: First, deep reinforcement learning algorithms typically use random initialization of network parameters, leading to blind exploration and slow convergence in the early stages of training, and making it prone to getting trapped in local optima, thus failing to find a globally optimal control strategy. Second, geothermal station equipment operation is subject to strict physical safety constraints (such as wellhead pressure limits and downhole liquid level safety lines). The random exploration actions of reinforcement learning in the early stages of training may lead to equipment damage or safety accidents. Although negative reward penalties can be set to constrain these actions, they often lag behind in dealing with serious boundary violations, making it difficult to fundamentally guarantee the absolute safety of the system.

[0004] Therefore, improving the convergence efficiency of the algorithm and ensuring the safety of the scheduling process are urgent problems to be solved. Summary of the Invention

[0005] This invention provides a deep reinforcement learning-based geothermal scheduling method for geothermal stations. This invention addresses the problems of slow convergence, susceptibility to local optima, and lack of security guarantees during the training process in existing reinforcement learning technologies.

[0006] This invention provides a deep reinforcement learning-based geothermal scheduling method for geothermal stations, the method comprising: A geothermal scheduling model is constructed using deep reinforcement learning algorithms. The initial network parameters of the geothermal scheduling model are mapped to the location vectors of individuals, and the improved sugarcane rat optimization algorithm is used to perform a global search in the parameter space to find the optimal initial network parameters. The geothermal scheduling model is initialized using optimal initial parameters, a safety action shielding mechanism is introduced, interactive learning is performed in the geothermal station simulation environment, and the network parameters are fine-tuned using the gradient descent method to obtain the optimal geothermal scheduling model. The optimal geothermal scheduling model is deployed to the field control system of the geothermal station to collect real-time operating data of the geothermal station, perform geothermal scheduling, and output optimal control commands.

[0007] The technical solution provided in this application has at least the following beneficial effects: An improved sugarcane rat optimization algorithm is used for global pre-training of deep reinforcement learning network parameters. Chaotic initialization and gray wolf cooperation mechanisms enhance the algorithm's global search capability and convergence accuracy, effectively solving the problems of slow convergence and local optima caused by random initialization in traditional reinforcement learning. A safety action shielding mechanism is introduced, which predicts the state at the next moment through the physical model and pre-shields unsafe exploration actions, fundamentally avoiding potential safety overstepping behaviors (such as evacuation and overpressure) during training and operation, greatly improving the system's safety and stability. The constructed multi-objective reward function comprehensively considers energy efficiency, energy consumption, load matching, and safety constraints, guiding the model to maximize system energy efficiency while ensuring heating demand.

[0008] In one alternative implementation, a deep reinforcement learning algorithm is used to construct a geothermal scheduling model, including: Set up state vectors based on indicators from historical operational data; Define the control variables for the equipment at the geothermal station and set the action space; A multi-objective reward function is constructed by considering the overall system's energy efficiency maximization, energy consumption minimization, load deviation minimization, and safety constraints. Based on deep reinforcement learning algorithms, a model structure including a policy network and a valuation network is constructed, and the initial network parameters to be optimized are determined. A geothermal scheduling model is constructed by combining state vectors, action space, multi-objective reward functions, and model structure.

[0009] In one alternative implementation, the state vector includes normalized geothermal wellhead temperature, wellhead pressure, downhole fluid level, reinjection temperature, ambient temperature, terminal heat load demand, and timestamp. The control variables of the action space include the frequency of the production well submersible pump, the frequency of the reinjection pump, and the opening degree of the main pipeline valve. The initial network parameters to be optimized include the weight matrices and bias vectors of the policy network and the valuation network.

[0010] In one alternative implementation, the formula for the multi-objective reward function is: In the formula, This is the instantaneous total reward value; The instantaneous energy efficiency ratio of the geothermal station; This is the rated maximum energy efficiency. This represents the total power consumption of the circulating pump. Rated power; This refers to the actual heat supply; For user-side load requirements; This is the rated maximum heating capacity; This is a safety penalty item; Let be the weighting coefficients, and let sum to 1.

[0011] In one alternative implementation, the initial network parameters of the geothermal scheduling model are mapped to individual location vectors, and an improved sugarcane rat optimization algorithm is used to perform a global search in the parameter space to find the optimal initial network parameters, including: The initial network parameters of the geothermal scheduling model are flattened by row or column, concatenated into a one-dimensional vector, and encoded as the location vector of each individual. The chaotic sequence is generated using the Logistic mapping and then mapped to the parameter space of individuals in the improved sugarcane rat optimization algorithm to obtain an initial population including several initial individuals. The fitness function is defined as the negative of the cumulative instantaneous comprehensive reward value. Based on the geothermal station simulation environment, the fitness function is used to calculate the fitness value of each individual in the initial population or the updated population of the previous iteration. Based on the fitness value, select the top three individuals, the second best individuals, and the third best individuals in terms of fitness from the initial population or the population updated in the previous iteration. Based on the gray wolf cooperation concept, a convergence factor is introduced. The optimal individual, the second-best individual, and the third-best individual are combined to update the position of the initial population or the population updated in the previous iteration, so as to obtain the updated population in the current iteration. The position update of the population is repeated. The iterative update of the population is terminated when the current iteration reaches the maximum number of iterations or when the fitness value of the best individual in the population updated in the current iteration meets the requirements. The optimal initial network parameters are obtained by decoding the position vector of the best individual in the updated population of the current iteration.

[0012] In one alternative implementation, the geothermal scheduling model is initialized using optimal initial parameters, a safety action shielding mechanism is introduced, interactive learning is performed in a geothermal station simulation environment, and the network parameters are fine-tuned using the gradient descent method to obtain the optimal geothermal scheduling model, including: Based on the optimal initial parameters, the geothermal scheduling model is initialized to obtain the initialized geothermal scheduling model. A safety action shielding mechanism is introduced into the initial geothermal scheduling model, and a physical model prediction module and a safety action shielding module are set up. Using the initialized geothermal scheduling model, we conduct trial and error learning in the geothermal station simulation environment, accumulate training data, and store the training data in the experience playback pool. Several training data were randomly collected from the experience replay pool, and the network parameters of the initialized geothermal scheduling model were fine-tuned using the gradient descent method. The optimal geothermal scheduling model was obtained when the model converged.

[0013] In one optional implementation, the physical model prediction module is used to predict the key prediction state at the next moment corresponding to the original action output by the initialized geothermal scheduling model. The key prediction state includes the predicted downhole liquid level and the predicted wellhead pressure. The security action shielding module is used to make a shielding determination based on the key predicted state at the next moment, and select to execute a security supply guarantee strategy based on the shielding determination result, so as to obtain an executable action after security filtering.

[0014] In one alternative implementation, an initialized geothermal scheduling model is used to perform trial-and-error learning in a geothermal station simulation environment, accumulating training data, and storing the training data in an experience replay pool, including: Historical operating data of the geothermal station is collected and preprocessed to obtain preprocessed historical operating data, which is then written into the state space of the initialized geothermal scheduling model to obtain the historical state at the corresponding time. Based on historical conditions, the initialized geothermal scheduling model is run in the geothermal station simulation environment to generate original actions in the action space. Using the physical model prediction module, the key prediction state of the next moment corresponding to the original action is predicted, and combined with the external environmental variables directly output by the working condition generator in the geothermal station simulation environment, the full prediction state of the next moment with the same structure as the historical state is obtained. The security action shielding module is used to make a shielding determination based on the critical predicted state of the next moment, and based on the shielding determination result, a security supply guarantee strategy is selected to be executed to obtain the executable actions after security filtering. In the geothermal station simulation environment, a multi-objective reward function is invoked to generate the corresponding historical instantaneous comprehensive reward value; The historical state, executable actions, historical instantaneous comprehensive reward value, and the full predicted state at the next moment are used as training data, and the training data is stored in the experience replay pool.

[0015] In one alternative implementation, the optimal geothermal scheduling model is deployed to the field control system of the geothermal station, real-time operating data of the geothermal station is collected, geothermal scheduling is performed, and optimal control commands are output, including: The optimal network parameters of the optimal strategy network in the optimal geothermal scheduling model are extracted, converted into a lightweight format, and the backpropagation computation graph required for training is removed, leaving only the forward inference computation graph, to obtain the lightweight optimal network parameters. Lightweight optimal network parameters are written into the non-volatile memory area of ​​the controller of the field control system of the geothermal station to generate a strategy inference function that can be called in real time. Collect real-time operating data from the geothermal station, preprocess it to obtain preprocessed real-time operating data, and generate the real-time state of the state space that conforms to the optimal geothermal scheduling model. The real-time state is input into the policy inference function to generate real-time actions in the action space; Based on the physical upper and lower limits of the geothermal station equipment, the normalized real-time actions are mapped back to the actual control values ​​to generate optimal control commands, including the actual production well submersible pump frequency value, the actual reinjection pump frequency value, and the actual main pipeline valve opening value. Based on the maximum allowable change in a single cycle, the rate of change of the optimal control command is limited to obtain the limited optimal control command, which is then written into the PLC register of the field control system to execute geothermal scheduling. Attached Figure Description

[0016] Figure 1 A flowchart illustrating the steps of a deep reinforcement learning geothermal scheduling method for a geothermal station, provided in an embodiment of the present invention. Detailed Implementation

[0017] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0018] The present invention will be further described below with reference to the accompanying drawings.

[0019] Reference Figure 1 The present invention provides a deep reinforcement learning geothermal scheduling method for a geothermal station, the method comprising: S1: Use deep reinforcement learning algorithms to build a geothermal scheduling model; S2: Map the initial network parameters of the geothermal scheduling model to the location vectors of individuals, and use the improved sugarcane rat optimization algorithm to perform a global search in the parameter space to find the optimal initial network parameters; S3: Initialize the geothermal scheduling model using optimal initial parameters, introduce a safety action shielding mechanism, conduct interactive learning in the geothermal station simulation environment, fine-tune the network parameters using the gradient descent method, and obtain the optimal geothermal scheduling model. S4: Deploy the optimal geothermal scheduling model to the field control system of the geothermal station, collect real-time operating data of the geothermal station, perform geothermal scheduling, and output the optimal control command.

[0020] The technical solution provided in this application has at least the following beneficial effects: An improved sugarcane rat optimization algorithm is used for global pre-training of deep reinforcement learning network parameters. Chaotic initialization and gray wolf cooperation mechanisms enhance the algorithm's global search capability and convergence accuracy, effectively solving the problems of slow convergence and local optima caused by random initialization in traditional reinforcement learning. A safety action shielding mechanism is introduced, which predicts the state at the next moment through the physical model and pre-shields unsafe exploration actions, fundamentally avoiding potential safety overstepping behaviors (such as evacuation and overpressure) during training and operation, greatly improving the system's safety and stability. The constructed multi-objective reward function comprehensively considers energy efficiency, energy consumption, load matching, and safety constraints, guiding the model to maximize system energy efficiency while ensuring heating demand.

[0021] In one alternative implementation, a deep reinforcement learning algorithm is used to construct a geothermal scheduling model, including: S11: Set the state vector based on the indicators of historical operation data; S12: Define the control variables for the equipment in the geothermal station and set the action space; S13: Construct a multi-objective reward function that integrates system energy efficiency maximization, energy consumption minimization, load deviation minimization, and safety constraints; S14: Based on deep reinforcement learning algorithms, construct a model structure that includes a policy network and a valuation network, and determine the initial network parameters to be optimized; In this embodiment, the number of nodes in the input layer conforms to the dimension of the state space, the hidden layer uses a fully connected layer (e.g., the number of neurons is [64, 32]), the activation function is the ReLU function, the number of nodes in the output layer conforms to the dimension of the action space, and the activation function is the Tanh function to limit the output to […]. The interval is [1,1]. S15: Combine state vectors, action space, multi-objective reward function and model structure to construct a geothermal scheduling model.

[0022] In one alternative implementation, the state vector includes normalized geothermal wellhead temperature, wellhead pressure, downhole fluid level, reinjection temperature, ambient temperature, terminal heat load demand, and timestamp. In this embodiment, the geothermal wellhead temperature ranges from [45, 90] ℃, and the measured value is normalized by dividing by 90. Wellhead pressure: range [0.1, 0.6] MPa, measured value divided by 0.6 for normalization; Downhole fluid level: The value range is [50, 300] meters. The measured value is normalized by subtracting 50 from the measured value and then dividing by 250. Reinjection temperature: range [30, 50]℃, measured value divided by 50 for normalization; Ambient temperature: range [-20, 15]℃, measured value plus 20 and then divided by 35 for normalization; Terminal heat load demand: predicted based on historical data, with a value range of [0, 2000] kW. The measured value is normalized by dividing by 2000. Timestamp: Reflects the periodicity of heating, with a value range of [0, 24) hours; The control variables of the action space include the production well submersible pump frequency in the range of [25, 50] Hz, the reinjection pump frequency in the range of [20, 45] Hz, and the main pipeline valve opening in the range of [0, 100]%. The initial network parameters to be optimized include the weight matrices and bias vectors of the policy network and the valuation network.

[0023] In one alternative implementation, the formula for the multi-objective reward function is: In the formula, The instantaneous comprehensive reward value guides the optimization direction of deep reinforcement learning algorithms. The goal is to maximize the long-term cumulative reward by adjusting network parameters, thereby achieving the optimal scheduling strategy. Theoretically, it is in the real number domain, but in actual operation, it is usually affected by various constraints within […]. Fluctuations within the range of 100,1 (depending on the size of the penalty term); The instantaneous energy efficiency ratio of a geothermal station is an indicator reflecting the energy conversion efficiency of the station. It is the ratio of effective heating / power supply to the total input power of the system (including submersible pumps, circulating pumps, etc.). As a normalization factor, the energy efficiency term is mapped to the [0,1] interval to eliminate the influence of dimensions and enable it to be weighted and summed with the energy consumption term and the deviation term. It is usually taken as the maximum value in historical operating data or the design theoretical value. This is the rated maximum energy efficiency. The total power consumption of the circulating pump is the sum of the instantaneous power of all pumps in the geothermal station (submersible pumps for production wells, reinjection pumps, and circulating pumps). The rated power is the sum of the rated power of all pump equipment. The normalization factor maps the power consumption term to the [0,1] interval to ensure that the weight coefficients of each item are on the same order of magnitude, which facilitates parameter tuning. This refers to the actual heat supply; For user-side load requirements; The maximum rated heating capacity is the maximum heating power of the geothermal station under design conditions. The normalization factor maps the deviation term to the [0,1] interval. This is a safety penalty item, a hard penalty value for violating physical safety constraints, triggered by the following conditions (logical OR relation): Condition 1: The downhole fluid level is below the safety warning line (80 meters) to prevent the well pump from running dry and burning out; Condition 2: The wellhead pressure exceeds the maximum threshold (0.55 MPa) to prevent pipe bursting or equipment damage. If triggered, then... The value is 100, otherwise it is 0. Let be the weighting coefficients, and let sum to 1.

[0024] In one alternative implementation, the initial network parameters of the geothermal scheduling model are mapped to individual location vectors, and an improved sugarcane rat optimization algorithm is used to perform a global search in the parameter space to find the optimal initial network parameters, including: S21: Flatten the initial network parameters of the geothermal scheduling model by row or column, concatenate them into a one-dimensional vector, and encode them as individual location vectors; S22: Use Logistic mapping to generate chaotic sequences, and map the chaotic sequences to the parameter space of individuals in the improved sugarcane rat optimization algorithm to obtain an initial population including several initial individuals; In the formula, For the first n+ 1. n There are several chaotic variables whose values ​​range from [0, 1]. The stability coefficient is typically 4. This sequence is ergodic and random, ensuring that the initial population is uniformly distributed in the solution space, avoiding getting trapped in local optima, which is superior to traditional random initialization. n Indicator of chaotic variables; In the formula, For the initial population, the first i An initial individual; For the first i One chaotic variable; These are the upper and lower bounds of the search space; i For individual indicators; S23: Define the fitness function as the negative of the cumulative instantaneous comprehensive reward value, and use the fitness function to calculate the fitness value of each individual in the initial population or the updated population of the previous iteration based on the geothermal station simulation environment. In this embodiment, the candidate initial network parameters corresponding to each individual in the initial population or the population updated in the previous iteration are loaded into the geothermal scheduling model in the geothermal station simulation environment. The model runs for several rounds in the geothermal station simulation environment, and the corresponding cumulative instantaneous comprehensive reward value is extracted. Then, the corresponding fitness value is calculated based on the fitness function. S24: Based on the fitness value, select the top three individuals, the second best individuals, and the third best individuals in terms of fitness from the initial population or the population updated in the previous iteration. S25: Based on the gray wolf cooperation idea, a convergence factor is introduced. The optimal individual, the second-best individual, and the third-best individual are combined to update the position of the initial population or the population updated in the previous iteration, so as to obtain the updated population in the current iteration. The formula is: In the formula, The first one obtained based on the gray wolf cooperative idea t+ Gray wolf guide position in 1st iteration; For the first t The iteration of the ... i Each updated individual, in the initial iteration, For the initial population, the first i An initial individual; For the first t+ 1, t The first, second, and third potential movement vectors of the next iteration; For the first t The next iteration Distance vectors between the best, second-best, and third-best individuals; This represents the vector of the first, second, and third control coefficients; These are the first, second, and third oscillation coefficients; A random number between [0, 1]; For control coefficients and oscillation coefficients; For the first t The convergence factor of each iteration is large in the early stage of the iteration, indicating large steps of global exploration, and approaches 0 in the later stage, indicating small steps of fine local exploration. t This represents the current iteration number; For the first t The best, second-best, and third-best individuals in the next iteration; In the formula, The non-linear adjustment index has a value of 1-3. =1: Linear cosine decay =2: Slower decay in the early stage, faster decay in the later stage (recommended). =3: Slower decay in the early stage, faster decay in the later stage; This is the threshold for the number of iterations; This is the initial convergence factor, usually around 2, which makes the early search step size larger and the coverage wider; Generate random numbers in the interval [0,1] ,like Then, the development phase begins, where the initial population or the population updated in the previous iteration is updated to obtain the updated population for the current iteration. The formula is: In the formula, The first generation generated during the development phase t+ The first iteration i A newer individual; For the first t The iteration of the ... i Each updated individual, in the initial iteration, For the initial population, the first i An initial individual; The first one obtained based on the gray wolf cooperative idea t+ Gray wolf guide position in 1st iteration; like Then, the exploration phase begins, where the initial population or the population updated in the previous iteration is updated to obtain the updated population for the current iteration. The formula is: In the formula, The first generation generated for the exploration phase t+ The first iteration i A newer individual; For the first t The iteration of the ... i Each updated individual, in the initial iteration, For the initial population, the first i An initial individual; The first one obtained based on the gray wolf cooperative idea t+ Gray wolf guide position in 1st iteration; For the first i A temporary location for an updated individual; The fitness function; The numbers are uniformly random numbers in the interval [0,1]. For the first t Random individuals in the next iteration; The individual position is randomly scaled using uniformly distributed random numbers in the [0,1] interval to increase search randomness, prevent the offset from being fixed for each update, and maintain population diversity. S26: Repeatedly update the position of the population, and terminate the iterative update of the population when the current iteration reaches the maximum number of iterations or when the fitness value of the best individual in the population updated in the current iteration meets the requirements. S27: Decode the position vector of the best individual in the updated population of the current iteration to obtain the optimal initial network parameters.

[0025] In one alternative implementation, the geothermal scheduling model is initialized using optimal initial parameters, a safety action shielding mechanism is introduced, interactive learning is performed in a geothermal station simulation environment, and the network parameters are fine-tuned using the gradient descent method to obtain the optimal geothermal scheduling model, including: S31: Based on the optimal initial parameters, initialize the geothermal scheduling model to obtain the initialized geothermal scheduling model; S32: Introduce a safety action shielding mechanism in the initialized geothermal scheduling model, and set up a physical model prediction module and a safety action shielding module; S33: Using the initialized geothermal scheduling model, conduct trial and error learning in the geothermal station simulation environment, accumulate training data, and store the training data in the experience playback pool; S34: Randomly collect some training data from the experience replay pool, use gradient descent to fine-tune the network parameters of the initialized geothermal scheduling model, and obtain the optimal geothermal scheduling model when the model converges. In this embodiment, after each interactive data storage, a mini-batch of size 64 is randomly sampled from the experience replay pool, and the network parameters are updated using the Adam optimizer with a learning rate of 1. e -4 With a discount factor of 0.99 and a total training step count of 100,000, the model converges very quickly since the initial parameters have been globally optimized. It usually reaches a stable optimal state in about 5,000 steps, thus obtaining the optimal geothermal scheduling model.

[0026] In one optional implementation, the physical model prediction module is used to predict the key predicted states at the next moment corresponding to the original actions output by the initialized geothermal scheduling model. The key predicted states include predicted downhole fluid level and predicted wellhead pressure. The physical model prediction module is configured with dynamic equations for geothermal well fluid level and pressure, the formulas of which are: In the formula, The dynamic equation for the liquid level in a geothermal well; For a moment t The state; t For time indexing; For a moment t The original action; This is the pressure equation; The predicted downhole fluid level in the key predicted state for the next moment; The predicted wellhead pressure in the key predicted state for the next moment; The security action shielding module is used to make a shielding determination based on the key predicted state at the next moment, and select to execute a security supply guarantee strategy based on the shielding determination result, so as to obtain an executable action after security filtering.

[0027] In one alternative implementation, an initialized geothermal scheduling model is used to perform trial-and-error learning in a geothermal station simulation environment, accumulating training data, and storing the training data in an experience replay pool, including: S331: Collect historical operating data of the geothermal station, preprocess it to obtain preprocessed historical operating data, and write it into the state space of the initialized geothermal scheduling model to obtain the historical state at the corresponding time. S332: Based on historical conditions, use the initialized geothermal scheduling model to run in the geothermal station simulation environment and generate original actions in the action space; S333: Using the physical model prediction module, predict the key prediction state of the next moment corresponding to the original action, and combine it with the external environmental variables directly output by the working condition generator in the geothermal station simulation environment to obtain the full prediction state of the next moment with the same structure as the historical state. S334: Using the safety action shielding module, a shielding determination is made based on the critical predicted state at the next moment. Based on the shielding determination result, a safety supply guarantee strategy is selected to obtain the executable action after safety filtering. The formula is as follows: In the formula, To mask the judgment result, if If the condition is met, the shielding result is deemed unsafe; otherwise, it is considered safe. The predicted downhole fluid level in the key predicted state for the next moment; This serves as a safety warning line for the downhole fluid level. The predicted wellhead pressure in the key predicted state for the next moment; This represents the maximum threshold value for wellhead pressure. For a moment t Executable actions; For a moment t The original action; Pre-defined actions (such as maintaining the actions from the previous moment or executing them at the lowest frequency) are determined by the security supply strategy. t For time indexing; S335: In the geothermal station simulation environment, call the multi-objective reward function to generate the corresponding historical instantaneous comprehensive reward value; S336: Use historical states, executable actions, historical instantaneous comprehensive reward values, and the full predicted state at the next moment as training data, and store the training data in the experience replay pool.

[0028] In one alternative implementation, the optimal geothermal scheduling model is deployed to the field control system of the geothermal station, real-time operating data of the geothermal station is collected, geothermal scheduling is performed, and optimal control commands are output, including: S41: Extract the optimal network parameters of the optimal strategy network in the optimal geothermal scheduling model, convert them into a lightweight format, remove the backpropagation computation graph required for training, and retain only the forward inference computation graph to obtain the lightweight optimal network parameters. S42: Write the lightweight optimal network parameters into the non-volatile memory area of ​​the controller of the field control system of the geothermal station to generate a strategy inference function that can be called in real time. S43: Collect real-time operating data of the geothermal station, preprocess it to obtain preprocessed real-time operating data, and generate the real-time state of the state space that conforms to the optimal geothermal scheduling model. S44: Input the real-time state into the policy inference function to generate real-time actions in the action space; S45: Based on the physical upper and lower limits of the equipment in the geothermal station, the normalized real-time actions are mapped back to the actual control values ​​to generate the optimal control commands, including the actual production well submersible pump frequency value, the actual reinjection pump frequency value, and the actual main pipeline valve opening value. S46: Based on the maximum allowable change in a single cycle, limit the rate of change of the optimal control command to obtain the limited optimal control command, and write it into the programmable logic controller (PLC) register of the field control system to execute geothermal scheduling.

[0029] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus, or computer program products. Therefore, embodiments of the present invention can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of the present invention can take the form of computer program products implemented on one or more computer-usable hardware devices (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0030] The embodiments of the present invention are described with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (apparatus), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0031] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0032] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0033] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. "And / or" indicates that either one or both can be chosen. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes the element.

[0034] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A deep reinforcement learning-based geothermal scheduling method for a geothermal station, characterized in that, The method includes: A geothermal scheduling model is constructed using deep reinforcement learning algorithms. The initial network parameters of the geothermal scheduling model are mapped to the location vectors of individuals, and the improved sugarcane rat optimization algorithm is used to perform a global search in the parameter space to find the optimal initial network parameters. The geothermal scheduling model is initialized using optimal initial parameters, a safety action shielding mechanism is introduced, interactive learning is performed in the geothermal station simulation environment, and the network parameters are fine-tuned using the gradient descent method to obtain the optimal geothermal scheduling model. The optimal geothermal scheduling model is deployed to the field control system of the geothermal station to collect real-time operating data of the geothermal station, perform geothermal scheduling, and output optimal control commands.

2. The deep reinforcement learning geothermal scheduling method for geothermal stations according to claim 1, characterized in that, A geothermal scheduling model is constructed using deep reinforcement learning algorithms, including: Set up state vectors based on indicators from historical operational data; Define the control variables for the equipment at the geothermal station and set the action space; A multi-objective reward function is constructed by considering the overall system's energy efficiency maximization, energy consumption minimization, load deviation minimization, and safety constraints. Based on deep reinforcement learning algorithms, a model structure including a policy network and a valuation network is constructed, and the initial network parameters to be optimized are determined. A geothermal scheduling model is constructed by combining state vectors, action space, multi-objective reward functions, and model structure.

3. The deep reinforcement learning geothermal scheduling method for geothermal stations according to claim 2, characterized in that, The state vector includes normalized geothermal wellhead temperature, wellhead pressure, downhole fluid level, reinjection temperature, ambient temperature, terminal heat load demand, and timestamp. The control variables of the action space include the frequency of the production well submersible pump, the frequency of the reinjection pump, and the opening degree of the main pipeline valve. The initial network parameters to be optimized include the weight matrices and bias vectors of the policy network and the valuation network.

4. The deep reinforcement learning geothermal scheduling method for geothermal stations according to claim 3, characterized in that, The formula for the multi-objective reward function is: In the formula, This refers to the instantaneous total reward value; The instantaneous energy efficiency ratio of the geothermal station; This is the rated maximum energy efficiency. This represents the total power consumption of the circulating pump. Rated power; This refers to the actual heat supply; For user-side load requirements; This is the rated maximum heating capacity; This is a safety penalty item; Let be the weighting coefficients, and let sum to 1.

5. The deep reinforcement learning geothermal scheduling method for geothermal stations according to claim 4, characterized in that, The initial network parameters of the geothermal scheduling model are mapped to individual location vectors. An improved sugarcane rat optimization algorithm is used to perform a global search in the parameter space to find the optimal initial network parameters, including: The initial network parameters of the geothermal scheduling model are flattened by row or column, concatenated into a one-dimensional vector, and encoded as the location vector of each individual. The chaotic sequence is generated using the Logistic mapping and then mapped to the parameter space of individuals in the improved sugarcane rat optimization algorithm to obtain an initial population including several initial individuals. The fitness function is defined as the negative of the cumulative instantaneous comprehensive reward value. Based on the geothermal station simulation environment, the fitness function is used to calculate the fitness value of each individual in the initial population or the updated population of the previous iteration. Based on the fitness value, select the top three individuals, the second best individuals, and the third best individuals in terms of fitness from the initial population or the population updated in the previous iteration. Based on the gray wolf cooperation concept, a convergence factor is introduced. The optimal individual, the second-best individual, and the third-best individual are combined to update the position of the initial population or the population updated in the previous iteration, so as to obtain the updated population in the current iteration. The position update of the population is repeated. The iterative update of the population is terminated when the current iteration reaches the maximum number of iterations or when the fitness value of the best individual in the population updated in the current iteration meets the requirements. The optimal initial network parameters are obtained by decoding the position vector of the best individual in the updated population of the current iteration.

6. The deep reinforcement learning geothermal scheduling method for geothermal stations according to claim 5, characterized in that, The geothermal scheduling model is initialized using optimal initial parameters, a safety action shielding mechanism is introduced, interactive learning is performed in a geothermal station simulation environment, and the network parameters are fine-tuned using the gradient descent method to obtain the optimal geothermal scheduling model, including: Based on the optimal initial parameters, the geothermal scheduling model is initialized to obtain the initialized geothermal scheduling model. A safety action shielding mechanism is introduced into the initial geothermal scheduling model, and a physical model prediction module and a safety action shielding module are set up. Using the initialized geothermal scheduling model, we conduct trial and error learning in the geothermal station simulation environment, accumulate training data, and store the training data in the experience playback pool. Several training data were randomly collected from the experience replay pool, and the network parameters of the initialized geothermal scheduling model were fine-tuned using the gradient descent method. The optimal geothermal scheduling model was obtained when the model converged.

7. The deep reinforcement learning geothermal scheduling method for geothermal stations according to claim 6, characterized in that, The physical model prediction module is used to predict the key prediction state at the next moment corresponding to the original action output by the initialized geothermal scheduling model. The key prediction state includes the predicted downhole liquid level and the predicted wellhead pressure. The security action shielding module is used to make a shielding determination based on the key predicted state at the next moment, and select to execute a security supply guarantee strategy based on the shielding determination result, so as to obtain an executable action after security filtering.

8. The deep reinforcement learning geothermal scheduling method for geothermal stations according to claim 7, characterized in that, Using an initialized geothermal scheduling model, trial and error learning is conducted in a geothermal station simulation environment to accumulate training data. The training data is then stored in an experience replay pool, including: Historical operating data of the geothermal station is collected and preprocessed to obtain preprocessed historical operating data, which is then written into the state space of the initialized geothermal scheduling model to obtain the historical state at the corresponding time. Based on historical conditions, the initialized geothermal scheduling model is run in the geothermal station simulation environment to generate original actions in the action space. Using the physical model prediction module, the key prediction state of the next moment corresponding to the original action is predicted, and combined with the external environmental variables directly output by the working condition generator in the geothermal station simulation environment, the full prediction state of the next moment with the same structure as the historical state is obtained. The security action shielding module is used to make a shielding determination based on the critical predicted state of the next moment, and based on the shielding determination result, a security supply guarantee strategy is selected to be executed to obtain the executable actions after security filtering. In the geothermal station simulation environment, a multi-objective reward function is invoked to generate the corresponding historical instantaneous comprehensive reward value; The historical state, executable actions, historical instantaneous comprehensive reward value, and the full predicted state at the next moment are used as training data, and the training data is stored in the experience replay pool.

9. The deep reinforcement learning geothermal scheduling method for geothermal stations according to claim 8, characterized in that, The optimal geothermal scheduling model is deployed to the field control system of the geothermal station, real-time operating data of the geothermal station is collected, geothermal scheduling is performed, and optimal control commands are output, including: The optimal network parameters of the optimal strategy network in the optimal geothermal scheduling model are extracted, converted into a lightweight format, and the backpropagation computation graph required for training is removed, leaving only the forward inference computation graph, to obtain the lightweight optimal network parameters. Lightweight optimal network parameters are written into the non-volatile memory area of ​​the controller of the field control system of the geothermal station to generate a strategy inference function that can be called in real time. Collect real-time operating data from the geothermal station, preprocess it to obtain preprocessed real-time operating data, and generate the real-time state of the state space that conforms to the optimal geothermal scheduling model. The real-time state is input into the policy inference function to generate real-time actions in the action space; Based on the physical upper and lower limits of the geothermal station equipment, the normalized real-time actions are mapped back to the actual control values ​​to generate optimal control commands, including the actual production well submersible pump frequency value, the actual reinjection pump frequency value, and the actual main pipeline valve opening value. Based on the maximum allowable change in a single cycle, the rate of change of the optimal control command is limited to obtain the limited optimal control command, which is then written into the PLC register of the field control system to execute geothermal scheduling.