Carbon transaction method and system, computer equipment and storage medium

Abstract

The invention provides a carbon transaction method and system, computer equipment and a storage medium, and belongs to the crossing field of artificial intelligence and energy management, and the method comprises the steps: constructing a long-period carbon transaction model which fits the market reality, providing a delay reward reinforcement learning algorithm DR-RLCT, fusing randomized return decomposition with a depth deterministic strategy gradient, and obtaining a carbon transaction model; the total cost at the end of the cycle is intelligently decomposed and redistributed into the instant agency reward of each step in the performance cycle by training the reward network, so that a delayed punishment signal is converted into a dense strategy guidance signal, and the credit distribution problem in the long-cycle decision is effectively solved; and finally, optimizing the strategy network by utilizing the agency reward, so that the optimal transaction action can be output. Compared with a traditional method, the method has the advantages that stable positive earnings can be achieved, penalty defaulting can be completely avoided, and a reliable intelligent decision-making scheme is provided for enterprises to achieve low-cost compliance operation in the carbon market.

Description

Technical Field

[0001] This invention belongs to the interdisciplinary field of artificial intelligence and energy management, and specifically relates to a carbon trading method, system, computer equipment, and storage medium. Background Technology

[0002] To address global climate change, countries have successively established carbon emission trading markets, gradually forming a carbon emission reduction policy system covering multiple sectors such as energy, industry, and transportation. These markets generally employ a cap-and-trade mechanism, whereby regulatory agencies set national or regional carbon emission caps and compliance periods, and issue initial carbon emission allowances to regulated companies through allocation or auctions. During the compliance period, companies are allowed to freely trade allowances based on their own carbon emissions and production needs. At the end of the compliance period, companies must surrender allowances equivalent to their actual cumulative emissions to fulfill their compliance obligations; otherwise, they will be subject to substantial economic penalties. This system, on the one hand, provides companies with flexible carbon allowance allocation space, allowing them to dynamically adjust their trading strategies based on market carbon price fluctuations and production emission plans; on the other hand, it introduces typical intertemporal decision-making characteristics, causing current trading behavior to have a delayed impact on the compliance results and economic costs at the end of the period.

[0003] Existing research on carbon trading largely focuses on energy management and carbon emission control, viewing carbon trading as a constraint to control excessive carbon emissions, while research on the operational mechanisms of the carbon trading market itself is relatively insufficient. Many carbon trading models are overly simplistic, failing to accurately reflect the price fluctuations and trading rules of the real carbon trading market. More importantly, these studies generally ignore the time-scale flexibility brought about by the carbon emission compliance cycle, leading companies to typically purchase allowances only for the current day's carbon emissions, failing to fully utilize potentially lower carbon prices in the future. The carbon trading models proposed in the literature model carbon prices as a piecewise linear function that increases with carbon emissions, but this only reflects the supply and demand balance of the carbon market, not the daily price fluctuations that carbon trading strategies can utilize. Wang et al. introduced carbon trading into their energy dispatch model, setting fixed daily carbon emission allowances and adjusting carbon trading strategies based on the actual daily carbon emissions. This method is essentially an immediate adjustment of carbon costs, lacking the flexibility and strategic depth of cross-day allowance optimization within the compliance cycle. Jiang et al. introduced the bidding and settlement process of the carbon trading market into their model, but it is also limited to daily trading. Without utilizing carbon emission compliance cycles, the flexibility of carbon trading will be severely limited, making it impossible to maintain low emission costs.

[0004] In studies considering carbon emission compliance cycles, the uncertainty of future carbon prices has become a core challenge. Li et al. set carbon prices as a fixed parameter, ignoring their uncertainty, thus failing to reflect price fluctuations in the actual carbon trading market. Deng et al. used multi-scenario future prices in a two-stage stochastic optimization. Li et al., under uncertainty, made carbon trading decisions by predicting future prices under a single scenario. However, such price prediction-dependent strategies may have two hidden dangers: first, speculative operations driven by price prediction, i.e., concentrated buying when low prices are predicted and concentrated selling when high prices are predicted, making trading highly sensitive to small prediction deviations and inducing strategy fluctuations; second, excessive waiting based on low price expectations, i.e., delaying buying when the model judges that lower prices may occur in the future, thus missing relatively favorable trading opportunities.

[0005] Both scenarios amplify the instability and risk of decision-making, potentially misleading companies into making incorrect decisions in an environment of price uncertainty. This leads to a fundamental problem: reinforcement learning agents are unable to effectively allocate credit over time, making it difficult to learn high-quality trading strategies that can mitigate default risks. This results in a triple dilemma for companies—difficult compliance, high costs, and decision-making blindness—leading to a vicious cycle of high volatility, low efficiency, and poor fairness in the carbon trading market. Summary of the Invention

[0006] To address the problem that reinforcement learning agents struggle to learn high-quality trading strategies that can mitigate default risks in the carbon quota trading market, this invention provides a carbon trading method, system, computer equipment, and storage medium.

[0007] To achieve the above objectives, the present invention provides a carbon trading method, comprising: A carbon quota trading model for the carbon trading market is constructed, and carbon emission status data and carbon emission compliance cycle endpoints are collected from the carbon trading market. The carbon emission status data includes the current carbon quota price in the target market, the current initial carbon emission status of the target controlled enterprises, and the carbon quota inventory status.

[0008] Based on the current carbon emission status data, the agent uses a pre-defined trading strategy network to cyclically utilize the Deep Deterministic Strategy Gradient Algorithm (DDPG) to output the carbon allowance trading action for the current time step. The carbon allowance trading model uses the carbon allowance trading action and the current carbon emission status data to update the carbon emission status data for the next time step. When the end of the compliance period is reached, the cyclical update of the carbon emission status data stops. The agent integrates the carbon emission status data, trading actions, and subsequences of the corresponding carbon emission status data for the next time step at each time step to form a complete interaction trajectory chain. Based on the complete interaction trajectory chain, the carbon allowance trading model calculates the total economic cost of carbon trading within the compliance period.

[0009] By using a pre-set reward regression network to randomly collect subsequence samples from the complete interaction trajectory chain, the cumulative transaction cost and total economic cost within the corresponding period of the subsequence are decomposed into agent rewards for each time step. Based on the agent rewards, the agent learns the long-term value of carbon quota trading actions at each time step, and adjusts the gradient parameters of the pre-set trading strategy network along the gradient direction that maximizes the long-term value, thus obtaining the delayed reward reinforcement learning carbon trading model DR-RLCT for carbon trading.

[0010] Preferably, the carbon emission status data of the target controlled enterprise is input into the Delayed Reward Reinforcement Learning Carbon Trading Model (DR-RLCT), which outputs the optimal carbon quota trading action at the current time step to guide the enterprise to complete the carbon trading.

[0011] Preferably, the reward regression network is trained using the Randomized Reward Decomposition (RRD) method, which specifically includes: pre-setting the reward regression network to randomly sample subsequences from the complete interaction trajectory chain and generating a proxy training reward for each subsequence; with the goal of minimizing the difference between the proxy training reward and the total economic cost, the parameters of the reward regression network are iteratively updated using the RRD method to finally obtain the trained reward regression network.

[0012] Preferably, the Deep Deterministic Policy Gradient Algorithm (DDPG) includes a Critic network and an Actor network.

[0013] Preferably, through the Critic network, the agent learns the long-term value of carbon allowance trading actions at each time step based on agent rewards; through the Actor network, the agent adjusts the gradient parameters of the preset trading strategy network along the gradient direction that maximizes the long-term value, thereby obtaining the Delayed Reward Reinforcement Learning Carbon Trading Model DR-RLCT for carbon trading.

[0014] Preferably, the total economic cost includes: the cumulative net expenditure on carbon emission allowance trading during the compliance period and the penalty for default corresponding to the carbon emission allowance shortfall at the end of the compliance period; the long-term value is the sum of the agent reward at the current time step and the estimated value of the state at the next time step.

[0015] Preferably, the initial state includes: the remaining time step of the carbon emission compliance cycle, the amount of carbon emission allowances held, the amount of carbon emissions, and the real-time carbon price; the carbon allowance trading action is a continuous value, with a positive value indicating the purchase of carbon emission allowances and a negative value indicating the sale of carbon emission allowances.

[0016] The present invention also provides a carbon trading system, comprising: A data acquisition model is used to construct a carbon quota trading model for the carbon trading market, and to collect carbon emission status data and carbon emission compliance cycle endpoints for the carbon trading market. The carbon emission status data includes the current carbon quota price in the target market, the current initial carbon emission status of the target controlled enterprises, and the carbon quota inventory status.

[0017] The reward network training module is used by the agent to output the carbon allowance trading action for the current time step based on the current carbon emission status data and the preset trading strategy network using the Deep Deterministic Policy Gradient Algorithm (DDPG). The carbon allowance trading model uses the carbon allowance trading action and the current carbon emission status data to update the carbon emission status data for the next time step. When the end of the compliance period is reached, the cyclical update of the carbon emission status data stops. The agent integrates the carbon emission status data, trading action, and the subsequence of the corresponding carbon emission status data for the next time step at each time step to form a complete interaction trajectory chain. Based on the complete interaction trajectory chain, the carbon allowance trading model calculates the total economic cost of carbon trading within the compliance period.

[0018] The decision network training module is used to randomly collect subsequence samples from the complete interaction trajectory chain using a pre-set reward regression network. The cumulative transaction cost and total economic cost within the corresponding period of the subsequence are decomposed into agent rewards for each time step. Based on the agent rewards, the agent learns the long-term value of carbon quota trading actions at each time step and adjusts the gradient parameters of the pre-set trading strategy network along the gradient direction that maximizes the long-term value, thus obtaining the delayed reward reinforcement learning carbon trading model DR-RLCT for carbon trading.

[0019] The present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement any of the steps in the carbon trading method.

[0020] The present invention also provides a computer-readable storage medium storing a computer program that, when loaded by a processor, can execute any of the steps in the carbon trading method.

[0021] The carbon trading method provided by this invention has the following beneficial effects: In the carbon quota trading market, the extremely lagging penalty signal of the total economic cost of carbon trading within the compliance period is intelligently decomposed and redistributed as immediate agent rewards for each step within the compliance period. This fundamentally solves the credit allocation problem in long-cycle decision-making, making reinforcement learning feasible and efficient in such scenarios. This method does not rely on the prediction of future carbon prices, but learns the optimal strategy directly through interaction with the environment, effectively avoiding decision fluctuations and speculative risks caused by prediction errors, and exhibiting stronger stability in a market environment with uncertain prices. The constructed carbon trading model explicitly depicts the real compliance cycle mechanism, transaction costs, and settlement rules, and has broad application potential. It solves the technical problem that reinforcement learning agents struggle to learn high-quality trading strategies that can avoid default risks, promoting the efficient and virtuous cycle development of enterprises and the carbon trading market. Attached Figure Description

[0022] To more clearly illustrate the embodiments and design schemes of the present invention, the accompanying drawings required for this embodiment will be briefly described below. The drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a flowchart of a carbon trading method according to an embodiment of the present invention; Figure 2 This is a diagram showing the daily wind power generation of an embodiment of the present invention; Figure 3 This is a daily power demand diagram according to an embodiment of the present invention; Figure 4 This is a daily carbon price chart according to an embodiment of the present invention; Figure 5 This is a diagram showing the penalty and reward for each performance cycle during the training phase of this invention. Figure 6 This is a diagram illustrating the penalty for breach of contract during each performance cycle in the training phase of this invention. Figure 7 This is a diagram showing the penalties and rewards for each performance cycle during the testing phase of this invention. Figure 8 This is a diagram showing the penalty for breach of contract during each performance period in the testing phase of this invention. Detailed Implementation

[0024] To enable those skilled in the art to better understand and implement the technical solutions of the present invention, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. The following embodiments are only used to more clearly illustrate the technical solutions of the present invention and should not be construed as limiting the scope of protection of the present invention.

[0025] This invention provides a carbon trading method, specifically as follows: Figure 1 As shown, it includes: S1. A carbon quota trading model simulating the carbon trading market, collecting carbon emission status data and the end point of the carbon emission compliance cycle in the carbon trading market; the carbon emission status data includes the carbon quota price of the target market at the current moment, the initial carbon emission status of the target controlled enterprise at the current moment, and the carbon quota inventory status.

[0026] Reinforcement learning has a natural advantage in dealing with uncertain environments, as it can learn optimal long-term policies gradually through continuous interaction with the environment. However, the delayed settlement caused by the performance cycle will cause a significant lag in reward signals, resulting in a significant decrease in the learning efficiency of traditional reinforcement learning algorithms in this scenario, making it difficult to learn high-quality policies. Existing research has proposed two types of methods to address the delayed reward problem: reward shaping and reward decomposition. The former guides the agent to explore effective paths and accelerates policy convergence by introducing additional prior information (such as shaping based on the latent function) into the reward function; however, if the policy invariance condition is not met, it may introduce bias and change the optimal policy of the original task. The latter improves credit allocation and alleviates the delayed reward problem by distributing the overall reward along the time dimension to each state-action pair, establishing an attribution relationship between decisions and long-term results; however, traditional methods need to calculate the complete trajectory step by step in long-span tasks, resulting in high computational complexity and limited scalability. Among them, stochastic reward decomposition shows good potential in alleviating delayed rewards by replacing full trajectory enumeration with random sampling of trajectory fragments, effectively reducing the computational overhead in long-term tasks. However, it has not been systematically studied in carbon trading scenarios that simultaneously involve price fluctuations, compliance cycle constraints, and long-term delayed feedback.

[0027] This invention considers a carbon allowance trading scenario operating under a "cap-and-trade" mechanism, and constructs a general long-term carbon allowance trading model with a carbon trading market compliance cycle mechanism. System participants consume energy from different sources at each time step t, generating corresponding carbon emissions. During the compliance cycle, participants can buy or sell carbon allowances in the carbon trading market to cope with load fluctuations and the uncertainty of carbon emissions. When the compliance cycle ends, the regulatory body will verify the participant's cumulative carbon emissions against their total carbon allowance holdings. If there is a allowance shortfall, a penalty will be paid based on the excess emissions.

[0028] The system's carbon emissions are categorized into two types: direct carbon emissions, which are emissions generated during the system's own power generation; and indirect carbon emissions, which are emissions inherent in the use of electricity from the external power grid. For ease of unified modeling, this invention converts all energy consumption into equivalent electrical energy and calculates carbon emissions using corresponding unit emission factors. Thus, the total carbon emissions of participants at time step t are calculated. It can be defined as:

[0029] (1) in, and These represent the carbon emission factors per unit of direct energy and indirect energy (unit: kgCO2 / kWh), respectively. and These represent the equivalent electrical energy consumption of direct and indirect energy sources at time step t (unit: kWh).

[0030] During the compliance period, participants will dynamically formulate carbon allowance trading strategies, taking into account factors such as carbon price fluctuations, uncertainties in production / operational loads, and current allowance holdings. At the end of the compliance period, a participant's carbon allowance deficit must be greater than or equal to zero; otherwise, it will be considered a default and subject to an economic penalty proportional to the deficit. The carbon allowance deficit at the end of the compliance period is defined as follows:

[0031] (2) in, This represents the initial carbon allowance. The quota trading volume at time step t (positive value for buying, negative value for selling). This refers to the length of the contract fulfillment period.

[0032] This model is applicable to any industry producer included in the carbon trading market management scope. The present invention will be illustrated in the experimental section using a data center as an example.

[0033] In a carbon allowance trading system with a carbon emission compliance cycle, the decision objective is to dynamically adjust the carbon allowance trading strategy throughout the entire compliance cycle to minimize total economic costs and avoid high penalties. The optimization objective function can be expressed as:

[0034] (3) The first item is the cumulative net expenditure on carbon allowance trading during the compliance period. The first term represents the price per unit of carbon allowance in the carbon trading market on day t; the second term represents the penalty payable at the end of the compliance period due to a shortfall in carbon allowances, where the penalty per unit is... Prices are typically significantly higher than the average daily trading price in the carbon market, therefore defaults will have a substantial impact on total costs. (Symbol) This represents the non-negative portion of the carbon quota deficit; it is taken when the deficit is positive and zero otherwise, and is used to quantify the degree of default. To ensure that trading activities comply with market rules, the feasible region of trading activities is defined as follows:

[0035] (4) in, This represents the effective carbon allowances held by market participants on day t. Indicates the maximum transaction volume. When... At that time, the lower world Ensure that the carbon allowances sold do not exceed the current holdings; when At that time, the upper limit ensures that the daily purchase volume does not exceed the preset maximum transaction size.

[0036] S2. Based on the current carbon emission status data, the agent uses a preset trading strategy network to cyclically utilize the Deep Deterministic Strategy Gradient Algorithm (DDPG) to output the carbon allowance trading action for the current time step. The carbon allowance trading model uses the carbon allowance trading action and the current carbon emission status data to update the carbon emission status data for the next time step. When the end of the compliance period is reached, the cyclical update of the carbon emission status data stops. The agent integrates the carbon emission status data, trading actions, and subsequences of the corresponding carbon emission status data for the next time step at each time step to form a complete interaction trajectory chain. Based on the complete interaction trajectory chain, the carbon allowance trading model calculates the total economic cost of carbon trading within the compliance period.

[0037] To solve the long-term carbon allowance trading optimization problem considering the carbon emission compliance cycle constructed in this invention, this embodiment models it as a Markov Decision Process (MDP), providing a framework for subsequent reinforcement learning algorithms. An MDP is typically defined as a quintuple. ,in Representing the state space, Represents the action space, Represents the reward function, Indicates state transition, This represents the discount factor. In this embodiment of the invention, the carbon quota trading system is used as the environment of the MDP. The interaction process between the agent and the environment is as follows: At each time step t, the agent observes the current state of the environment, executes a trading decision applied to the environment, and receives an immediate reward from the environment. Subsequently, the environment transitions to the next state according to the state transition, and this state will be observed by the agent at the next time step t+1. The components of the MDP are defined in detail below:

[0038] State space: At time step t, the environmental state vector observed by the agent... It consists of the following key variables: (5) in, This indicates the remaining time steps until the end of the performance period. This indicates the current carbon allowance holdings. This indicates the current carbon emissions. This indicates the current price of carbon allowances in the carbon trading market.

[0039] Action space: At time step t, the agent's action space This represents the volume of carbon allowance trading (positive for buying, negative for selling), and its feasible region. It is given by formula (4).

[0040] Reward function: The reward function is designed based on the objective function of minimizing the total transaction cost defined in formula (3). Since the optimization objective of RL is to maximize the expected cumulative reward, this embodiment of the invention transforms the original objective function of minimizing carbon transaction costs into the following reward function:

[0041] (6) State transition: State transitions reflect the dynamic characteristics of the environment. In this invention, the state... They are divided into two categories: internal states and external states. Internal states are controllable and mainly include: , For a controllable state, performing the corresponding action will result in a definite state transition, which can be formalized as the following process:

[0042] (7) (8) External conditions are uncontrollable, mainly including , Uncontrollable states require real-time state transitions based on the dataset. Furthermore, the overall state transition mechanism follows the Markov property assumption, meaning the next state depends only on the current state and the current action choice, and is independent of previous historical states and actions.

[0043] To address the long-term carbon allowance trading optimization problem with carbon emission compliance cycles proposed in this invention, and to cope with the challenge of delayed rewards caused by the extreme lag in default penalties, this invention proposes a DR-RLCT algorithm. This algorithm organically combines RRD with Deep Deterministic Policy Gradient (DDPG), achieving both effective redistribution of delayed rewards and policy optimization in a continuous action space, thereby improving the learning efficiency and execution performance in carbon allowance trading decisions.

[0044] The pseudocode for the DR-RLCT algorithm is shown in Algorithm 1. The algorithm is trained on a per-episode basis (i.e., a complete carbon trading cycle). Within each episode, the agent continuously interacts with the environment based on the current strategy and receives feedback from the environment. After completing the interaction of each episode, the entire episode's interaction data is stored in the ReplayBuffer for subsequent network training. Lines 8 and 9 of the pseudocode describe the core of the algorithm: reward redistribution and policy optimization.

[0045] Reward redistribution, based on the RRD algorithm, decomposes the total reward (the sum of transaction costs and default penalties) of a complete performance cycle into immediate agent rewards for each step by training a reward regression network. This provides a stable feedback signal with higher relevance to the final goal for reinforcement learning strategy optimization. Strategy optimization, based on the DDPG algorithm, utilizes the agent rewards generated by reward redistribution to train the policy network and value network in a continuous action space, achieving dynamic optimization of the trading strategy. The alternating updates and bidirectional promotion of reward redistribution and strategy optimization effectively overcome the delayed reward problem caused by carbon trading performance cycle settlement, thereby accelerating algorithm learning and improving strategy performance.

[0046] S3. Using a pre-set reward regression network, sub-sequence samples are randomly collected from the complete interaction trajectory chain. The cumulative transaction cost and total economic cost within the corresponding period of the sub-sequence are decomposed into agent rewards for each time step. Based on the agent rewards, the agent learns the long-term value of carbon quota trading actions at each time step and adjusts the gradient parameters of the pre-set trading strategy network along the gradient direction that maximizes the long-term value to obtain the delayed reward reinforcement learning carbon trading model DR-RLCT for carbon trading.

[0047] The agent randomly collects subsequence samples from the experience replay buffer and calls the trained reward regression network to generate a proxy reward for each sample. Based on the proxy reward, the agent's Critic network (value evaluator) learns the long-term value of the corresponding action in each state (i.e., the sum of the current proxy reward and the estimated value of the state in the next time step). The agent's Actor network (policy network) adjusts its internal parameters along the gradient direction that maximizes the long-term value according to the long-term value output by the Critic network, optimizes the trading policy network, and obtains the trained decision network model.

[0048] For the carbon quota trading optimization problem addressed in this invention, the agent can only receive a penalty signal regarding whether it has defaulted at the end of each compliance period. This signal is crucial for evaluating the quality of the trading strategy throughout the entire period. However, due to its significant lag, it cannot provide timely and effective guidance for strategy learning. To address this typical delayed reward characteristic, a reward redistribution design based on the RRD algorithm is implemented, corresponding to line 5 of the overall pseudocode for DR-RLCT (Algorithm 1). The detailed process is described in Algorithm 2.

[0049] The core idea of ​​reward redistribution is to use training rewards to regress to the network. The total cost incurred by the agent during the complete carbon trading compliance cycle (including carbon trading costs within the compliance cycle and the final default penalty) is decomposed along the time dimension into carbon trading costs corresponding to each trading decision point, thereby establishing an attribution mapping relationship between each decision point and the final default penalty in the time series. By reallocating the delayed penalty signal into an instant agent reward signal at the time step level, the reward reallocation mechanism provides immediate feedback for DDPG policy learning, significantly improving the policy learning efficiency in delayed reward scenarios. The mathematical form of this process is as follows:

[0050] Given a complete fulfillment cycle interaction trajectory chain , This indicates the length of the contract performance period. Let... The environmental reward signal is the carbon trading cost at the t-th trading point within the compliance period; at the end of the period, if the value described by formula (2) is... but Attached penalty for breach of contract; the agent reward signal provided by RRD is represented as ; Let the total carbon trading costs and default penalties for this trajectory be:

[0051] (9) The theoretical basis of reward redistribution stems from reward decomposition. As the sole environmental supervision for reward modeling, traditional reward decomposition methods for a trajectory of length T require enumerating all state-action pairs in the trajectory, calculating the agent reward for each step, and summing them over time to approximate the total reward of the trajectory. Strict matching. While this approach can theoretically achieve accurate contribution attribution, its computational complexity is O(T), making it extremely expensive in scenarios with long task time spans (e.g., long carbon emission compliance cycles). To address this, RRD introduces Monte Carlo sampling: instead of enumerating all time steps in the trajectory, it randomly selects a subset of time steps of length K < T from the complete trajectory and uses the surrogate rewards on these subsets to approximate the total reward for the entire trajectory. This reduces the computational complexity from O(T) to O(K), significantly decreasing computational overhead for long-duration tasks while maintaining an unbiased estimate of the overall reward distribution of the trajectory. RRD trains its reward model by minimizing the following loss function:

[0052] (10) in, This is a subset of time-step indices for random sampling, with a fixed length of K (less than T). The sampling distribution is unbiased (uniform sampling is used in this invention).

[0053] The strategy optimization, based on the immediate reward signal generated by reward redistribution, employs the DDPG algorithm to optimize the carbon allowance trading strategy (see line 6 of the pseudocode for Algorithm 1). DDPG is an off-policy reinforcement learning algorithm based on an Actor-Critic architecture, specifically designed to optimize strategies in a continuous action space. In the carbon allowance trading decision-making problem studied in this invention, the action space is typically continuous. DDPG utilizes joint training of the Actor and Critic networks to jointly drive the optimization of the trading strategy and the learning of the value function.

[0054] The primary task of the Actor network is to output a deterministic trading action based on the current state of the environment. Given a state, the Actor network outputs a continuous action through a function, where represents the parameters of the Actor network. The optimization objective of the Actor network is to maximize the return of the environment, that is, to ensure that the selected trading action minimizes the sum of carbon trading costs and default penalties within the compliance period. To achieve this objective, the Actor network needs to utilize the Q-value information provided by the Critic network, using the Q-value to guide action selection. Specifically, the optimization objective of the Actor network is to update its policy along the gradient direction of the Q-value provided by the Critic network, as shown in the following formula:

[0055] (11) in, Different random behavior strategies The generated state distribution It is the gradient of the Critic network. This is the gradient of the Actor network. Through this optimization, the Actor network can select the action that maximizes the Q-value (i.e., the expected future reward) based on the feedback from the Critic network.

[0056] The task of the Critic network is to evaluate the quality of the output actions of the Actor network. It generates a Q-value for each state-action pair (s, a). Let represent the expected future return after taking action a in state s. The optimization objective of the Critic network is to minimize the difference between its predicted value and the target value calculated based on the Bellman equation. The Critic network is trained by minimizing the following loss function:

[0057] (12) in, The target Q value is calculated as follows: (13) By minimizing the loss function The Critic network can efficiently update the Q-value, enabling it to accurately estimate the future rewards of state-action pairs.

[0058] Furthermore, to improve training stability and avoid drastic fluctuations in Q-value estimation, DDPG introduces a target network and an experience replay mechanism. DDPG creates a target network and a critique network for each of the Actor and Critics networks. The parameters of the target network are synchronized with the parameters of the main network through a soft update method. The update formula for the target network is as follows:

[0059] (14) (15) in, τ is a small hyperparameter with a range of 0 < τ ≪ 1, ensuring smooth updates to the target network. By using the target network, DDPG can stably update the Q-value, thus avoiding over-updates and training instability. The experience replay mechanism is used to store the agent's experience interacting with the environment and randomly select mini-batch samples from it for training. In this way, DDPG can break the correlation between samples, enhance training stability, and prevent the policy from getting trapped in local optima during training. The policy optimization pseudocode is shown in Algorithm 3.

[0060] The primary task of an Actor network is to output a deterministic transaction action based on the state fed back from the current environment. Given a state... Actor networks use functions Output a continuous action ,in These are the parameters of the Actor network. The optimization objective of the Actor network is to maximize environmental rewards, that is, to ensure that the chosen trading actions minimize the sum of carbon trading costs and default penalties within the compliance period. To achieve this objective, the Actor network needs to utilize the Q-value information provided by the Critic network to guide action selection. The Critic network's task is to evaluate the quality of the Actor network's output actions. It evaluates each state-action pair... Generate a Q value , indicating the state Take action below The expected return of the future. The optimization objective of the Critic network is to minimize the difference between its predicted value and the target value calculated based on the Bellman equation. Furthermore, to improve training stability and avoid drastic fluctuations in Q-value estimation, DDPG introduces a target network and an experience replay mechanism. By using the target network, DDPG can stably update the Q-value, thus avoiding over-updating and training instability. The experience replay mechanism is used to store the agent's experience interacting with the environment and randomly sample mini-batches from it for training. In this way, DDPG can break the correlation between samples, enhance training stability, and prevent the policy from getting trapped in local optima during training.

[0061] To verify the effectiveness of the general carbon quota trading model and the DR-RLCT algorithm, this invention conducts a simulation study using a data center as a case study. The embodiments of this invention construct a sophisticated data center carbon trading simulation environment based on real workloads and carbon market volatility characteristics. The proposed DR-RLCT algorithm is then compared and analyzed with the traditional reinforcement learning method DDPG to evaluate its strategy optimization capability and compliance performance under delay penalty signal scenarios.

[0062] This invention considers data center power supply from the public power grid (UT) and local wind turbines (WT). The system prioritizes the use of near-zero carbon clean energy provided by WT. Due to the intermittency and volatility of wind power, when WT generation is insufficient to meet the current power demand of the data center, the system will obtain high-carbon power from UT. Therefore, in formula (1) Wind power generation per time step t , Public grid electricity usage at time step t . Calculation using polynomial functions established in the literature:

[0063] (16) in, To cut into wind speed, To cut off the wind speed, The rated wind speed is given, and c1, c2, and c3 are constants. This invention is set... , and The speeds are 2.5 m / s, 12.5 m / s, and 25 m / s respectively; c1, c2, and c3 are set to 0.105, 0.07, and 0.25 respectively.

[0064] The total energy consumption of the data center at time step t is estimated using a linear energy consumption model commonly used in the literature: (17) in, This represents the server's average CPU utilization. and These represent the idle power and full load power of a single server, respectively. This refers to the number of servers. In this invention, the invention sets... , , , and The respective power ratings are 0.56kW, 0.80kW, 0.97kW, 0.966kW, and 5000kW.

[0065] Set the direct carbon emission factor in formula (1) The carbon emission factor of the power grid is 10gCO2 / kWh. The initial carbon quota is 647 gCO2 / kWh. This embodiment of the invention sets the initial carbon quota in formula (2). This is 70% of the estimated historical carbon emissions. Set the values ​​in formula (3) as follows: The price is 10 times the unit price of carbon allowances within the compliance period. In this embodiment of the invention, a compliance period is set at 20 trading days, and the carbon allowance trading period is one day.

[0066] This invention collects wind speed data, CPU utilization data, and carbon price data from the State Grid Corporation of China, Alibaba Cloud's cluster tracking program, and the Shanghai Environment and Energy Exchange. Each data set contains 2720 records. This invention divides the dataset into 136 performance cycles, each with a length of 20 days, approximately one month (excluding non-trading days). 80 performance cycles are used as the training set, containing 1600 data points; the remaining 56 performance cycles are used as the test set, containing 1120 data points. Using the computational models defined in equations (16) and (17), daily wind power and energy consumption data sequences for the data center are obtained. Figure 2 , Figure 3 and Figure 4 The statistical results of wind power data, data center energy consumption data, and carbon price data used in this invention are presented respectively.

[0067] To verify the effectiveness of the proposed algorithm, this invention selects the DDPG algorithm as the benchmark and trains the DDPG algorithm and the DR-RLCT algorithm designed in this invention under the same dataset and parameter configuration. Then, the performance is evaluated on the test set. Figure 5 and Figure 6 The trends of total reward (episodic reward) and non-compliance penalty (Non-compliance penalty) for the two algorithms during each compliance cycle in the training phase are presented. In carbon trading scenarios with compliance cycles, maintaining the total reward near zero is quite difficult due to the high Non-compliance penalty triggered by exceeding carbon emission limits at the end of the cycle. Experimental results show that the DR-RLCT algorithm learns the optimal trading strategy faster than the DDPG algorithm. The DR-RLCT algorithm converges after approximately the 19th compliance cycle, with the total reward stabilizing near zero and almost completely eliminating the Non-compliance penalty; while the DDPG algorithm fails to converge until the end of training, with both total reward and Non-compliance penalty fluctuating wildly at high levels. This is because the delayed default penalty at the end of the cycle makes it difficult for the DDPG algorithm to effectively establish a causal relationship between early-stage trading behavior and the final penalty, thus making it difficult to learn good trading strategies to avoid default penalties during long performance cycles. In contrast, the DR-RLCT algorithm uses a return decomposition mechanism to map the delayed default penalty into an immediately perceptible agent reward in advance, enabling the strategy to identify and avoid potential default risks early in the training process, forming a more efficient and stable trading strategy.

[0068] Figure 7 and Figure 8 The performance of the two algorithms on the test set after training is presented. The results show that the average total revenue of the DR-RLCT algorithm in each performance cycle is 1.16 × 10⁻⁶. 4 The mean penalty for non-performance is 0, indicating that it completely avoids default penalties and can still make stable trading decisions on unlearned data. In contrast, the DDPG algorithm's mean total return per performance period is -2.99 × 10⁻⁶. 5 The average penalty for non-compliance is 3.35 × 10⁻⁶. 5 Furthermore, the volatility is significant. This is highly consistent with the poor performance during the training phase, reflecting that the trading strategy has hardly improved under the delay penalty scenario, has failed to learn how to deal with the decision-making challenges brought about by the delay reward, and is even less able to meet the compliance requirements during the fulfillment period.

[0069] This invention addresses the problem of delayed rewards in carbon trading scenarios with emission compliance cycles, a problem stemming from the obstacle of end-of-cycle penalty mechanisms to long-term credit allocation. This invention constructs a market-compliant carbon trading model, encompassing compliance cycle rules, transaction costs, settlement processes, and carbon price volatility characteristics, and proposes the DR-RLCT method. This method combines Randomized Reward Decomposition (RRD) with Deep Deterministic Policy Gradient (DDPG) to redistribute intra-cycle transaction costs and end-of-cycle penalties into time-series proxy rewards. In a data center case study, DR-RLCT significantly outperforms DDPG on unseen compliance cycles, achieving stable positive returns and completely avoiding penalties, while DDPG remains in a loss-making state with high penalties. The results validate the effectiveness of reward redistribution in delayed reward scenarios and demonstrate the robustness and scalability of DR-RLCT. Future research will further explore more complex market dynamics, multi-agent environments, and adaptive policy migration to promote the wider application of this method in sustainable energy markets.

[0070] Based on the same inventive concept, this invention also provides a carbon trading system, comprising: A data acquisition model is used to construct a carbon quota trading model for the carbon trading market, and to collect carbon emission status data and carbon emission compliance cycle endpoints for the carbon trading market. The carbon emission status data includes the current carbon quota price in the target market, the current initial carbon emission status of the target controlled enterprises, and the carbon quota inventory status.

[0071] The reward network training module is used by the agent to output the carbon allowance trading action for the current time step based on the current carbon emission status data and the preset trading strategy network using the Deep Deterministic Policy Gradient Algorithm (DDPG). The carbon allowance trading model uses the carbon allowance trading action and the current carbon emission status data to update the carbon emission status data for the next time step. When the end of the compliance period is reached, the cyclical update of the carbon emission status data stops. The agent integrates the carbon emission status data, trading action, and the subsequence of the corresponding carbon emission status data for the next time step at each time step to form a complete interaction trajectory chain. Based on the complete interaction trajectory chain, the carbon allowance trading model calculates the total economic cost of carbon trading within the compliance period.

[0072] The decision network training module is used to randomly collect subsequence samples from the complete interaction trajectory chain using a pre-set reward regression network. The cumulative transaction cost and total economic cost within the corresponding period of the subsequence are decomposed into agent rewards for each time step. Based on the agent rewards, the agent learns the long-term value of carbon quota trading actions at each time step and adjusts the gradient parameters of the pre-set trading strategy network along the gradient direction that maximizes the long-term value, thus obtaining the delayed reward reinforcement learning carbon trading model DR-RLCT for carbon trading.

[0073] This invention also provides a computer device. At the hardware level, the computer device includes a processor, an internal bus, a network interface, memory, and non-volatile memory, and may also include other hardware required for business operations. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to implement the carbon trading method provided above.

[0074] The present invention also provides a computer-readable storage medium storing a computer program that can be used to execute the carbon trading method provided above.

[0075] Specific limitations regarding the carbon trading method calculation system can be found in the limitations section on carbon trading methods above, and will not be repeated here. Each module in the aforementioned carbon trading system can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the computer device's memory as software, so that the processor can call and execute the corresponding operations of each module.

[0076] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification. Furthermore, the above embodiments only illustrate several implementation methods of this application, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A carbon trading method, characterized in that, include: Construct a carbon quota trading model for the carbon trading market, and collect carbon emission status data and the end point of the carbon emission compliance cycle in the carbon trading market; The carbon emission status data includes the current carbon allowance price in the target market, the current initial carbon emission status of the target controlled enterprises, and the carbon allowance inventory status. Based on the current carbon emission status data, the intelligent agent uses a preset trading strategy network to cycle through the Deep Deterministic Strategy Gradient Algorithm (DDPG) to output the carbon quota trading action at the current time step. The carbon allowance trading model uses the carbon allowance trading actions and the current carbon emission status data to update the carbon emission status data for the next time step; when the end of the compliance period is reached, the cyclical update of the carbon emission status data stops; the agent integrates the carbon emission status data, trading actions and the subsequence of the corresponding carbon emission status data for the next time step at each time step to form a complete interaction trajectory chain; the carbon allowance trading model calculates the total economic cost of carbon trading within the compliance period based on the complete interaction trajectory chain. By using a pre-set reward regression network to randomly collect subsequence samples from the complete interaction trajectory chain, the cumulative transaction cost and total economic cost within the corresponding period of the subsequence are decomposed into agent rewards for each time step. Based on the agent rewards, the agent learns the long-term value of carbon quota trading actions at each time step, and adjusts the gradient parameters of the pre-set trading strategy network along the gradient direction that maximizes the long-term value, thus obtaining the delayed reward reinforcement learning carbon trading model DR-RLCT for carbon trading.

2. The carbon trading method according to claim 1, characterized in that, The carbon emission status data of the target controlled enterprise is input into the Delayed Reward Reinforcement Learning Carbon Trading Model (DR-RLCT), which outputs the optimal carbon quota trading action at the current time step to guide the enterprise to complete the carbon trading.

3. The carbon trading method according to claim 1, characterized in that, The reward regression network is trained using the Randomized Reward Decomposition (RRD) method, specifically including: the preset reward regression network randomly samples subsequences from the complete interaction trajectory chain and generates a proxy training reward for each subsequence; with the goal of minimizing the difference between the proxy training reward and the total economic cost, the parameters of the reward regression network are iteratively updated using the RRD method to finally obtain the trained reward regression network.

4. A carbon trading method according to claim 1, characterized in that, The Deep Deterministic Policy Gradient Algorithm (DDPG) includes a Critic network and an Actor network.

5. A carbon trading method according to claim 4, characterized in that, Through the Critic network, the agent learns the long-term value of carbon allowance trading actions at each time step based on agent rewards; through the Actor network, the agent adjusts the gradient parameters of the preset trading strategy network along the gradient direction that maximizes the long-term value, thus obtaining the Delayed Reward Reinforcement Learning Carbon Trading Model DR-RLCT for carbon trading.

6. A carbon trading method according to claim 1, characterized in that, The total economic cost includes: the cumulative net expenditure on carbon emission allowance trading during the compliance period and the penalty for default corresponding to the carbon emission allowance shortfall at the end of the compliance period; the long-term value is the sum of the agent reward at the current time step and the estimated value of the state at the next time step.

7. A carbon trading method according to claim 1, characterized in that, The initial state includes: the remaining time step of the carbon emission compliance cycle, the amount of carbon emission allowances held, the amount of carbon emissions, and the real-time carbon price; the carbon allowance trading action is a continuous value, with a positive value indicating the purchase of carbon emission allowances and a negative value indicating the sale of carbon emission allowances.

8. A carbon trading system, characterized in that, include: Data collection models are used to construct carbon quota trading models for the carbon trading market, and to collect carbon emission status data and the end point of the carbon emission compliance cycle in the carbon trading market. The carbon emission status data includes the current carbon allowance price in the target market, the current initial carbon emission status of the target controlled enterprises, and the carbon allowance inventory status. The reward network training module is used by the agent to output the carbon quota trading action at the current time step based on the current carbon emission status data and the preset trading strategy network cyclically using the Deep Deterministic Strategy Gradient Algorithm (DDPG). The carbon allowance trading model uses the carbon allowance trading actions and the current carbon emission status data to update the carbon emission status data for the next time step; when the end of the compliance period is reached, the cyclical update of the carbon emission status data stops; the agent integrates the carbon emission status data, trading actions and the subsequence of the corresponding carbon emission status data for the next time step at each time step to form a complete interaction trajectory chain; the carbon allowance trading model calculates the total economic cost of carbon trading within the compliance period based on the complete interaction trajectory chain. The decision network training module is used to randomly collect subsequence samples from the complete interaction trajectory chain using a pre-set reward regression network. The cumulative transaction cost and total economic cost within the corresponding period of the subsequence are decomposed into agent rewards for each time step. Based on the agent rewards, the agent learns the long-term value of carbon quota trading actions at each time step and adjusts the gradient parameters of the pre-set trading strategy network along the gradient direction that maximizes the long-term value, thus obtaining the delayed reward reinforcement learning carbon trading model DR-RLCT for carbon trading.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is loaded by the processor, it is able to perform the steps of the method according to any one of claims 1 to 7.

Images