Edge computing power transaction method based on reinforcement learning

Abstract

The invention discloses an edge computing power transaction method based on reinforcement learning, and relates to the technical field of edge computing and artificial intelligence, and the method comprises the steps: obtaining edge computing power resource data, market data and historical transaction data; constructing a state space based on the edge computing power resource data, calculating a state vector, setting a value range of a preliminary transaction based on the market data and the historical transaction data, and obtaining an action space based on the value range; designing a reward function based on the market demand model to obtain an experience tuple; inputting the trained double-depth Q network based on the experience tuple, outputting a Q value corresponding to the action, and selecting the action with the highest Q value as a transaction decision; executing a transaction decision, and performing adjustment feedback on the dual-depth Q network according to the market data and the historical transaction data; the learning efficiency and accuracy of the pricing strategy are improved by adopting a reinforcement learning algorithm; an objective pricing mechanism based on data driving is established, and the objectivity and scientificity are ensured by using historical transaction data, computing power resource data and market data.

Description

Technical Field

[0001] This application belongs to the field of edge computing and artificial intelligence technology, and specifically relates to an edge computing power trading method based on reinforcement learning. Background Technology

[0002] Existing computing power trading methods include cost-plus pricing schemes, simple market bidding models, single-index parameter pricing techniques, and auction models. The disadvantages are: the pricing basis is singular and inaccurate; simple cost-plus pricing does not take into account the performance differences of computing power resources; the price markup ratio is the same for different performance nodes, which cannot reflect the true value and cannot meet users' differentiated needs for computing power with different performance.

[0003] Pricing methods based on market supply and demand are crude and fail to take into account the unique attributes of each computing node. The acquisition and analysis of market supply and demand information are not timely and accurate, resulting in delayed or excessive price adjustments.

[0004] Lacking adaptability to dynamic market changes, existing pricing models are mostly static and difficult to adjust in a timely manner according to rapid market changes. When new computing power application scenarios emerge with new requirements for specific performance indicators, the models cannot quickly capture these changes, causing prices to become decoupled from market value.

[0005] Facing changes in competitors' strategies, the lack of an effective response mechanism, the inability to analyze the impact in a timely manner and adjust one's own pricing strategy, makes it difficult to maintain competitiveness.

[0006] The ability to handle complex computing resource scenarios is insufficient, large-scale computing resource management is difficult, existing technologies are unable to quickly and accurately analyze the multi-dimensional parameters of massive edge computing nodes, the pricing process is lengthy and inaccurate, and it cannot meet the real-time requirements of large-scale transactions.

[0007] The characteristics of diverse computing resources are not adequately considered. Different industry application scenarios have different requirements for computing resources, and existing technologies are unable to formulate reasonable prices for diverse needs. Summary of the Invention

[0008] To address the aforementioned problems, this application provides an edge computing power trading method based on reinforcement learning, the improvement of which includes: Obtain edge computing resource data, market data, and historical transaction data; A state space is constructed based on edge computing resource data, and state vectors are calculated. The initial transaction value range is set based on market data and historical transaction data, and the action space is obtained based on the value range. A reward function is designed based on the market demand model, and empirical tuples are obtained. Based on the input of empirical tuples, a well-trained dual-deep Q-network is generated, and the Q-value corresponding to the action is output. The action with the highest Q-value is selected as the trading decision. Execute transaction decisions and adjust the dual-depth Q network based on transaction data and historical transaction data.

[0009] Optionally, the training process of the dual deep Q-network includes: A training module consisting of an online network and a target network is constructed using a dual-depth Q-network algorithm, and the parameters θ of the online network and the target network are initialized. - Create an experience replay buffer to store training experience tuples of the agent's interaction with the environment. Training experience tuples with preset values ​​are randomly sampled from the experience replay buffer. The state vector s is input into the online network. Based on the current state, action a is selected from action space A. The target value is output and the target network is used to calculate the target Q value. The online network parameters are updated by minimizing the mean squared error loss function, and the online network weights are periodically copied to the target network for synchronous updates, resulting in a well-trained dual-deep Q network. Among them, the dual-depth network update process is triggered based on market data and historical transaction data.

[0010] Optionally, the edge computing resource data includes one or more of the following: CPU core count, clock frequency, instruction set efficiency, storage capacity, read / write speed, data fault tolerance, memory capacity, memory bandwidth, network bandwidth, latency loss rate, network stability factor, data center clustering, energy cost, or market demand.

[0011] Optionally, the step of constructing a state space based on edge computing resource data and calculating state vectors includes: Processor status parameters are constructed based on the number of CPU cores, clock frequency, and instruction set efficiency. ; Distributed storage state parameters are constructed based on storage capacity, read / write speed, and data fault tolerance. ; Memory state parameters are constructed based on memory capacity and memory bandwidth. ; Network state parameters are constructed based on network bandwidth, latency loss rate, and network stability factors. ; Geographic location status parameters are constructed based on data center clustering, energy costs, and market demand. ; based on , , , and Construct the state vector s.

[0012] Optionally, the step of setting a preliminary transaction value range based on market data and historical transaction data, and obtaining the action space based on the value range, includes: Based on market data and historical transaction data, a preliminary range of transaction values ​​is set, and the range of values ​​is divided into n discrete points to form an action space; The division of discrete points is based on market data and historical transaction data.

[0013] Optionally, the reward function includes:

[0014] Where r represents the reward value, Indicates the return weight. Indicates the weight of purchase volume. Indicates revenue, Indicates the quantity purchased; The revenue is determined based on a market demand model.

[0015] Optionally, after obtaining edge computing resource data, market data, and historical transaction data, and before constructing the state space based on the edge computing resource data and market data, the process also includes integrating and preprocessing the edge computing resource data, market data, and historical transaction data: Edge computing resource data, market data, and historical transaction data are categorized into numerical data and non-numerical data. For numerical data, the normalized values ​​are mapped to the [0,1] interval using a min-max normalization function. For non-numerical data, normalization is performed according to resource scarcity and cost.

[0016] Based on the same inventive concept, this application also provides an edge computing power trading device based on reinforcement learning, the improvement of which includes: The data acquisition unit is used to obtain edge computing resource data, market data, and historical transaction data. The building unit is used to construct the state space based on edge computing resource data, calculate the state vector, set the initial transaction value range based on market data and historical transaction data, obtain the action space based on the value range, and design the reward function based on the market demand model to obtain the experience tuple. The decision unit is used to input the trained dual-deep Q-network based on the experience tuples, output the Q value corresponding to the action, and select the action with the highest Q value as the transaction decision. The feedback unit is used to execute trading decisions and adjust the dual-depth Q network based on trading data and historical trading data.

[0017] Optionally, the training process of the dual deep Q-network includes: A training module consisting of an online network and a target network is constructed using a dual-depth Q-network algorithm, and the parameters θ of the online network and the target network are initialized. - Create an experience replay buffer to store training experience tuples of the agent's interaction with the environment. Training experience tuples with preset values ​​are randomly sampled from the experience replay buffer. The state vector s is input into the online network. Based on the current state, action a is selected from action space A. The target value is output and the target network is used to calculate the target Q value. The online network parameters are updated by minimizing the mean squared error loss function, and the online network weights are periodically copied to the target network for synchronous updates, resulting in a well-trained dual-deep Q network. Among them, the dual-depth network update process is triggered based on market data and historical transaction data.

[0018] Optionally, the edge computing resource data includes one or more of the following: CPU core count, clock frequency, instruction set efficiency, storage capacity, read / write speed, data fault tolerance, memory capacity, memory bandwidth, network bandwidth, latency loss rate, network stability factor, data center clustering, energy cost, or market demand.

[0019] Optionally, the step of constructing a state space based on edge computing resource data and calculating state vectors includes: Processor status parameters are constructed based on the number of CPU cores, clock frequency, and instruction set efficiency. ; Distributed storage state parameters are constructed based on storage capacity, read / write speed, and data fault tolerance. ; Memory state parameters are constructed based on memory capacity and memory bandwidth. ; Network state parameters are constructed based on network bandwidth, latency loss rate, and network stability factors. ; Geographic location status parameters are constructed based on data center clustering, energy costs, and market demand. ; based on , , , and Construct the state vector s.

[0020] Optionally, the step of setting a preliminary transaction value range based on market data and historical transaction data, and obtaining the action space based on the value range, includes: Based on market data and historical transaction data, a preliminary range of transaction values ​​is set, and the range of values ​​is divided into n discrete points to form an action space; The division of discrete points is based on market data and historical transaction data.

[0021] Optionally, the reward function includes:

[0022] Where r represents the reward value, Indicates the return weight. Indicates the weight of purchase volume. Indicates revenue, Indicates the quantity purchased; The revenue is determined based on a market demand model.

[0023] Optionally, after obtaining edge computing resource data, market data, and historical transaction data, and before constructing the state space based on the edge computing resource data and market data, the process also includes integrating and preprocessing the edge computing resource data, market data, and historical transaction data: Edge computing resource data, market data, and historical transaction data are categorized into numerical data and non-numerical data. For numerical data, the normalized values ​​are mapped to the [0,1] interval using a min-max normalization function. For non-numerical data, normalization is performed according to resource scarcity and cost.

[0024] Furthermore, this application also provides a computing device, comprising: at least one processor and a memory; The memory is used to store one or more programs; When the one or more programs are executed by the one or more processors, an edge computing power trading method based on reinforcement learning as described above is implemented.

[0025] In another aspect, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed, implements the edge computing power trading method based on reinforcement learning as described above.

[0026] This application provides an edge computing power trading method based on reinforcement learning, comprising: obtaining edge computing power resource data, market data, and historical transaction data; constructing a state space based on the edge computing power resource data, calculating state vectors, setting a preliminary transaction value range based on market data and historical transaction data, and obtaining an action space based on the value range; designing a reward function based on a market demand model to obtain an experience tuple; inputting the experience tuple into a trained dual-deep Q-network, outputting the Q-value corresponding to the action, and selecting the action with the highest Q-value as the trading decision; executing the trading decision, and adjusting and feeding back the dual-deep Q-network based on market data and historical transaction data; The reinforcement learning algorithm (Double Deep Q-Network, DDQN) is adopted to overcome the problems of low efficiency and difficulty in convergence to the optimal policy when the traditional algorithm is dealing with a large-scale state-action space. The algorithm structure and parameter update mechanism are optimized to improve the learning efficiency and accuracy of pricing strategy. Establish a data-driven objective pricing mechanism by utilizing a large amount of historical transaction data, computing power resource parameter data, and market dynamic data to reduce the influence of human factors, enabling the model to automatically learn and formulate pricing strategies, and ensuring the objectivity and scientific nature of pricing. This application uses a multi-dimensional pricing model, taking into account detailed parameters such as CPU, distributed storage, memory, network, and geographical location, to accurately assess the value of computing power nodes and set reasonable prices for computing power resources with different performance. Establish a real-time updated value assessment mechanism to ensure that prices always reflect the actual value of resources, taking into account factors such as changes in the hardware performance of computing nodes and changes in the market environment.

[0027] Other features and advantages of this application will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures pointed out in the description, claims and drawings. Attached Figure Description

[0028] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0029] Figure 1 A flowchart illustrating the implementation of an edge computing power trading method based on reinforcement learning provided in this application is shown. Figure 2 The diagram shows the architecture of the edge computing power trading method model provided in this application; Figure 3 This application provides a business process diagram of the edge computing power trading method model. Figure 4 A schematic diagram of an edge computing power trading device based on reinforcement learning provided in this application is shown. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0031] Example 1 This application provides an edge computing power trading method based on reinforcement learning, such as Figure 1 ,include: Obtain edge computing resource data, market data, and historical transaction data; A state space is constructed based on edge computing resource data, and state vectors are calculated. The initial transaction value range is set based on market data and historical transaction data, and the action space is obtained based on the value range. A reward function is designed based on the market demand model, and empirical tuples are obtained. Based on the input of empirical tuples, a well-trained dual-deep Q-network is generated, and the Q-value corresponding to the action is output. The action with the highest Q-value is selected as the trading decision. Execute trading decisions and adjust the dual-depth Q network based on market data and historical trading data.

[0032] The architecture diagram of this application is as follows: Figure 2 It is divided into a data layer, a model core layer, and an application layer. The data layer includes a data collection module and a data preprocessing module. The model core layer includes a state space construction module, an action space determination module, a reward function module, and a DDQN training module. The application layer includes a pricing decision module and a model update module.

[0033] Data collection module: Computing resource data collection: Obtain resource data such as hardware level (CPU, distributed storage, memory, network, geographical location) of computing nodes through the orchestration and management system.

[0034] Optionally, the edge computing resource data includes one or more of the following: CPU core count, clock frequency, instruction set efficiency, storage capacity, read / write speed, data fault tolerance, memory capacity, memory bandwidth, network bandwidth, latency loss rate, network stability factor, data center clustering, energy cost, or market demand.

[0035] Market data collection: Collect market data from multiple sources, including computing power demand trends in different industries, total supply, competitor computing power service prices and service quality evaluations, etc.

[0036] Historical transaction data collection: Extract transaction records from the computing power trading platform database, including price, computing power quantity, transaction time, and information of buyers and sellers.

[0037] Data preprocessing module: Integrates data from different data sources but belonging to the same computing power node or market dimension, and cleans the data to remove errors or incomplete information.

[0038] Edge computing resource data, market data, and historical transaction data are categorized into numerical data and non-numerical data. For numerical data (such as the number of CPU cores, memory capacity, etc.), a min-max normalization function is used. Let the original data be X, the minimum value of a certain data dimension be Xmin, and the maximum value be Xmax. The normalization formula is: The normalized values ​​are mapped to the [0,1] interval. For non-numerical data (such as geographical location), normalization is performed according to the resource scarcity and cost to facilitate subsequent model training and calculation.

[0039] State space construction module: Optionally, the step of constructing a state space based on edge computing resource data and calculating state vectors includes: Based on the number of CPU cores C and clock frequency and instruction set efficiency Build processor status parameters ;

[0040] in, Indicates the weight of the number of CPU cores. Indicates clock frequency weighting. Indicates instruction set efficiency weights; Based on storage capacity V, read / write speed R, and data fault tolerance. Constructing distributed storage state parameters ;

[0041] in, Indicates storage capacity weight. Indicates read / write speed weights. The weight represents the data fault tolerance capability, where m represents the number of faulty nodes the storage system can tolerate, and N represents the total number of nodes. Memory state parameters are constructed based on memory capacity M and memory bandwidth B. ;

[0042] in, Indicates memory capacity weight. Indicates memory bandwidth weight; Network state parameters are constructed based on network bandwidth W, latency loss rate L, and network stability factor S. ;

[0043] in, Indicates network bandwidth weight. Indicates the weight of the delay loss rate. Indicates the weight of the network stability factor; The delay loss rate is calculated by assuming the ideal transmission time is T. ideal The actual transmission time is T. actual The delay loss rate can be expressed as: The network stability factor S = 1 - (PacketLossRate + JitterRate), where PacketLossRate is the network packet loss rate and JitterRate is the jitter rate. Geographic location status parameters are constructed based on data center clustering, energy costs, and market demand. ; based on , , , and Construct the state vector s.

[0044] Action space determination module: The initial trading range is set based on market data and historical trading data. The range of values ​​is divided into n discrete points (which can be set according to the policies of the computing power trading platform), forming an action space. ; The division of discrete points is based on market data and historical transaction data.

[0045] The formula for calculating the price point is:

[0046] in, This represents the i-th price point. This represents the minimum value of the initial transaction. This indicates the maximum value of the initial transaction.

[0047] Reward function module:

[0048] Where r represents the reward value, Indicates the return weight. Indicates the weight of purchase volume. Indicates revenue, Indicates the quantity purchased; The revenue is determined based on a market demand model.

[0049] income (where Q is the transaction price and P is the transaction computing power quantity), the relationship between purchase quantity Q and price P is derived from the market demand model. (where k is a constant related to market size and β is the price elasticity coefficient) is determined.

[0050] The weights of different types of computing power resources can be adjusted. For high-performance and scarce resources, the revenue weight can be appropriately increased, while for ordinary and easily accessible resources, more attention should be paid to the purchase volume weight. Dynamic adjustment allows the reward function to adapt to market changes and resource characteristics, and a reasonable pricing strategy can be formulated.

[0051] DDQN training module: A training module consisting of an online network and a target network is constructed using a dual-depth Q-network algorithm, and the parameters θ of the online network and the target network are initialized. - Create an experience replay buffer, storing the training experience tuples (s, a, r, s) of the agent's interactions with the environment. ’ Stored in the experience replay buffer; Randomly sample training experience tuples (s, a, r, s) with preset values ​​from the experience replay buffer. ’ ), r represents the reward value, s ’ To represent the successor state, break the data correlation, improve training efficiency and stability, input the state vector s into the online network, select action a from action space A according to the current state, output the target value, and use the target network to calculate the target Q value; The online network parameters are updated by minimizing the mean squared error loss function, and the online network weights are periodically copied to the target network for synchronous updates, resulting in a well-trained dual-deep Q network. Among them, the dual-depth network update process is triggered based on market data and historical transaction data.

[0052] The loss function that minimizes the mean squared error is:

[0053] in, This represents minimizing the mean squared error loss. Represents the mathematical expectation. Indicates the target Q value. This represents the Q-value predicted by the online network.

[0054] The formula for calculating the target Q value is:

[0055] in, It is a discount factor. It is the successor state. It is a candidate action for the successor state.

[0056] By reasoning, the formula for calculating the target Q value is:

[0057] in, This refers to the action that enables the online network to calculate the maximum Q value.

[0058] Pricing decision module: When a new computing power transaction request is made, the corresponding computing power node status information is input into the trained DDQN model (online network). The model outputs the Q value corresponding to each action (price point), selects the price with the highest Q value as the pricing suggestion, and feeds it back to the transaction system to complete the pricing decision.

[0059] Market monitoring and model update module: Continuously monitor market dynamics, including supply and demand, competitor strategies, and changes in computing power requirements from new application scenarios. When significant market changes occur, trigger the model update process, adjust relevant parameters of the model's core architecture (reward function weights, price range, state space construction rules, etc.) based on the new market data, and initiate retraining to ensure the model adapts to the new market environment and maintains consistent accuracy in pricing decisions. Simultaneously, receive transaction result information from the trading system to evaluate model performance and guide optimization.

[0060] Example 2 Based on the same inventive concept, this application also provides an optimal business process for an edge computing power trading method based on reinforcement learning, such as... Figure 3 ,include: Step 1, Submission of Business Requirements: Computing power consumers submit their business requirements to the orchestration management system through the computing power operation platform interface, including information such as computing power performance (CPU, memory, storage, etc.), network bandwidth, service duration, data security and privacy requirements, and geographical location preferences.

[0061] Step 2, Requirements Analysis: The orchestration management system analyzes business requirements into identifiable parameters and conditions.

[0062] Step 3, Requirement Matching: Search for matching in the resource template library. The resource template library stores resource combination templates for different business scenarios, including information such as the type and quantity of computing resources, performance parameter range and deployment architecture. Select one or more of the most suitable templates.

[0063] Step 4, Resource Template Matching: Send the relevant parameters of computing resources of the matching template to the transaction pricing model.

[0064] Step 5, Data Processing: The transaction pricing model cleans and normalizes the received parameters (same as the data layer preprocessing).

[0065] 1) Modeling: 1. Based on preprocessing parameters, modeling and learning are performed according to the built-in algorithm and model structure, and comprehensive analysis of multiple indicators such as CPU, distributed storage, memory, network and geographical location is conducted.

[0066] 2. State space construction (same as the state space construction module of the core layer of the model).

[0067] 3. Action space determination (same as the action space determination module of the core layer of the model).

[0068] 4. Reward function design (same as the reward function design module in the core layer of the model).

[0069] 2) Reinforcement learning: 1. A multilayer fully connected neural network (MLP) is used to approximate the Q-function. The number of neurons in the input layer is equal to the dimension of the state space, receiving the state vectors of the computing nodes. There are 2-3 hidden layers, with the first layer having approximately half to two-thirds the number of neurons as the input layer, decreasing sequentially in subsequent layers. The ReLU activation function is used to avoid gradient vanishing and accelerate convergence. The number of neurons in the output layer is equal to the dimension of the action space, outputting the Q-value estimate for each action.

[0070] 2. Train the neural network using DQN and DDQN algorithms, update the parameters by minimizing the MSE loss function (formula same as DDQN training module), use the DDQN method to calculate the target Q value (same as DDQN training module), and use the experience replay mechanism to improve training efficiency and stability (same as DDQN training module).

[0071] Step 6, Parameter Optimization: As new business needs and resource template information are input, continuously update and optimize model parameters (such as DDQN neural network weights and learning rate) to adapt to market changes and new scenarios, and provide accurate, reasonable and competitive pricing.

[0072] Step 7, Pricing Feedback: The transaction pricing model calculates the price and feeds it back to the orchestration management system, which integrates it with the resource templates to form a business solution quote.

[0073] Step 8, Resource Allocation: If the consumer accepts the quote, the orchestration management system allocates computing resources according to the resource template and settles the fee with the consumer based on the actual resources used and the duration of use.

[0074] Step 9: Model update and optimization based on transaction data: After the business is completed, collect actual transaction data (pricing, purchase volume, resource usage, consumer feedback, etc.), analyze the reasons for deviations, adjust parameters, improve the accuracy and adaptability of the model, and enrich the training dataset to cope with future scenario changes.

[0075] Example 3 Based on the same inventive concept, this application also provides an edge computing power trading device based on reinforcement learning, such as... Figure 4 ,include: The data acquisition unit is used to obtain edge computing resource data, market data, and historical transaction data. The building unit is used to construct the state space based on edge computing resource data, calculate the state vector, set the initial transaction value range based on market data and historical transaction data, obtain the action space based on the value range, and design the reward function based on the market demand model to obtain the experience tuple. The decision unit is used to input the trained dual-deep Q-network based on the experience tuples, output the Q value corresponding to the action, and select the action with the highest Q value as the transaction decision. The feedback unit is used to execute trading decisions and adjust the dual-depth Q network based on actual trading data.

[0076] A training module consisting of an online network and a target network is constructed using a dual-depth Q-network algorithm, and the parameters θ of the online network and the target network are initialized. - Create an experience replay buffer, storing the experience tuples (s, a, r, s) of the agent's interactions with the environment. ’ Stored in the experience replay buffer; An empirical tuple (s, a, r, s) of preset values ​​is randomly sampled from the empirical replay buffer. ’ The state vector s is input into the online network, and action a is selected from action space A according to the current state. The target value is output, and the target Q value is calculated using the target network. The online network parameters are updated by minimizing the mean squared error loss function, and the online network weights are periodically copied to the target network for synchronous updates, resulting in a well-trained dual-deep Q network. Specifically, the dual-depth network update process is triggered based on changes in market dynamics in market data.

[0077] Optionally, the edge computing resource data includes one or more of the following: CPU core count, clock frequency, instruction set efficiency, storage capacity, read / write speed, data fault tolerance, memory capacity, memory bandwidth, network bandwidth, latency loss rate, network stability factor, data center clustering, energy cost, or market demand.

[0078] Optionally, the step of constructing a state space based on edge computing resource data and calculating state vectors includes: Processor status parameters are constructed based on the number of CPU cores, clock frequency, and instruction set efficiency. ; Distributed storage state parameters are constructed based on storage capacity, read / write speed, and data fault tolerance. ; Memory state parameters are constructed based on memory capacity and memory bandwidth. ; Network state parameters are constructed based on network bandwidth, latency loss rate, and network stability factors. ; Geographic location status parameters are constructed based on data center clustering, energy costs, and market demand. ; based on , , , and Construct the state vector s.

[0079] Optionally, the step of setting a preliminary transaction value range based on market data and historical transaction data, and obtaining the action space based on the value range, includes: Based on market data and historical transaction data, a preliminary range of transaction values ​​is set, and the range of values ​​is divided into n discrete points to form an action space; The division of discrete points is based on market data and historical transaction data.

[0080] Optionally, the reward function includes:

[0081] Where r represents the reward value, Indicates the return weight. Indicates the weight of purchase volume. Indicates revenue, Indicates the quantity purchased; The revenue is determined based on a market demand model.

[0082] Optionally, after obtaining edge computing resource data, market data, and historical transaction data, and before constructing the state space based on the edge computing resource data and market data, the process also includes integrating and preprocessing the edge computing resource data, market data, and historical transaction data: Edge computing resource data, market data, and historical transaction data are categorized into numerical data and non-numerical data. For numerical data, the normalized values ​​are mapped to the [0,1] interval using a min-max normalization function. For non-numerical data, normalization is performed according to resource scarcity and cost.

[0083] Example 4 Based on the same inventive concept, the present invention also provides an electronic device. The electronic device of this embodiment includes at least one processor and at least one storage medium electrically connected to the processor, wherein the storage medium stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method described above.

[0084] Example 5 Based on the same inventive concept, the present invention also provides a storage medium storing instructions executable by at least one processor, the instructions being executed by at least one processor to enable at least one processor to perform the method described above.

[0085] Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for edge computing power transaction based on reinforcement learning, characterized in that, The application relates to a method for constructing a double deep Q network for edge computing resource trading. The method comprises the following steps: obtaining edge computing resource data, market data and historical transaction data; constructing a state space based on the edge computing resource data, calculating a state vector, setting a value range of a preliminary transaction based on the market data and the historical transaction data, and obtaining an action space based on the value range; designing a reward function based on a market demand model to obtain an experience tuple; inputting the experience tuple into a trained double deep Q network to output a Q value corresponding to an action, and selecting an action with the highest Q value as a transaction decision; 2. A method as claimed in claim 1, characterized in that adjusting and feeding back the double deep Q network according to transaction data and historical transaction data. A double deep Q network algorithm is adopted to construct a training module including an online network and a target network, and an online network parameter θ and a target network parameter θ are initialized - An experience replay buffer is created, and a training experience tuple of an agent and an environment is stored in the experience replay buffer. The training process of the double deep Q network comprises the following steps: randomly sampling a preset number of training experience tuples from an experience replay buffer, inputting a state vector s into an online network, selecting an action a from an action space A according to the current state, outputting a target value, and calculating a target Q value by using a target network; updating the online network parameters by minimizing a mean square error loss function, periodically copying the online network weights to the target network for synchronous updating, and obtaining the trained double deep Q network; 3. A method as claimed in claim 1, characterized in that wherein the double deep network updating process is triggered according to the market data and the historical transaction data. The edge computing resource data comprises one or more of the following:

4. A method as claimed in claim 3, characterised in that, CPU core number, clock frequency, instruction set efficiency, storage capacity, read-write speed, data fault tolerance capability, memory capacity, memory bandwidth, network bandwidth, delay loss rate, network stability factor, data center aggregation degree, energy cost or market demand heat. Building processor state parameters based on CPU core count, clock frequency, and instruction set efficiency ; Distributed storage state parameters are constructed based on storage capacity, read-write speed, and data fault tolerance capability ; Memory state parameter construction based on memory capacity and memory bandwidth ; Constructing network state parameters based on network bandwidth, delay loss rate and network stability factor ; Constructing a geographical position state parameter based on data center aggregation, energy cost and market demand heat ; Based on , , , and a state vector s is constructed.

5. The method of claim 1, wherein, The state space is constructed based on the edge computing resource data, and the state vector is calculated, which comprises the following steps: The value range of the preliminary transaction is set based on the market data and the historical transaction data, the value range is divided into n discrete points to form an action space, and the action space is obtained based on the value range. The reward function comprises:

6. A method as claimed in claim 1, characterized in that wherein the income is determined based on a market demand model. wherein r represents a reward value, represents a revenue weight, represents a purchase amount weight, represents a revenue, represents a purchase amount; After obtaining the edge computing resource data, the market data and the historical transaction data, and before constructing the state space based on the edge computing resource data and the market data, the edge computing resource data, the market data and the historical transaction data are integrated and preprocessed, which comprises the following steps:

7. A method as claimed in claim 1, characterized in that dividing the edge computing resource data, the market data and the historical transaction data into numerical data and non-numerical data; for the numerical data, the minimum-maximum normalization function is used for processing, and the normalized value is mapped to the interval [0, 1]; and for the non-numerical data, normalization is performed according to the resource shortage degree and the cost. The application further relates to a device for constructing a double deep Q network for edge computing resource trading.

8. A reinforcement learning-based edge computing power transaction device, characterized in that, The device comprises the following units: a data acquisition unit for obtaining edge computing resource data, market data and historical transaction data; a construction unit for constructing a state space based on the edge computing resource data, calculating a state vector, setting a value range of a preliminary transaction based on the market data and the historical transaction data, and obtaining an action space based on the value range; a reward function design unit for designing a reward function based on a market demand model to obtain an experience tuple; a double deep Q network training unit for inputting the experience tuple into a trained double deep Q network to output a Q value corresponding to an action, and selecting an action with the highest Q value as a transaction decision. A decision unit is configured to input the trained double deep Q network based on the experience tuples, output a Q value corresponding to an action, and select an action with the highest Q value as a transaction decision; A feedback unit is configured to execute the transaction decision, and adjust and feed back the double deep Q network based on transaction data and historical transaction data.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the steps of the edge computing power transaction method based on reinforcement learning in any of claims 1-7.

10. An electronic device, comprising: The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the steps of the edge computing power transaction method based on reinforcement learning in any of claims 1-7. The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the steps of the edge computing power transaction method based on reinforcement learning in any of claims 1-7. The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the steps of the edge computing power transaction method based on reinforcement learning in any of claims 1-7.

Images