Electric power data configuration method and system based on dual deep reinforcement learning

Abstract

The invention belongs to the technical field of power data configuration, and provides a power data configuration method and system based on dual deep reinforcement learning, and the method comprises the steps: obtaining a power data collection task; analyzing the obtained electric power data acquisition task to obtain an electric power data acquisition task attribute feature; and according to the obtained collection task attribute characteristics and the dual deep reinforcement learning scheduling model, performing optimal scheduling of the power data, and completing configuration of the power data. According to the dual deep reinforcement learning scheduling model, calculation of Q values and selection actions are placed in two different neural networks respectively, the Q values of all the actions are output according to the current environment state of the power data in the evaluation network, the maximum Q value is selected to execute the actions, and the target Q value after the actions are executed serves as the learning target of the evaluation network. And network parameters are updated and evaluated in combination with a gradient descent method, and optimization training of the dual deep reinforcement learning scheduling model is completed.

Description

Technical Field

[0001] The present invention belongs to the technical field of power data configuration, and specifically relates to a power data configuration method and system based on dual deep reinforcement learning. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] The electricity consumption information collection system plays an important supporting role in precise load control, demand-side regulation, and supply-demand balance. It can collect electricity consumption information in real time, conduct routine event monitoring, and implement billing control. Therefore, efficient upload of terminal electricity consumption data has become the basis for the development of various businesses. However, traditional electricity consumption information collection systems often face high load problems in the case of multi-task, high concurrency, and large-scale collection. High load may not only lead to a decrease in the collection rate and delay in task completion time, but also directly lead to the development of upper-level business of the system, such as remote fee control, electricity spot trading, etc.; this problem has become one of the challenges that the electricity consumption information collection system needs to solve in practical applications.

[0004] To meet the challenge, the current method of distributing fixed parameters in batches to configure power data is static. Although it can alleviate system pressure to a certain extent, it is not realistic to use it to solve job scheduling problems in real-time dynamic and complex environments such as electricity spot trading. Heuristic methods such as genetic algorithms (GA), ant colony optimization (ACO) or particle swarm optimization (PSO) can also be used to reach a local optimal solution with a certain probability, but the robustness is poor and it is impossible to continuously maintain the load balancing state of system resources.

[0005] With the vigorous development of deep learning and reinforcement learning technologies, the adaptive configuration method based on intelligent agents has gradually become an important means to solve the problem of multi-task resource allocation. A related study proposed a method of switching scheduling algorithms using multi-task learning. The scheduling algorithm is changed by introducing multi-task learning in the scheduling framework. Reinforcement learning is widely used in multi-task scheduling, relying on its ability to adjust strategies according to environmental feedback to achieve the best long-term goals; a related scholar proposed a multi-task deep reinforcement learning method for scalable parallel task scheduling, using the output branches of multi-task learning to match parallel scheduling tasks, and sharing neural network layers to achieve effective utilization of resource competition between parallel tasks; the Proximal Policy Optimization (PPO) algorithm can be applied to solve complex task scheduling problems. However, the existing power data configuration method has the risk of reduced collection success rate and delayed task completion time when supporting minute-level and large-scale power data collection. Summary of the invention

[0006] To solve the above problems, the present invention proposes a power data configuration method and system based on dual deep reinforcement learning, which adopts an adaptive configuration method for uploading power consumption data of distribution network terminals based on dual deep reinforcement learning (Double Deep Q-Network, referred to as DDQN), combined with the experience replay mechanism (Experience Replay) and the improved particle swarm optimization (PSO) pre-training method, reduces the data transmission, data analysis and data storage pressure of the front-end communication platform of the existing acquisition system, and improves the configuration efficiency of power data in a multi-task environment.

[0007] According to some embodiments, a first solution of the present invention provides a power data configuration method based on dual deep reinforcement learning, which adopts the following technical solutions:

[0008] A power data configuration method based on dual deep reinforcement learning, comprising:

[0009] Obtain power data collection tasks;

[0010] Analyze the acquired power data collection tasks to obtain attribute characteristics of the power data collection tasks;

[0011] According to the acquired task attribute characteristics and the dual deep reinforcement learning scheduling model, the power data is optimized and scheduled to complete the configuration of the power data;

[0012] Among them, the dual deep reinforcement learning scheduling model places the calculation of Q value and the selection of action in the evaluation network and the target network respectively. The evaluation network outputs the Q value of each action according to the current environmental state of the power data and selects the maximum Q value to execute the action. The target network uses the target Q value after executing the action as the learning target of the evaluation network, and updates the evaluation network parameters in combination with the gradient descent method to complete the optimization training of the dual deep reinforcement learning scheduling model.

[0013] As a further technical limitation, the reinforcement learning of the acquired power data collection task includes a power collection environment and an intelligent agent; the power collection environment includes a collection terminal cluster and a power consumption information collection task, as well as a feature description related to the data collection task and the collection status; based on reinforcement learning, the intelligent agent is used as a scheduler, and actions are selected according to the current state to assign tasks to appropriate machines, and feedback is obtained from the environment as a reward; by continuously maximizing the accumulation of rewards, the intelligent agent finds the optimal scheduling sequence.

[0014] Furthermore, an improved particle swarm optimization algorithm is used to obtain the optimal scheduling sequence of the intelligent agent; the improved particle swarm optimization algorithm adopts a linearly decreasing adaptive inertia weight and a fitness function that considers the optimization target completion time.

[0015] Furthermore, the key elements of the reinforcement learning agent include action space, state space and reward function. The agent selects and executes actions according to the environmental state, and the reward function is used to motivate the agent to adjust its actions to achieve a maximized reward value. When the scheduling is not completed, the agent considers scheduling the task to a set of task scheduling sequences with the shortest completion time and a high task collection success rate, obtains reward values ​​based on environmental feedback, and updates the collection task sorting.

[0016] As a further technical limitation, during the learning process of the target network, the agent obtains the current state from the environment, and the evaluation network outputs the Q value of each action and executes the action; if the task scheduling is completed, the target value is the actual reward value obtained; otherwise, the target network calculates the target Q value as the target value; the mean square error loss is calculated by combining the temporal difference target value and the Q value of the evaluation network, and the evaluation network parameters are updated according to the obtained mean square error loss and the gradient descent method.

[0017] As a further technical limitation, during the optimization training of the dual deep reinforcement learning scheduling model, an experience replay control unit for marking experience data is added to randomly extract previous experience for learning to break the correlation between the data.

[0018] According to some embodiments, a second solution of the present invention provides a power data configuration system based on dual deep reinforcement learning, which adopts the following technical solutions:

[0019] A power data configuration system based on dual deep reinforcement learning, comprising:

[0020] An acquisition module, which is configured to acquire power data acquisition tasks;

[0021] A parsing module, which is configured to parse the acquired power data collection task to obtain attribute characteristics of the power data collection task;

[0022] A configuration module, which is configured to optimize the scheduling of power data according to the obtained collection task attribute characteristics and the dual deep reinforcement learning scheduling model, and complete the configuration of the power data;

[0023] Among them, the dual deep reinforcement learning scheduling model places the calculation of Q value and the selection of action in the evaluation network and the target network respectively. The evaluation network outputs the Q value of each action according to the current environmental state of the power data and selects the maximum Q value to execute the action. The target network uses the target Q value after executing the action as the learning target of the evaluation network, and updates the evaluation network parameters in combination with the gradient descent method to complete the optimization training of the dual deep reinforcement learning scheduling model.

[0024] According to some embodiments, a third solution of the present invention provides a computer-readable storage medium, which adopts the following technical solution:

[0025] A computer-readable storage medium stores a program thereon, which, when executed by a processor, implements the steps in a power data configuration method based on dual deep reinforcement learning as described in the first solution of the present invention.

[0026] According to some embodiments, a fourth solution of the present invention provides an electronic device, which adopts the following technical solution:

[0027] An electronic device comprises a memory, a processor and a program stored in the memory and running on the processor, wherein when the processor executes the program, the steps in a power data configuration method based on dual deep reinforcement learning as described in the first scheme of the present invention are implemented.

[0028] According to some embodiments, a fifth solution of the present invention provides a computer program product, which adopts the following technical solution:

[0029] A computer program product includes software code, wherein the program in the software code executes the steps in a power data configuration method based on dual deep reinforcement learning as described in the first embodiment of the present invention.

[0030] Compared with the prior art, the present invention has the following beneficial effects:

[0031] The present invention combines the experience replay mechanism with the improved particle swarm optimization (PSO) pre-training method to optimize the learning efficiency and task performance of the model, makes full use of the reinforcement learning framework of DDQN, and combines the characteristics of terminal power consumption data upload to achieve the optimization and improvement of the adaptive configuration strategy. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The drawings in the specification that constitute a part of this embodiment are used to provide a further understanding of this embodiment. The schematic embodiments of this embodiment and their descriptions are used to explain this embodiment and do not constitute improper limitations on this embodiment.

[0033] Figure 1 is a structural block diagram of a power data acquisition system in Embodiment 1 of the present invention;

[0034] Figure 2 This is a flow chart of a power data configuration method based on dual deep reinforcement learning in Embodiment 1 of the present invention;

[0035] Figure 3 This is a flowchart of initializing the dual deep reinforcement learning scheduling model in Embodiment 1 of the present invention;

[0036] Figure 4This is a structural block diagram of a power data configuration system based on dual deep reinforcement learning in Embodiment 2 of the present invention. DETAILED DESCRIPTION

[0037] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0038] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.

[0039] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0040] In the present invention, terms such as "upper", "lower", "left", "right", "front", "back", "vertical", "horizontal", "side", "bottom", etc. indicate directions or positional relationships based on the directions or positional relationships shown in the accompanying drawings. They are relational words determined only for the convenience of describing the structural relationships of the various parts or elements of the present invention, and do not specifically refer to any part or element in the present invention and should not be understood as limitations on the present invention.

[0041] In the present invention, terms such as "fixed connection", "connected", "connection", etc. should be understood in a broad sense, indicating that it can be fixedly connected, integrally connected or detachably connected; it can be directly connected or indirectly connected through an intermediate medium. For relevant scientific research or technical personnel in this field, the specific meanings of the above terms in the present invention can be determined according to specific circumstances, and they cannot be understood as limitations on the present invention.

[0042] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0043] Embodiment 1

[0044] Embodiment 1 of the present invention introduces a power data configuration method based on dual deep reinforcement learning.

[0045] This embodiment combines the experience replay mechanism with the improved particle swarm optimization algorithm to optimize the learning efficiency and task performance of the model; it makes full use of the reinforcement learning framework of DDQN, combined with the characteristics of terminal power consumption data upload, to achieve the optimization and improvement of the adaptive configuration strategy, and improve the collection efficiency and task completion efficiency in a multi-task environment.

[0046] like Figure 1 As shown in the figure, reinforcement learning in power data collection task scheduling can be divided into two main parts: power collection environment and intelligent agent; the power collection environment includes the collection terminal cluster and power consumption information collection task, as well as feature descriptions related to data collection tasks and collection status, such as data collection task volume, collection task time requirements and channel transmission status, which are the basic information for scheduling operations. Based on reinforcement learning, the intelligent agent acts as a scheduler, assigning tasks to appropriate machines by selecting actions according to the current state, and obtaining feedback from the environment as rewards. By continuously maximizing the accumulation of rewards, the intelligent agent can eventually find the optimal scheduling sequence.

[0047] In this embodiment, the task list can be defined as:

[0048] T={t 1 ,t 2 ,t 3 ,…,t m} (1)

[0049] Among them, t i represents the i-th task, and m is the total number of tasks in the task list T. Task t i It can be expressed as a vector (l i ,m i ,p i ,d i ), where l i Represents task t i The execution time of m i , p i and d i Respectively represent tasks t i The required priority, number of collection terminals, and deadline.

[0050] In this embodiment, the acquisition terminal can be defined as:

[0051] R = {r 1 ,r 2 ,r 3 ,…,r n} (2)

[0052] Given that data collection terminals have different configurations, such as processing time, service life, transmission rate, and status, they can be j =(mips j ,serv j ,trans j ,state j ) to represent the virtual acquisition terminal r j , where j∈[1,n].

[0053] The scheduler is responsible for assigning the received tasks to the appropriate acquisition terminals. The execution time (ET) is defined as the time from the start to the completion of a specific task, that is:

[0054]

[0055] Among them, mips j Indicates the MIPS value of acquisition terminal j.

[0056] In the task scheduling of electricity consumption information collection, the maximum completion time among all collection terminals is defined as the makespan, which is used to evaluate the efficiency and quality of the task scheduling algorithm. When the number of collection terminals is limited, a lower makespan means less time required to complete the task. In task scheduling, the makespan is:

[0057] MS=max({T 0 ,T 1 ,…,T m}) (4)

[0058] Among them, T j =γ j -start(r j ),j=0,1,…,m, is the completion time of the jth acquisition terminal, γ j represents the time when the jth collection terminal completes all assigned tasks, start(r j ) is the time when the task starts. MS is the maximum completion time of all tasks, that is, the completion time. The average completion time and standard deviation of the completion time of all tasks are shown in formula (5) and formula (6) respectively:

[0059]

[0060] In this embodiment, after the collection tasks are decomposed, the collection task priorities are set and sent to each collection terminal according to the priority. Each collection terminal uploads data according to the timing sequence. When the current environment data transmission state is observed at time t, the intelligent agent selects an action according to the preset strategy (i.e., the priority order of each collection task). After the terminal completes the data upload according to the timing sequence, the current environment state S t Transfer to the next environment state S with a certain state transition probability t+1 At the same time, according to the completion time and success rate of this collection task, the agent will be given a reward R t+1 , if the reward is positive, the probability of the collection priority ranking of the collection task being selected in the next state will increase. If the reward is negative, the corresponding probability of being selected will decrease. The sequence generated by this process is: S 0 ,A0 ,R 1 ,S 1 ,A 1 ,R 2 ,S 2 ,…,S T-1 ,A T-1 ,R T ,S T .

[0061] In this embodiment, a Markov decision process (MDP) is used to model the reinforcement learning problem. Through continuous learning, the agent can continuously optimize its decisions in the interaction with the environment. The environment can be simplified by the Markov property in formula (7):

[0062] p(S t+1 |S t )=S 1 ,S 2 ,…,S t (7)

[0063] The agent in reinforcement learning consists of three key elements: action space, state space, and reward function. The agent selects and executes actions based on the state of the environment. The reward function is used to motivate the agent to adjust its actions to maximize the reward value.

[0064] (1) State space

[0065] The state space reflects the allocation and use of tasks and collection terminals. For each collection terminal state, the completion time of the currently assigned task can be calculated. The global maximum completion time is also recorded and can be used to calculate the time required by V current Represents the machine state. For the task state, the agent observes the task queue to be scheduled and can use T wait The state space is described as S = {S 1 ,S 2 ,…}, where S i =(V current ,T wait ).

[0066] (2) Action Space

[0067] The action space reflects the behavior of the scheduler when assigning tasks to the collection terminals. For each task schedule, there are m power collection terminals, and the action space is the set of serial numbers of the m power collection terminals. Therefore, the action space set can be represented by A = {1, 2, 3, ..., m}. The criteria for the agent to select actions are to minimize the completion time and maximize the collection success rate. Therefore, when the schedule is not completed, the agent can consider scheduling the task to the task scheduling sequence set with the shortest completion time and high task collection success rate, and obtain the reward value based on the environmental feedback, and then update the collection task sorting.

[0068] (3) Reward Function

[0069] The reward function is the feedback provided by the environment for the actions selected by the agent. The agent adjusts its actions based on the reward feedback obtained to better adapt to the environment.

[0070] The goal of this embodiment is to obtain the optimal completion time. A new reward function is used, as shown in formula (8):

[0071] R=-(MS(S′)-MS(S))ρ (8)

[0072] Among them, MS(S′) is the completion time of entering state S′ after executing the action in task scheduling, S is the previous state, and ρ is a constant used to adjust the reward to an appropriate range; rewards are given by slowing down the growth of completion time; therefore, the scheduler is more inclined to schedule tasks to acquisition terminals with lighter loads and lower completion times.

[0073] like Figure 2 As shown, this embodiment adopts the task scheduling model of DDQN, and the calculation of Q value and action selection are respectively placed in two different neural networks to avoid the problem that Q value is easily overestimated. The model consists of an evaluation network and a target network, and the network structures of the two are the same but the parameters are different; the role of the evaluation network is to output the Q value of each action (the mapping method from task to acquisition terminal) according to the current environment state (the state of the acquisition terminal and the task to be scheduled), and select the action with the largest Q value to execute. The target network is used to calculate the target Q value after executing the action as the learning target of the evaluation network to avoid the Q value being overestimated; during the learning process, the agent first obtains the current state from the environment, and the evaluation network outputs the Q value of each action and executes the action. If the task scheduling is completed, the target value is the actual reward value; otherwise, the target Q value is calculated by the target network as the target value; then the temporal difference (TD) target value and the Q value of the evaluation network are used for mean square error loss, and the evaluation network parameters are updated by the gradient descent algorithm.

[0074] (1) Action selection

[0075] Based on the status t+1, evaluate the Q network to find the action that maximizes the expected reward of the deep Q network (DQN), that is, action a * is the action that can produce the maximum Q value, and is defined as shown in formula (9):

[0076]

[0077] (2) Target value calculation

[0078] The target Q network calculation of the model (s t+1 ,a*), that is, the Q value corresponding to the a* action in the target network; y t Calculated by formula (10),

[0079] y t =r t +γQ(s t+1 ,a * ,θ - ) (10)

[0080] Since the target Q network and the evaluation Q network are independent of each other, a* is not necessarily the maximum value under the target Q network parameters, which can be expressed by the inequality described by formula (11); therefore, the error can be effectively reduced and the network over-estimation problem can be avoided, that is:

[0081]

[0082] The target Q value can be calculated by formula (10) in Double DQN, that is:

[0083]

[0084] The training of the evaluation Q network is performed by minimizing the mean square error (MSE) between the target value and the estimated value. Formula (13) shows the optimized MSE loss function, that is:

[0085] L(θ)=E s,a [(y t -Q(s t+1 ,a,θ)) 2 ] (13)

[0086] Perform gradient descent to update the network, and the gradient of the parameter θ is shown in formula (14):

[0087]

[0088] Evaluating Q-Network Parameters Then update. After a certain step size C, the model updates the temporal difference (TD) target network parameters to set θ -=θ, reinforcement learning and deep learning gradient descent algorithms are performed together. The experience sample data obtained by the agent is used to train the network. As a result, the network gradually approaches the action value function, and the agent is able to learn the optimal strategy.

[0089] Considering that the gradient descent method is used in network training, the gradient value can be used to evaluate the network and assist in determining whether to update the target network. In the initial stage, the value of the target network update interval is defined. Then the gradient value during network training is observed. If the gradient value is lower than the threshold, the network tends to converge and the number of steps in the update interval can be reduced. If the update interval is reached but the gradient is still large, the former can be appropriately increased. Therefore, dynamic adjustment of the target network update interval can be achieved.

[0090] In this embodiment, an experience playback control unit is introduced, specifically:

[0091] The experience storage unit is used to store previous learning experiences, because Q-learning can learn not only from current and past experiences, but also from other experiences. Therefore, the training data is stored in the replay buffer, so that the model can randomly extract previous experiences for learning, thus breaking the correlation between data.

[0092] Since in the experience data, some samples can help the network improve learning quality and accelerate convergence, while others are not important. The simple data management method of deleting old data when the queue is full does not fully utilize the value of the experience data. Therefore, before the experience playback unit, a control unit is added to mark the experience data, and a new data management method is used.

[0093] The data input to the experience playback unit carries two pieces of information that can be used to mark its value: one is the TD error δ = y t -Q(s t ,a,θ). That is, the difference between the real Q value and the estimated Q value obtained by the two networks is calculated. The training goal is to make the expectation of δ as small as possible. Therefore, δ can mark the importance of samples to a certain extent. However, due to the influence of the random environment, δ is not completely accurate and therefore cannot fully reflect the state. Another piece of information is the reward r. The principle of selecting actions in reinforcement learning is to obtain the maximum reward; the state S t By performing action A t Enter the next state S t+1 , a reward r is obtained. A high reward means that the action has high value.

[0094] Before the data is stored in the storage unit, the control unit adds a flag. This flag limits the life cycle of the sample data stored in the storage unit, thereby adjusting the probability of different data being sampled and learned. The flag converts the data into (s, a, r, s', τ age ), where τ age It can be calculated by formula (15):

[0095] τ age =L ini +rα 1 -|δ|α 2 (15)

[0096] Among them, L ini is the initial life cycle set for each data; the reward r and TD error δ are used to adjust τ age The value of α 1 and α 2 is a constant used to adjust r and δ to the appropriate range. According to the reward calculation formula, the reward is a value less than zero. The smaller the absolute value of the TD error, the more accurate the prediction performed by the neural network. τ age Indicates the life cycle length of sample data in the storage unit, through τ age Realize life management. Each time the data is sampled and learned, τ age The value of is reduced by one, when τ age When it becomes zero, the corresponding sample is deleted from the storage unit. When the storage unit is full, τ is deleted first. age Less valuable empirical data. Therefore, a kind of priority management can be indirectly achieved. The more valuable the data is, the longer its life cycle in the storage unit is, and the greater the probability of being sampled and learned. On the contrary, the probability of data with little value being sampled and learned is low.

[0097] Deep learning requires a large amount of labeled data to improve performance. However, in the early stages of reinforcement learning, experience is gained through interaction with the environment, and data is relatively scarce. Therefore, this embodiment adopts a reinforcement learning task scheduling method based on pre-training to improve algorithm performance; specifically: using an improved particle swarm optimization (PSO) algorithm to obtain the best scheduling sequence; using a deep reinforcement learning neural network to fit these optimized solutions, so that the intelligent agent can obtain prior knowledge.

[0098] The PSO algorithm initializes the position and speed of randomly generated particles, resulting in low quality of the initial particles. In order to construct a high-quality initial particle swarm, this embodiment uses the amount of data that needs to be collected when the task is executed to construct it; the collection performance of the collection terminal and the amount of data that needs to be processed when the task is executed are normalized to facilitate the subsequent selection of the most appropriate collection terminal to allocate the collection terminal; that is:

[0099]

[0100] Among them, r jcomp represents the processing performance of the jth acquisition terminal; t ilength represents the amount of data that needs to be collected when the i-th task is executed; max(r comp ) and min(r comp ) represent the acquisition terminals with the highest and lowest acquisition performance in the computing acquisition terminal combination; max(t length ) and min(t length ) represent the tasks with the largest and smallest data volumes to be executed in the task set, respectively; these values ​​are used to normalize the computing power of the computing acquisition terminal and the computing requirements of the task, and obtain Ur jcomp and Ut ilength , whose value is between 0 and 1.

[0101] Through normalization, it is convenient to preferentially assign acquisition terminals with high computing performance to tasks with high computing requirements, and randomly assign the remaining acquisition terminals and tasks to construct the initial particle swarm. This initialization method combines heuristic assignment and random assignment to ensure the quality of the initial particle swarm while retaining sufficient search space for the algorithm.

[0102] The initialization process in this embodiment is as follows Figure 3 shown.

[0103] The performance of the PSO algorithm depends on the optimized parameter settings and inertia weight. The inertia weight strikes a balance between exploration and exploitation when searching for the optimal solution in the search area. This embodiment uses a new inertia weight strategy "Linear Decreasing Adaptive Inertia Weight (LDAIW)", namely:

[0104]

[0105] Among them, ω 1 and ω 2 Represents the initial and final values ​​of the inertia weight. These two parameters determine the range of change of the inertia weight from the beginning to the end. Ps represents the population size or a certain proportional factor, which affects the adjustment speed of the inertia weight. MAXitr is the maximum number of iterations, which represents the total operation cycle of the algorithm. Itr is the current number of iterations, which represents the current stage of the algorithm operation.

[0106] The particle success rate (Ps) is calculated using formula (18), where N represents the total number of particles, Si represents the success state of the particle, and n represents the index of the particle, i.e., i = 1 to n; that is:

[0107]

[0108] Based on the load balancing mechanism, the improved PSO in this embodiment uses a new fitness function to enable particles to obtain better solutions in task scheduling; the new fitness function is:

[0109] f=σMS+(1-σ)CT var (19)

[0110] Among them, MS is the optimized target completion time; CT var is the variance of the completion time of all acquisition terminals; σ is a constant between 0 and 1, which is used to adjust the MS and CT var The optimization ratio between them; the fitness function f is optimized to evaluate the optimization objective by minimizing the value of the fitness function, which can further reduce the completion time.

[0111] The improved PSO algorithm is used to generate multiple groups of optimization solutions, which are mapping sequences of tasks to edge acquisition terminals; they can be used to pre-train the evaluation Q network of the dual DQN to obtain better prior knowledge of the environment; the intelligent agent can achieve lower completion time in task scheduling.

[0112] Case Analysis

[0113] Taking a company's electricity consumption information collection system as the experimental background platform, according to the time-of-use electricity price business and policy requirements, the electricity consumption information collection system batches down the data upload period parameters of the smart energy meter to realize minute-level electricity consumption data collection, which is used to support the user's time-of-use electricity fee settlement, and at the same time supports the real-time electricity consumption data collection of the spot market transaction business, so as to realize the synchronization of purchase and sale prices and real-time transactions. Based on the above two types of electricity consumption data collection business, more than 1,000 electricity consumption data collections of industrial and commercial users of the company from January 2022 to December 2023 were screened, including 81,000 examples of terminal electricity consumption collection sample data as sample set data. In order to verify the performance of the model, this embodiment selects task completion time and collection rate as evaluation indicators.

[0114] In this embodiment, the proposed deep Q network reinforcement learning model is compared with six baseline methods (static scheduling algorithm (First-Come, First-Served, referred to as FCFS), dynamic scheduling algorithm (Earliest Deadline Firs, referred to as EDF), biological heuristic optimization algorithm based on swarm intelligence (Ant Colony Optimization, referred to as ACO), meta-heuristic optimization algorithm based on evolutionary computation (Genetic Algorithm, referred to as GA), scheduling algorithm based on graph neural network and reinforcement learning (Graph Neural Network with Reinforcement Learning, referred to as GNN-RL) and intelligent scheduling algorithm based on deep reinforcement learning (DRLBTSA)). The results of the comparative experiment are shown in Table 1. The experimental data show that the performance of this embodiment in the power data collection task is better than other baseline models, and the collection success rate reaches 99.7%.

[0115] Compared with other methods, the success rate of the model in this embodiment is significantly improved, which is mainly due to the following key factors: First, compared with reinforcement learning models such as GNN-RL, the model in this embodiment is pre-trained before task execution, fully learning and mastering the prior knowledge in the power system acquisition task scheduling to optimize subsequent tasking. Secondly, the model performance in this embodiment is better than the traditional non-deep learning scheduling algorithm, which means that adaptively selecting the optimal scheduling scheme in a complex task environment can further improve the efficiency and success rate of task completion. In addition, the model effect in this embodiment is better than the metaheuristic algorithm, which proves that although the metaheuristic algorithm performs well in global search, it is easy to fall into the local optimal solution. In contrast, the reinforcement learning mechanism of the model in this embodiment can dynamically adjust the strategy through continuous interaction and environmental feedback, which can not only better explore the task space, but also effectively avoid the dilemma of local optimality, so as to find a better scheduling solution.

[0116] Table 1 Comparative experimental results

[0117]

[0118] To verify the contribution of each module to the model performance, this example conducts a systematic ablation experiment: taking the complete model as the benchmark, the corresponding modules are removed based on the model variants in Table 2 to evaluate their impact on the final performance. In the experiment, this example focuses on the following three modules: pre-training module (PTM), double Q mechanism (Double Q) and experience replay mechanism (ER)

[0119] The results of the ablation experiment are shown in Table 3. When the pre-training module is removed, the average task completion time of the model increases to 53 seconds. This shows that the pre-training module can effectively learn the prior knowledge in the power data acquisition task and improve the learning efficiency of the model. After removing the double Q mechanism, the task completion time of the model increases to 62 seconds, and the acquisition success rate drops to 98.8%, indicating that the double Q mechanism can effectively reduce the Q value over-estimation problem, thereby improving the stability of the model. Finally, when the experience replay mechanism is removed, the task completion time of the model is 71 seconds, and the acquisition success rate drops to 99.1%, indicating that the experience replay mechanism plays an important role in effectively utilizing historical data and improving the acquisition success rate. The ablation experiment results show that the above three modules have made significant contributions to improving the model's task scheduling performance and acquisition success rate, verifying the rationality and effectiveness of this model design.

[0120] Table 2 Model variants

[0121]

[0122] Table 3 Ablation experiment results

[0123]

[0124] This embodiment is based on the DDQN terminal power consumption data upload adaptive configuration method, by combining the experience playback mechanism and the improved PSO pre-training method, to achieve efficient scheduling and adaptive configuration of the power consumption information collection system in multiple business scenarios; make full use of the dual DQN reinforcement learning framework, effectively optimize and configure the terminal power consumption data upload, and solve the problems of reduced collection rate and delayed task completion time in the case of multi-task concurrent collection.

[0125] Combined with the example analysis, it is verified that the method proposed in this embodiment has achieved significant improvements in collection efficiency and task completion time, more accurately reflects the collection and scheduling requirements in a multi-task environment, improves the overall performance of the power consumption information collection system, and realizes the efficiency and intelligence of multi-task collaborative scheduling.

[0126] Embodiment 2

[0127] Embodiment 2 of the present invention introduces a power data configuration system based on dual deep reinforcement learning.

[0128] like Figure 4 A power data configuration system based on dual deep reinforcement learning is shown, comprising:

[0129] An acquisition module, which is configured to acquire power data acquisition tasks;

[0130] A parsing module, which is configured to parse the acquired power data collection task to obtain attribute characteristics of the power data collection task;

[0131] A configuration module, which is configured to optimize the scheduling of power data according to the obtained collection task attribute characteristics and the dual deep reinforcement learning scheduling model, and complete the configuration of the power data;

[0132] Among them, the dual deep reinforcement learning scheduling model places the calculation of Q value and the selection of action in the evaluation network and the target network respectively. The evaluation network outputs the Q value of each action according to the current environmental state of the power data and selects the maximum Q value to execute the action. The target network uses the target Q value after executing the action as the learning target of the evaluation network, and updates the evaluation network parameters in combination with the gradient descent method to complete the optimization training of the dual deep reinforcement learning scheduling model.

[0133] The detailed steps are the same as those of the power data configuration method based on dual deep reinforcement learning provided in Example 1, and will not be repeated here.

[0134] Embodiment 3

[0135] Embodiment 3 of the present invention provides a computer-readable storage medium.

[0136] A computer-readable storage medium stores a program thereon, which, when executed by a processor, implements the steps in a power data configuration method based on dual deep reinforcement learning as described in Embodiment 1 of the present invention.

[0137] The detailed steps are the same as those of the power data configuration method based on dual deep reinforcement learning provided in Example 1, and will not be repeated here.

[0138] Embodiment 4

[0139] A fourth embodiment of the present invention provides an electronic device.

[0140] An electronic device comprises a memory, a processor and a program stored in the memory and running on the processor, wherein when the processor executes the program, the steps in a power data configuration method based on dual deep reinforcement learning as described in Embodiment 1 of the present invention are implemented.

[0141] The detailed steps are the same as those of the power data configuration method based on dual deep reinforcement learning provided in Example 1, and will not be repeated here.

[0142] Embodiment 5

[0143] Embodiment 5 of the present invention provides a computer program product.

[0144] A computer program product includes software code, wherein the program in the software code executes the steps in the power data configuration method based on dual deep reinforcement learning as described in Embodiment 1 of the present invention.

[0145] The detailed steps are the same as those of the power data configuration method based on dual deep reinforcement learning provided in Example 1, and will not be repeated here.

[0146] The above description is only a preferred embodiment of the present embodiment and is not intended to limit the present embodiment. For those skilled in the art, the present embodiment may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present embodiment shall be included in the protection scope of the present embodiment.

Claims

1. A power data configuration method based on dual deep reinforcement learning, characterized in that: include: Obtain power data collection tasks; Analyze the acquired power data collection tasks to obtain attribute characteristics of the power data collection tasks; According to the acquired task attribute characteristics and the dual deep reinforcement learning scheduling model, the power data is optimized and scheduled to complete the configuration of the power data; Among them, the dual deep reinforcement learning scheduling model places the calculation of Q value and the selection of action in the evaluation network and the target network respectively. The evaluation network outputs the Q value of each action according to the current environmental state of the power data and selects the maximum Q value to execute the action. The target network uses the target Q value after executing the action as the learning target of the evaluation network, and updates the evaluation network parameters in combination with the gradient descent method to complete the optimization training of the dual deep reinforcement learning scheduling model.

2. A power data configuration method based on dual deep reinforcement learning as described in claim 1, characterized in that: The obtained reinforcement learning of the power data collection task includes a power collection environment and an intelligent agent; the power collection environment includes a collection terminal cluster and a power consumption information collection task, as well as a feature description related to the data collection task and the collection state; Based on reinforcement learning, the intelligent agent is used as a scheduler, which selects actions according to the current state to assign tasks to appropriate machines and obtains feedback from the environment as rewards. By continuously maximizing the accumulation of rewards, the intelligent agent finds the optimal scheduling sequence.

3. A power data configuration method based on dual deep reinforcement learning as described in claim 2, characterized in that: An improved particle swarm optimization algorithm is used to obtain the optimal scheduling sequence of the intelligent agent; the improved particle swarm optimization algorithm adopts a linearly decreasing adaptive inertia weight and a fitness function that considers the optimization target completion time.

4. A power data configuration method based on dual deep reinforcement learning as described in claim 2, characterized in that: The key elements of the reinforcement learning agent include action space, state space and reward function. The agent selects and executes actions according to the environment state. The reward function is used to motivate the agent to adjust actions to achieve the maximum reward value. When the scheduling is not completed, the agent considers scheduling the task to a set of task scheduling sequences with the shortest completion time and the highest task collection success rate, obtains reward values ​​based on environmental feedback, and updates the collection task sorting.

5. A power data configuration method based on dual deep reinforcement learning as described in claim 1, characterized in that: During the learning process of the target network, the agent obtains the current state from the environment, and the evaluation network outputs the Q value of each action and executes the action; If the task scheduling is completed, the target value is the actual reward value obtained; otherwise, the target network calculates the target Q value as the target value; the mean square error loss is calculated by combining the temporal difference target value and the Q value of the evaluation network, and the evaluation network parameters are updated based on the obtained mean square error loss and gradient descent method.

6. A power data configuration method based on dual deep reinforcement learning as described in claim 1, characterized in that: During the optimization training of the dual deep reinforcement learning scheduling model, an experience replay control unit for marking experience data is added to randomly extract previous experience for learning to break the correlation between data.

7. A power data configuration system based on dual deep reinforcement learning, characterized in that: include: An acquisition module, which is configured to acquire power data acquisition tasks; A parsing module, which is configured to parse the acquired power data collection task to obtain attribute characteristics of the power data collection task; A configuration module, which is configured to optimize the scheduling of power data according to the obtained collection task attribute characteristics and the dual deep reinforcement learning scheduling model, and complete the configuration of the power data; Among them, the dual deep reinforcement learning scheduling model places the calculation of Q value and the selection of action in the evaluation network and the target network respectively. The evaluation network outputs the Q value of each action according to the current environmental state of the power data and selects the maximum Q value to execute the action. The target network uses the target Q value after executing the action as the learning target of the evaluation network, and updates the evaluation network parameters in combination with the gradient descent method to complete the optimization training of the dual deep reinforcement learning scheduling model.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of a power data configuration method based on dual deep reinforcement learning as described in any one of claims 1 to 6 are implemented.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the program, the steps of a power data configuration method based on dual deep reinforcement learning as described in any one of claims 1-6 are implemented.

10. A computer program product comprising software code, characterized in that The program in the software code executes the steps of a power data configuration method based on dual deep reinforcement learning as described in any one of claims 1-6.

Images