A Low-Latency Dynamic Routing Method for Distributed Power Aggregation Control

By building a traffic prediction model and a deep learning routing decision model, combining a valued function network and a policy neural network, the problem of difficulty in obtaining the current network status in large networks is solved, and high-accuracy path selection and low-latency networks are achieved Performance significantly improves the real-time and accuracy of distributed power aggregation control.

CN118200225BActive Publication Date: 2024-11-22WUXI GUANGYING ELECTRIC POWER DESIGN CO LTD +2
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410351314.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-03-26
Publication Date
2024-11-22
Estimated Expiration
2044-03-26

AI Technical Summary

Technical Problem

Traditional routing algorithms are difficult to obtain the current network topology status in real time in large networks, resulting in the network parameters referenced during decision-making being historical data, which affects the timely issuance and real-time nature of regulatory instructions.

Method used

Build a traffic prediction model, combine valued function networks and policy neural networks, realize network traffic prediction and routing decisions through deep learning, adopt end-to-end training methods, build an experience pool and update network parameters, and optimize network parameter prediction and routing decisions.

Benefits of technology

A path selection accuracy of 90.58% is achieved, saving 14.92% network latency compared to the traditional method, reducing latency jitter, increasing throughput, and reducing packet loss rate to zero.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118200225B_ABST
    Figure CN118200225B_ABST
Patent Text Reader

Abstract

The present invention relates to the field of network routing technology, and specifically discloses a low-latency dynamic routing method for distributed power aggregation control, the method comprising constructing a traffic prediction model in network routing; the traffic prediction model is used to build a routing environment; defining and initializing a valued function network and a policy neural network, copying the valued function network and the policy neural network, and creating a target valued function network and a target policy neural network; introducing random noise, and determining an initial state based on a traffic prediction model; processing the random noise and the initial state by a policy neural network to construct an experience pool; and randomly sampling in the experience pool to train the valued function network. The present invention combines a network traffic prediction model with a routing decision model based on deep learning, and through an end-to-end training method, finally achieves a path selection accuracy of 90.58%, and can improve network delay savings by 14.92% compared to the traditional sliding average prediction method.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of network routing, and in particular to a low-latency dynamic routing method for distributed power source aggregation control. Background Art

[0002] With the advancement of the construction of new power systems, a large number of distributed photovoltaics, distributed energy storage, electric vehicle charging piles and other distributed source-load-storage energy are connected to the distribution network, which has an important impact on the safe and stable operation of the distribution network, and it is necessary to regulate the distributed source-load-storage equipment. However, the communication technology of distributed power sources is complex, and the complexity and delay uncertainty of the communication network have an adverse effect on the aggregate regulation of distributed power sources. Therefore, it is necessary to consider a low-latency dynamic routing method that is adaptive to network changes to ensure the timely issuance of control instructions and high real-time performance.

[0003] Traditional heuristic algorithms may take a long time to make decisions in actual systems. More importantly, the network status of the entire topology at the current moment of routing decision cannot be obtained in time. Especially in large-scale network topologies, the measurement of network status may also take a certain amount of time, resulting in the network status used as a reference when making decisions becoming "history".

[0004] When making routing decisions, the current network status should be referenced, but this is often difficult to do in actual systems, and the only references are often some network parameters at historical times. In such a situation, if there is an accurate prediction of network parameters, especially the traffic information of each link in the topology, it will promote more accurate routing decisions. How to optimize this "history"-based network parameter prediction process is the technical problem that the technical solution of the present invention wants to solve. Summary of the invention

[0005] The object of the present invention is to provide a low-latency dynamic routing method for distributed power source aggregation control to solve the problems raised in the above background technology.

[0006] To achieve the above object, the present invention provides the following technical solutions:

[0007] A low-latency dynamic routing method for distributed power aggregation control includes:

[0008] Step 1: construct a traffic prediction model in network routing; the traffic prediction model is used to build a routing environment;

[0009] Step 2: Define and initialize the valued function network P(S, A|θ p ) and the policy neural network μ(S|θ μ ), copy the valued function network P(S, A|θ p ) and the policy neural network μ(S|θμ ), create a target valued function network P′(S, A|θ P′ ) and the target policy neural network μ′(S|θ μ′ );θ p is the parameter of the valued function network, θ μ is the parameter of the policy neural network, θ P′ is the parameter of the target valued function network, θ u′ are the parameters of the policy neural network;

[0010] Step 3: Introduce random noise Z and determine the initial state S1 based on the traffic prediction model;

[0011] Step 4: The strategy neural network processes the random noise Z and the initial state S1 to construct the experience pool K;

[0012] Step 5: Randomly sample from the experience pool K and train the value function network;

[0013] Step 6: Return to step S300 until the cycle ends M times; M is the randomly generated number of rounds.

[0014] As a further solution of the present invention: when building a traffic prediction model in network routing, a network traffic matrix in the form of a time series is used as input, the network traffic matrix includes a delay matrix and a bandwidth matrix covering each node in the network routing topology, and the output is the traffic information of each link in the topology.

[0015] As a further solution of the present invention: for a topology with a total of P overlay network nodes, the delay matrix and the bandwidth matrix have the same shape (p+2)*(p+2), including a target server and a target user.

[0016] As a further solution of the present invention: before inputting the delay matrix and the bandwidth matrix into the traffic prediction model, the delay matrix and the bandwidth matrix are expanded into a vector form through a Flatten layer, and the traffic prediction model includes an LSTM structure containing 128 hidden states. After the matrix expanded into a vector form is processed by the LSTM structure, it is input into a fully connected network with the same length as the vector; the traffic prediction model also includes a residual network structure.

[0017] As a further solution of the present invention: the traffic prediction model adopts L1 form of Loss during training;

[0018] When the model's prediction output is Y = [Y0, Y1, ..., Y mn-1 ], the actual bandwidth delay value When , the Loss function of the prediction model is expressed as:

[0019] is the kth data in the ith data set, where the data set includes a bandwidth data set and a delay data set;

[0020] Where N is the number of data sets, m and n are the two sizes of the delay matrix and bandwidth matrix respectively;

[0021] The input bandwidth data and latency data are standardized:

[0022]

[0023]

[0024] Among them, μ B , σ B , μ D , σ D are the mean and standard deviation of the bandwidth data set and the delay data set, respectively.

[0025] As a further solution of the present invention: when there is only one coverage network node in the topology, the cross entropy loss function is used as the Loss function for training:

[0026] Where id is the index of the coverage network node, O is the output vector of the traffic prediction model, and O id The probability of the overlay network node with index id being selected.

[0027] As a further solution of the present invention: the step of processing the random noise Z and the initial state S1 by the strategy neural network to construct the experience pool K includes:

[0028] Select action A based on initial state S1 and noise Z t =μ(S t |θ μ )+Z, execute action A t And observe the reward R t and the new state S t+1 , t∈[1,T], T is the maximum number of search steps; the reward R t and the new state S t+1 Obtained from the feedback of the traffic prediction model;

[0029] Create an empty experience pool K, statistics {S, A, R, S t+1}, insert into the experience pool K.

[0030] As a further solution of the present invention: the step of randomly sampling in the experience pool K and training the valued function network includes:

[0031] Randomly sample W samples from the experience pool K {S i , A i , R i , S i+1};

[0032] Calculate the mean square error between the valued function network and the target valued function network, and update the valued function network according to the mean square error. The expression is:

[0033]

[0034] Where L is the loss function; W is the total number of summation items; y i =R i +γP′(S i+1 ,μ′(S i+1 |θ μ′ )),y i Represents the state S of the target valued function network at the next moment i+1 With action A i P value; P(S i , A i |θ p ) represents the online network's selection action A in the current state i The P value generated by the evaluation; γ is the attenuation coefficient, γ∈[0,1];

[0035] The update process of the target value function network is:

[0036] Γθ p +(1-Γ)θ P′ →θ P′ ;

[0037] Γθ μ +(1-Γ)θ μ′ →θ μ′ ;

[0038] Where Γ is the update coefficient, which takes a value of 0.001.

[0039] As a further solution of the present invention: Step S400 and Step S500 are executed in a loop until the maximum number of search steps T is reached.

[0040] As a further solution of the present invention: 80% of the data samples in the experience pool are used for training, and 20% of the data samples in the experience pool are used for evaluation.

[0041] Compared with the prior art, the beneficial effects of the present invention are as follows: the present invention combines the network traffic prediction model with the routing decision model based on deep learning, and finally achieves a path selection accuracy of 90.58% through end-to-end training, and can improve the network delay saving by 14.92% compared with the traditional sliding average prediction method; better suppresses the range and duration of delay increase, and the introduction of the algorithm improves network performance, significantly reduces delay, reduces jitter, and increases throughput. At the same time, the packet loss rate is reduced, and the minimum is zero. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention.

[0043] Figure 1 Flow chart of a low-latency dynamic routing method for distributed power aggregate control.

[0044] Figure 2 A schematic diagram of network traffic prediction of a low-latency dynamic routing algorithm that adapts to network changes in the present invention. DETAILED DESCRIPTION

[0045] In order to make the technical problems, technical solutions and beneficial effects to be solved by the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0046] Figure 1 A flow chart of a low-latency dynamic routing method for distributed power aggregate control is shown. In an embodiment of the present invention, a low-latency dynamic routing method for distributed power aggregate control is provided, the method comprising:

[0047] Step S100: constructing a traffic prediction model in network routing; the traffic prediction model is used to build a routing environment;

[0048] Step S200: define and initialize the valued function network P(S, A|θ p ) and the policy neural network μ(S|θ μ ), copy the valued function network P(S, A|θ p ) and the policy neural network μ(S|θ μ ), create a target valued function network P′(S, A|θ P′ ) and the target policy neural network μ′(S|θ μ′ );θ p is the parameter of the valued function network, θ μis the parameter of the policy neural network, θ P′ is the parameter of the target valued function network, θ μ′ are the parameters of the policy neural network;

[0049] Step S300: introducing random noise Z, and determining the initial state S1 based on the traffic prediction model;

[0050] Step S400: The strategy neural network processes the random noise Z and the initial state S1 to construct an experience pool K;

[0051] Step S500: randomly sampling in the experience pool K and training a valued function network;

[0052] Step S600: Return to step S300 until the cycle ends M times; M is the randomly generated number of rounds.

[0053] In an example of the technical solution of the present invention, step one is a pre-step, in which a traffic prediction model is constructed in network routing, and a network traffic matrix in the form of a time series is used as input. The network traffic matrix contains network traffic information between each node in the coverage network routing topology, including a delay matrix and a bandwidth matrix, which is used for accurate network parameter prediction, specifically the traffic information of each link in the topology.

[0054] Furthermore, both the delay matrix and the bandwidth matrix have the same shape (p+2)*(p+2), which contains the target server and the target user, for a topology with a total of P overlay network nodes.

[0055] Among them, before being input into the prediction model, the two matrix inputs are expanded into a vector form through a Flatten layer. The model mainly includes an LSTM structure containing 128 hidden states, and passes through a fully connected network with the same length as the input vector and combined with a residual network structure. LSTM is the most widely used time series neural network and is very suitable for processing time series tasks. The introduction of the residual structure can suppress data noise and improve the prediction effect.

[0056] Among them, the network traffic prediction model uses L1 form of loss when training. If the prediction output of the model is expressed as Y = [Y0, Y1, ..., Y mn-1 ]; Actual bandwidth delay value The Loss function of the prediction model is expressed as:

[0057] is the kth data in the ith data set, where the data set includes a bandwidth data set and a delay data set;

[0058] Where N is the number of samples, m and n are the two sizes of the delay matrix and bandwidth matrix respectively;

[0059] In order to ensure that the numerical magnitudes of bandwidth data and delay data are equal and facilitate the convergence of model training, the input bandwidth data and delay data are standardized:

[0060]

[0061]

[0062] Among them, μ B , σ B , μ D , σ D are the mean and standard deviation of the bandwidth data set and the delay data set, respectively.

[0063] The number of nodes in the overlay network routing is limited to 1. If the actual overlay network node index to be selected is id, and the output of the model is vector O, then O id The meaning of is the probability of the overlay network node with index id being selected, and Use the cross entropy loss function as the Loss function for training:

[0064] Where id is the index of the coverage network node, O is the output vector of the traffic prediction model, and O id The probability of the overlay network node with index id being selected.

[0065] The experimental environment is set up in Linux, using virtual machines Vm workstation 16.1.2, Linux system and Ubuntu16.04, and the programming environment is Anaconda3, Python3.7. A multi-host multi-stream network environment is built based on the Mininet platform, and the network topology is constructed through Python scripts. The network traffic prediction model and the routing decision model are trained separately. During the experiment, 80% of the data samples are randomly selected for training, and 20% of the data are used for evaluation. The network better suppresses the range and duration of delay increase. The introduction of the algorithm improves network performance, significantly reduces latency, reduces jitter, and increases throughput. At the same time, the packet loss rate is reduced to a minimum of zero.

[0066] As a preferred embodiment of the technical solution of the present invention, steps 2 to 5 are a network training process based on the environment provided by the traffic prediction model, specifically as follows:

[0067] Step 1: Define the valued function network P(S, A|θ p ) and the policy neural network μ(S|θ μ), initialize the weight parameters, and copy the online network to the target network P′(S, A|θ P′ ) and μ′(S|θ μ′ ), θ p is the network parameter of the valued function; θ μ is the policy neural network parameter, θ P′ and θ μ′ are the parameters of the target valued function network and the target policy neural network respectively;

[0068] The above content mentions two networks, namely the online network and the target network, and their descriptions are as follows:

[0069] "Online network" and "target network" are two different neural network structures, which play different roles in the learning process. The online network refers to the neural network that directly interacts with the environment and continuously updates its parameters during the learning process. The valued function network P(S, A|θ p ) and the policy neural network μ(S|θ μ ) are all online networks, which learn through real-time interaction with the environment and continuously adjust their parameters based on the feedback received (such as rewards).

[0070] The target network is a copy of the online network, but its parameter updates are not real-time, but at a slower speed. In deep reinforcement learning, this practice helps stabilize the learning process. P′ ) and μ′(S|θ μ′ ) is the target network, and its parameter θ P′ and θ μ′ Copied from online networks, but not updated instantly with each step of learning.

[0071] The benefit of setting up two networks is that using the target network can reduce fluctuations and instability during the learning process. By periodically copying parameters from the online network to the target network instead of updating the target network in real time, it can ensure that the target remains relatively stable during the learning process, thus facilitating the convergence of the algorithm.

[0072] Step 2: Clear the experience pool K;

[0073] Clearing the experience pool K means creating a storage space, which is not complicated.

[0074] Step 3: Introduce random noise Z and use action to explore and obtain the initial state S1;

[0075] The random noise Z and the initial state S1 are generated in the environment provided by the traffic prediction model, which belongs to an initial condition, and the noise is random.

[0076] Step 4: Select action A based on the current state and noise t =μ(S t |θ μ )+Z, execute action A t And observe the reward R t and the new state S t+1 , t∈[1,T], T is the maximum number of search steps;

[0077] Step 4 has three steps: first, action selection (A t =μ(S t |θ μ )+Z), Action A t At time t, the action is selected based on the current state S t (Initially S1), and the parameters θ of the policy neural network μ The policy network typically tries to learn which action to perform in a given state to maximize the total reward, and Z represents some kind of noise added to the actions.

[0078] Second, perform actions and observe rewards and new states. After performing action A t Afterwards, the environment (based on the environment provided by the traffic prediction model) provides a reward R t and the new state S t+1 The reward is a signal that indicates how good the action taken is in achieving the goal, which is represented by model accuracy in this application.

[0079] Third, the time step and the maximum number of search steps, t is the current time step, T is the maximum number of search steps allowed, and in one cycle, action selection and evaluation are performed continuously T times.

[0080] Step 5: Store the data {S, A, R, S′} of the agent’s interaction with the environment into the experience pool K, and randomly sample W samples {S i , A i , R i , S i+1} as training data;

[0081] The data generated in the above content are counted and stored in the experience pool K in the data structure of {S, A, R, S′}. When selecting samples, they can be directly selected from the experience pool K.

[0082] Step 6: Update the network by calculating the mean square error between the online network and the target P value network. The expression is:

[0083]

[0084] Among them, y i =R i+γP′(S i+1 ,μ′(S i+1 |θ μ′ )), where: y i Indicates the state S of the target network at the next moment i+1 With action A i P value; P(S i , A i |θ p ) represents the online network's selection action A in the current state i The P value generated by the evaluation; γ is the decay coefficient, γ∈[0,1].

[0085] The above content is an online network (valued function network P(S, A|θ p ) and the policy neural network μ(S|θ μ )) is the update process, L is the loss function, and the loss function L is a standard used to measure the difference between the network prediction value and the actual value. In this scenario, it is used to measure the difference between the prediction value of the online network and the prediction value of the target network.

[0086] The above calculation process is essentially the average of the summation, where W is the number of summation terms. This average is for all considered state-action (S i , A i ) is the average of the loss function values.

[0087] y i is an estimate of future rewards. In reinforcement learning, this usually includes the immediate reward R i and the discounted value of the estimated value of the next state; where the immediate reward R i In state S i Next, execute action A i Rewards received when.

[0088] γ is the decay coefficient used to calculate the current value of future rewards. It is a value between 0 and 1 that determines the importance of future rewards relative to immediate rewards. The closer the value is to 1, the more important the future rewards are.

[0089] P′(S i+1 ,μ′(S i+1 |θ μ′ )) is the target network's response to the next state S i+1 and the action A selected by the policy network μ′ in this state i+1 The P-value prediction is an estimate of the value of the future state.

[0090] P(S i , A i |θ p ) is the online network's response to the current state Si and action A i The P-value prediction of . It is the network's assessment of the value of the current state-action pair.

[0091] In summary, the loss function L in the above process is used to evaluate and update the parameters θ of the online network. p , in order to more accurately predict the value of each state-action pair. By minimizing this loss, we learn how to better predict the consequences of actions and thus make better decisions in reinforcement learning tasks.

[0092] Step 7: Update the target network to:

[0093] Γθ p +(1-Γ)θ P′ →θ P′ ;

[0094] Γθ μ +(1-Γ)θ μ′ →θ μ′ ;

[0095] Where, Γ is the update coefficient, which is 0.001;

[0096] Step 8: Return to step 4 and proceed to the next step after reaching the maximum search step number T;

[0097] Step 9: Return to step 3 until the cycle ends M times, where M is the randomly generated number of rounds.

[0098] Steps 3 to 9 are the outer loop, and steps 4 to 8 are the inner loop. The learning process is implemented with the help of two layers of loops.

[0099] The functions that can be achieved by the low-latency dynamic routing method for distributed power aggregation control are all completed by a computer device. The computer device includes one or more processors and one or more memories. The one or more memories store at least one program code. The program code is loaded and executed by the one or more processors to implement the low-latency dynamic routing method for distributed power aggregation control.

[0100] The processor takes out instructions from the memory one by one, analyzes the instructions, and then completes the corresponding operations according to the instruction requirements, generating a series of control commands, so that the various parts of the computer can automatically, continuously and coordinately move to become an organic whole, realize the input of programs, the input of data, and the calculation and output of results. The arithmetic operations or logical operations generated in this process are all completed by the operator; the memory includes a read-only memory (ROM), which is used to store computer programs, and a protection device is provided outside the memory.

[0101] Exemplarily, the computer program may be divided into one or more modules, one or more modules are stored in a memory and executed by a processor to implement the present invention. One or more modules may be a series of computer program instruction segments capable of implementing specific functions, and the instruction segments are used to describe the execution process of the computer program in a terminal device.

[0102] Those skilled in the art will understand that the description of the above service equipment is merely an example and does not constitute a limitation on the terminal equipment. It may include more or fewer components than described above, or a combination of certain components, or different components, for example, it may include input and output devices, network access devices, buses, etc.

[0103] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the terminal device, and uses various interfaces and lines to connect various parts of the entire staff terminal.

[0104] The memory can be used to store computer programs and / or modules. The processor can realize various functions of the terminal device by running or executing the computer programs and / or modules stored in the memory and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as an information collection template display function, a product information release function, etc.); the data storage area can store data created according to the use of the berth status display system (such as product information collection templates corresponding to different product types, product information that different product providers need to release, etc.). In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0105] If the module / unit integrated in the terminal device is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable medium. Based on this understanding, the present invention implements all or part of the modules / units in the above-mentioned embodiment system, and can also be completed by instructing the relevant hardware through a computer program. The above-mentioned computer program can be stored in a computer-readable medium, and the computer program can realize the functions of the above-mentioned various system embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. Computer-readable media may include: any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal and software distribution medium, etc.

[0106] It should be noted that, in this article, the term "comprises" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of more restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0107] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A low-latency dynamic routing method for distributed power aggregation control, It is characterized in that The method comprises: Step S100: constructing a traffic prediction model in network routing; the traffic prediction model is used to build a routing environment; Step S200: define and initialize a valued function network and policy neural network , copy the valued function network and policy neural network , create a target valued function network and target policy neural network ; are the parameters of the valued function network, are the parameters of the policy neural network, are the parameters of the target valued function network, are the parameters of the policy neural network; Step S300: Introduce random noise Z and determine the initial state based on the traffic prediction model ; Step S400: The strategy neural network is used to analyze the random noise Z and the initial state Process and build the experience pool K; Step S500: randomly sampling in the experience pool K and training a valued function network; Step S600: Return to step S300 until the cycle is completed M times; M is the randomly generated number of rounds; When building a traffic prediction model in network routing, the network traffic matrix in the form of a time series is used as input. The network traffic matrix contains the delay matrix and bandwidth matrix covering each node in the network routing topology, and the output is the traffic information of each link in the topology; For a topology with a total of P overlay network nodes, the delay matrix and the bandwidth matrix have the same shape (p+2)*(p+2), including the target server and the target user; The step of processing the random noise Z and the initial state by the strategy neural network to construct the experience pool K includes: According to the initial state and noise Z selection action +Z, execute action And observe the rewards and the new state , , T is the maximum number of search steps; the reward and the new state Obtained from the feedback of the traffic prediction model; Create an empty experience pool K, statistics , insert into the experience pool K; The steps of randomly sampling in the experience pool K and training the valued function network include: Randomly sample W samples from the experience pool K ; Calculate the mean square error between the valued function network and the target valued function network, and update the valued function network according to the mean square error. The expression is: ; Where L is the loss function; W is the total number of summation items; = + , Represents the state of the target valued function network at the next moment With action P value; Indicates the selection action of the online network in the current state The P value generated by the evaluation of is the attenuation coefficient, ; The update process of the target value function network is: In the formula is the update coefficient, and its value is 0.

001.

2. According to claim 1, the low-latency dynamic routing method for distributed power aggregation control, It is characterized in that Before inputting the delay matrix and the bandwidth matrix into the traffic prediction model, the delay matrix and the bandwidth matrix are expanded into a vector form through a Flatten layer. The traffic prediction model includes an LSTM structure containing 128 hidden states. The matrix expanded into a vector form is processed by the LSTM structure and input into a fully connected network with a length equal to the vector. The traffic prediction model also includes a residual network structure.

3. According to claim 1, the low-latency dynamic routing method for distributed power aggregate control, It is characterized in that Step S400 and step S500 are executed repeatedly until the maximum number of search steps T is reached.

4. The low-latency dynamic routing method for distributed power aggregate control according to any one of claims 1 to 3, It is characterized in that 80% of the data samples in the experience pool are used for training, and 20% of the data samples in the experience pool are used for evaluation.