Dynamic routing method and device, computer equipment, storage medium and program product

Abstract

The invention relates to a dynamic routing method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: determining routing measurement information in a local area network according to observation information of the local area network in a computing power network; the observation information at least comprises current computing power resource information of the computing power network, current network topological structure information of the local area network and current network state information; regenerating a routing decision table of the local area network according to the routing metric information; and determining a target transmission path corresponding to the to-be-transmitted data in the local area network according to the regenerated routing decision table. By adopting the method, the data transmission efficiency of the computing power network can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a dynamic routing method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] The computing power network interconnects dynamically distributed computing resources based on network connections. Through the flexible scheduling of multi-dimensional resources such as computing, network, and storage, it achieves global optimization of connections and computing power in the network, bringing new opportunities for the development of the computing industry for users and operators.

[0003] In traditional technology, static routing is generally used for data transmission in computing power networks. However, with the development and advancement of network technology, a large number of network devices are connected to the network, including a large number of mobile network access devices, which directly lead to frequent changes in the network topology. If static routing is still used, it will lead to slower overall data transmission, which in turn leads to lower data transmission efficiency in the computing power network. Summary of the invention

[0004] Based on this, it is necessary to provide a dynamic routing method, device, computer equipment, computer-readable storage medium and computer program product that can improve the data transmission efficiency of the computing power network in response to the above-mentioned technical problems.

[0005] In a first aspect, the present application provides a dynamic routing method, comprising:

[0006] Determine routing metric information within the local area network according to observation information of the local area network in the computing power network; the observation information at least includes current computing power resource information of the computing power network, current network topology information of the local area network, and current network status information;

[0007] Regenerating a routing decision table of the local area network according to the routing metric information;

[0008] According to the regenerated routing decision table, a target transmission path corresponding to the data to be transmitted in the local area network is determined.

[0009] In one embodiment, regenerating the routing decision table of the local area network according to the routing metric information includes:

[0010] Determining routing metric information of a transmission path to each destination address based on routing metric information within the local area network;

[0011] Filtering out, from the transmission paths reaching the respective destination addresses, the transmission paths whose routing metric information meets a preset condition as target transmission paths reaching the respective destination addresses;

[0012] The routing decision table of the local area network is regenerated according to the target transmission path to reach each destination address.

[0013] In one embodiment, determining routing metric information within the local area network according to observation information of the local area network in the computing power network includes:

[0014] Inputting the observation information into the trained neural network model of the agent corresponding to the local area network to obtain a link weight adjustment matrix within the local area network;

[0015] Determining link weight data within the local area network according to the link weight adjustment matrix;

[0016] Routing metric information within the local area network is determined based on the link weight data.

[0017] In one embodiment, determining the link weight data within the local area network according to the link weight adjustment matrix includes:

[0018] Obtaining initial link weight data within the local area network;

[0019] Determining, according to the link weight adjustment matrix, an adjustment value of the initial link weight data within the local area network;

[0020] The initial link weight data is adjusted according to the adjustment value of the initial link weight data to obtain the link weight data within the local area network.

[0021] In one embodiment, the trained neural network model of the agent corresponding to the local area network is trained in the following manner:

[0022] Obtaining first sample observation information of an intelligent agent corresponding to a local area network in the computing power network and first sample action information corresponding to the first sample observation information, as well as second sample observation information of intelligent agents corresponding to other local area networks in the computing power network and second sample action information corresponding to the second sample observation information;

[0023] According to the first sample observation information, the first sample action information, the second sample observation information and the second sample action information, the neural network model to be trained of the intelligent agent corresponding to the local area network is trained to obtain the first initial update parameter of the neural network model to be trained of the intelligent agent corresponding to the local area network, and the first reward information of the intelligent agent corresponding to the local area network;

[0024] According to the second initial update parameter of the neural network model to be trained of the intelligent agent corresponding to the other local area network, the second reward information of the intelligent agent corresponding to the other local area network and the first reward information, the first initial update parameter is updated until the preset number of training times is reached to obtain the trained neural network model of the intelligent agent corresponding to the local area network.

[0025] In one embodiment, the updating of the first initial update parameter according to the second initial update parameter of the neural network model to be trained of the agent corresponding to the other local area network, the second reward information of the agent corresponding to the other local area network, and the first reward information comprises:

[0026] Normalizing the first reward information and the second reward information to obtain processed first reward information and processed second reward information;

[0027] According to the processed second reward information, a fusion process is performed on the difference between the second initial update parameter and the first initial update parameter to obtain a fusion update parameter;

[0028] The first initial update parameter is updated according to the fusion update parameter to obtain the first target update parameter of the neural network model to be trained of the intelligent agent corresponding to the local area network.

[0029] In one embodiment, before determining the routing metric information inside the local area network according to the observation information of the local area network in the computing power network, it also includes:

[0030] Receive the current network topology information of the local area network perceived by the server, and receive the current computing power resource information of the computing power network perceived by the computing power router;

[0031] Acquire the current network status information of the local area network perceived by the agent corresponding to the local area network, and combine the current computing resource information, the current network topology information and the current network status information to obtain the observation information of the local area network

[0032] In a second aspect, the present application also provides a dynamic routing device, including:

[0033] An information determination module, configured to determine routing metric information within a local area network according to observation information of the local area network in the computing power network; the observation information at least includes current computing power resource information of the computing power network, current network topology information of the local area network, and current network status information;

[0034] A regeneration module, used to regenerate the routing decision table of the local area network according to the routing metric information;

[0035] The path determination module is used to determine the target transmission path corresponding to the data to be transmitted in the local area network according to the regenerated routing decision table.

[0036] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0037] Determine routing metric information within the local area network according to observation information of the local area network in the computing power network; the observation information at least includes current computing power resource information of the computing power network, current network topology information of the local area network, and current network status information;

[0038] Regenerating a routing decision table of the local area network according to the routing metric information;

[0039] According to the regenerated routing decision table, a target transmission path corresponding to the data to be transmitted in the local area network is determined.

[0040] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0041] Determine routing metric information within the local area network according to observation information of the local area network in the computing power network; the observation information at least includes current computing power resource information of the computing power network, current network topology information of the local area network, and current network status information;

[0042] Regenerating a routing decision table of the local area network according to the routing metric information;

[0043] According to the regenerated routing decision table, a target transmission path corresponding to the data to be transmitted in the local area network is determined.

[0044] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0045] Determine routing metric information within the local area network according to observation information of the local area network in the computing power network; the observation information at least includes current computing power resource information of the computing power network, current network topology information of the local area network, and current network status information;

[0046] Regenerating a routing decision table of the local area network according to the routing metric information;

[0047] According to the regenerated routing decision table, a target transmission path corresponding to the data to be transmitted in the local area network is determined.

[0048] The above-mentioned dynamic routing method, apparatus, computer equipment, computer-readable storage medium and computer program product determine the routing metric information inside the local area network based on the observation information of the local area network in the computing power network; the observation information includes at least the current computing power resource information of the computing power network, the current network topology structure information of the local area network and the current network status information; then, based on the routing metric information, the routing decision table of the local area network is regenerated, and finally, based on the regenerated routing decision table, the target transmission path corresponding to the data to be transmitted in the local area network is determined. In this way, when data is transmitted in the computing power network, the current computing power resource information of the computing power network, the current network topology information of the local area network and the current network status information of the local area network are comprehensively considered, and the routing decision table of the local area network is regenerated based on these observation information, thereby achieving the purpose of dynamically changing the routing decision; then, based on the regenerated routing decision table, the target transmission path corresponding to the data to be transmitted in the local area network is determined, so that the data to be transmitted in the local area network can be transmitted according to the corresponding latest determined target transmission path, so that the data to be transmitted in the local area network can be transmitted according to the adapted target transmission path at all times, thereby achieving the effect of dynamic routing, which is conducive to speeding up the overall data transmission speed of the computing power network, thereby improving the data transmission efficiency of the computing power network, and at the same time avoiding the use of static routing, which leads to slower overall data transmission and lower data transmission efficiency of the computing power network. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0050] Figure 1 A schematic diagram of a flow chart of a dynamic routing method in one embodiment;

[0051] Figure 2 A flowchart of the steps of determining routing metric information within a local area network in one embodiment;

[0052] Figure 3 A flowchart of the training steps of a neural network model in which the training of an agent corresponding to a local area network is completed in one embodiment;

[0053] Figure 4 A schematic diagram of a flow chart of a dynamic routing method in another embodiment;

[0054] Figure 5It is a structural block diagram of a distributed intelligent routing system for a computing power network in one embodiment;

[0055] Figure 6 is a structural block diagram of a dynamic routing device in one embodiment;

[0056] Figure 7 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0058] As computing networks have developed, users' computing power needs have become increasingly diverse and large-scale. At the same time, a large number of heterogeneous computing resources have also participated in forming a modern new computing network environment. In this process, operators need to provide users with good network service quality. Although the standard of network service quality depends on the specific scenario, routing is always an important part that affects network service quality.

[0059] The main content of routing is to select a suitable path for data packets passing through the network. Static route allocation is the most primitive routing method. As long as the network topology is stable, the static routing table will be stable. However, with the development and progress of network technology, a large number of network devices are connected to the network, including a large number of mobile network access devices. They directly lead to the rapid growth of network scale and frequent changes in network topology, which has caused great difficulties in the application of traditional static routing methods. Dynamic routing optimization methods have emerged.

[0060] The dynamic routing optimization method needs to consider not only the dynamic changes of the network structure, but also the changes in network status (such as jitter, bandwidth, packet loss rate, etc.) and computing power resources (such as CPU, GPU, memory). The network structure can be perceived by the master control module, the changes in network status within the local area network can be perceived by the distributed module, and the changes in computing power resources can be perceived by the computing power routing. The dynamic routing optimization method dynamically changes the routing decision according to the above changes to provide good network service quality. Based on this, the present application proposes a dynamic routing method, which dynamically generates a routing decision table to improve the data transmission efficiency of the computing power network.

[0061] In order to more clearly illustrate the dynamic routing method provided in the embodiment of the present application, some terms are explained as follows:

[0062] Multi-agent DDPG: Also known as MADDPG (Muli-Agent Deep Deterministic Policy Gradient), it is a reinforcement learning algorithm that aims to enable multiple agents to optimize their own strategies while sharing global information through a centralized training and decentralized execution architecture. During training, agents can access the states and actions of other agents and evaluate their behaviors with a centralized Critic network (part of the Q-Learning algorithm based on value functions); while at execution, each agent relies only on its own local observations to make decisions. MADDPG effectively solves the problem of multi-agent collaboration or competition by retaining the independence of each agent and ignoring unnecessary complex interactions. It is suitable for complex tasks in high-dimensional continuous action spaces.

[0063] CTDE (Centralized Training with Decentralized Execution): It is a multi-agent reinforcement learning architecture that allows agents to share global information during the training phase to improve the efficiency of policy learning; in the execution phase, each agent makes independent decisions based on its own local observations, thus adapting to distributed and dynamic environmental requirements. Through this architecture, CTDE not only ensures the ability of agents to collaborate, but also enhances the flexibility and robustness of the system. It is an effective way to solve the problems of collaboration and competition in multi-agent systems.

[0064] In an exemplary embodiment, Figure 1 As shown, a dynamic routing method is provided. This embodiment uses the method applied to a terminal as an example for illustration. It is understandable that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. Among them, the terminal can be but is not limited to various personal computers, laptops, smart phones, tablet computers, etc.; the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services. In this embodiment, the following steps S101 to S103 are included. Among them:

[0065] Step S101, determine the routing metric information inside the local area network based on the observation information of the local area network in the computing power network; the observation information at least includes the current computing power resource information of the computing power network, the current network topology structure information of the local area network and the current network status information.

[0066] The computing network includes multiple local area networks; for example, Figure 5 The computing power network includes LAN A, LAN B... LAN X.

[0067] Among them, the observation information of the local area network includes a variety of information, namely the current computing power resource information of the computing power network, the current network topology structure information of the local area network, and the current network status information.

[0068] Among them, the current computing power resource information of the computing power network refers to the computing power resource information of the computing power network at the current moment, specifically including CPU (Central Processing Unit), GPU (Graphics Processing Unit), memory, etc.

[0069] The current network status information of the LAN refers to the network status information of the LAN at the current moment, including jitter, bandwidth, packet loss rate, delay, etc.

[0070] The current network topology structure information of the local area network refers to the network topology structure information of the local area network at the current moment.

[0071] The routing metric information refers to the routing metric value of the transmission path within the local area network. Each transmission path corresponds to a routing metric value, and the routing metric value is re-determined based on the observation information of the local area network.

[0072] The routing metric represents the cost of the transmission path within the LAN, specifically the cost of reaching the destination address indicated by the transmission path, and is an important basis in the routing selection process. In actual scenarios, the smaller the routing metric of the transmission path, the more preferred it is; and, among multiple transmission paths to the same destination address, the transmission path with the smallest corresponding routing metric will be added to the routing decision table corresponding to the LAN.

[0073] Among them, routing metrics can be expressed not only by cost, but also by number of hops, bandwidth, delay, load, reliability, maximum transmission unit, etc.

[0074] Exemplarily, a terminal (such as a control device in a local area network) obtains observation information of a local area network in a computing power network, such as current computing power resource information of the computing power network, current network topology information of the local area network, and current network status information; the observation information is then input into a trained neural network model deployed by an intelligent agent corresponding to the local area network, and the observation information is processed by the neural network model to obtain a link weight adjustment matrix within the local area network; finally, based on the link weight adjustment matrix within the local area network, routing metric information within the local area network is obtained, thereby achieving the purpose of adjusting the routing metric information.

[0075] Step S102: regenerate the routing decision table of the local area network according to the routing metric information.

[0076] Each LAN corresponds to a routing decision table. The routing decision table of the LAN includes target transmission paths (i.e., optimal transmission paths) to different destination addresses, such as the optimal transmission path L1 to destination address M1, the optimal transmission path L2 to destination address M2, the optimal transmission path L3 to destination address M3, etc.

[0077] It should be noted that when the observation information of the LAN changes, the terminal will regenerate the routing decision table of the LAN, thereby achieving the purpose of dynamic routing. Moreover, based on the observation information of each LAN, the routing metric information inside each LAN can be determined, and then the routing decision table of each LAN can be regenerated according to the routing metric information inside each LAN.

[0078] Exemplarily, the terminal determines target transmission paths to different destination addresses according to routing metric information within the local area network, and regenerates a routing decision table of the local area network according to the target transmission paths to different destination addresses.

[0079] Step S103, determining the target transmission path corresponding to the data to be transmitted in the local area network according to the regenerated routing decision table.

[0080] The data to be transmitted carries a destination address, and according to the destination address, a corresponding target transmission path can be found from the routing decision table.

[0081] The target transmission path corresponding to the data to be transmitted refers to the target transmission path that reaches the target address carried in the data to be transmitted.

[0082] Exemplarily, the terminal queries the regenerated routing decision table according to the destination address carried by the data to be transmitted in the local area network, obtains the target transmission path to reach the destination address, and uses it as the target transmission path corresponding to the data to be transmitted. Furthermore, the terminal can also transmit the data to be transmitted according to the target transmission path corresponding to the data to be transmitted, so that the data to be transmitted is transmitted to the destination address through the target transmission path.

[0083] For example, assuming that the destination address carried by the data to be transmitted in the local area network is M1, and the target transmission path to the destination address M1 in the regenerated routing decision table is L1, then the target transmission path corresponding to the data to be transmitted is L1.

[0084] In the above-mentioned dynamic routing method, the routing metric information inside the local area network is determined based on the observation information of the local area network in the computing power network; the observation information at least includes the current computing power resource information of the computing power network, the current network topology structure information of the local area network and the current network status information; then, based on the routing metric information, the routing decision table of the local area network is regenerated, and finally, based on the regenerated routing decision table, the target transmission path corresponding to the data to be transmitted in the local area network is determined. In this way, when data is transmitted in the computing power network, the current computing power resource information of the computing power network, the current network topology information of the local area network and the current network status information of the local area network are comprehensively considered, and the routing decision table of the local area network is regenerated based on these observation information, thereby achieving the purpose of dynamically changing the routing decision; then, based on the regenerated routing decision table, the target transmission path corresponding to the data to be transmitted in the local area network is determined, so that the data to be transmitted in the local area network can be transmitted according to the corresponding latest determined target transmission path, so that the data to be transmitted in the local area network can be transmitted according to the adapted target transmission path at all times, thereby achieving the effect of dynamic routing, which is conducive to speeding up the overall data transmission speed of the computing power network, thereby improving the data transmission efficiency of the computing power network, and at the same time avoiding the use of static routing, which leads to slower overall data transmission and lower data transmission efficiency of the computing power network.

[0085] In an exemplary embodiment, the above step S102 regenerates the routing decision table of the local area network based on the routing metric information, and specifically includes the following contents: determining the routing metric information of the transmission path to each destination address based on the routing metric information inside the local area network; filtering out the transmission path whose routing metric information meets the preset conditions from the transmission path to each destination address respectively, as the target transmission path to each destination address; and regenerating the routing decision table of the local area network based on the target transmission path to each destination address.

[0086] The routing metric information within the local area network includes routing metric information of the transmission path to each destination address.

[0087] There are multiple transmission paths to the same destination address. For example, the transmission paths to the destination address M1 include L1, L4, L5, L6, and the like.

[0088] The routing metric information meeting the preset condition means that the value corresponding to the routing metric information (ie, the routing metric value) is the smallest.

[0089] For each local area network, the regenerated routing decision table includes the target transmission path to each destination address.

[0090] Exemplarily, the terminal determines the routing metric value of the transmission path to each destination address based on the routing metric value within the local area network, and then selects the transmission path with the smallest routing metric value from the transmission path to each destination address as the target transmission path to each destination address; finally, the historical target transmission path to each destination address in the historical routing decision table of the local area network is replaced with the target transmission path to each destination address, so as to obtain a new routing decision table of the local area network.

[0091] For example, the transmission paths to the destination address M1 include L1, L4, L5, and L6, and the routing metric value of the transmission path L1 is the smallest, which means that the optimal transmission path to the destination address M1 is L1; and so on, the optimal transmission path L2 to the destination address M2 and the optimal transmission path L3 to the destination address M3 can be obtained; based on the optimal transmission path L1 to the destination address M1, the optimal transmission path L2 to the destination address M2, and the optimal transmission path L3 to the destination address M3, the routing decision table of the local area network can be regenerated.

[0092] In this embodiment, the routing decision table of the local area network is regenerated based on the routing metric information within the local area network, thereby achieving the purpose of dynamically changing the routing decision, which is beneficial to improving the determination accuracy of the routing decision table, so that the data to be transmitted in the local area network can be transmitted according to the adapted target transmission path at all times, thereby accelerating the overall data transmission speed of the computing power network.

[0093] In an exemplary embodiment, Figure 2 As shown, the above step S101, based on the observation information of the local area network in the computing power network, determines the routing metric information inside the local area network, including the following steps S201 to S203. Among them:

[0094] Step S201, input the observation information into the trained neural network model of the agent corresponding to the local area network to obtain the link weight adjustment matrix inside the local area network.

[0095] Step S202: determining link weight data within the local area network according to the link weight adjustment matrix.

[0096] Step S203: determining routing metric information within the local area network according to the link weight data.

[0097] Each local area network corresponds to an intelligent agent, and each intelligent agent is deployed with a trained neural network model, such as a reinforcement learning model; the trained neural network model includes a value function evaluation network, a target evaluation network, an action strategy network, and a target action strategy network. Figure 5, LAN A corresponds to agent 1, LAN B corresponds to agent 2... LAN X corresponds to agent N, and agent 1, agent 2... agent N are all deployed with a trained neural network model.

[0098] The trained neural network model refers to a network model that outputs a link weight adjustment matrix within the local area network based on input observation information of the local area network. The link weight adjustment matrix within the local area network serves as an output action of the trained neural network model.

[0099] The local area network includes multiple links, and each link corresponds to an initial link weight data, such as 0.5, 0.6, etc., which is used to represent the cost or overhead of each link.

[0100] The link weight adjustment matrix includes the adjustment value of the initial link weight data of each link in the local area network, such as 0.1, 0.2, etc.

[0101] The link weight data within the local area network includes link weight data of each link within the local area network; the link weight data of each link within the local area network is used to represent the adjusted cost or adjusted overhead of each link.

[0102] Among them, within the local area network, each transmission path includes multiple links. For example, the transmission path L1 to the destination address M1 is a→b→c→d, which includes a total of 3 links, namely a→b, b→c and c→d.

[0103] Among them, within the local area network, the routing metric information of the transmission path to the destination address is determined comprehensively based on the link weight data corresponding to the links included in the transmission path. For example, the routing metric information of the transmission path to the destination address is equal to the sum of the link weight data corresponding to the links included in the transmission path.

[0104] Exemplarily, the terminal inputs the observation information into the trained neural network model of the intelligent agent corresponding to the local area network, and predicts and processes the observation information through the trained neural network model to obtain the corresponding output action, that is, the link weight adjustment matrix within the local area network; then, according to the link weight adjustment matrix within the local area network, the adjustment value of the initial link weight data of the link within the local area network is determined, and the initial link weight data of the link within the local area network is adjusted according to the adjustment value of the initial link weight data of the link within the local area network to obtain the link weight data corresponding to the link within the local area network; finally, from the link weight data corresponding to the link within the local area network, the link weight data corresponding to the link included in the transmission path to each destination address is screened out, and the link weight data corresponding to the link included in the transmission path to each destination address are added respectively to obtain the weight data of the transmission path to each destination address, which is used as the routing metric information of the transmission path to each destination address, and the routing metric information within the local area network is obtained based on the routing metric information of the transmission path to each destination address.

[0105] For example, the transmission path L1 to the destination address M1 is a→b→c→d, which includes 3 links in total, namely a→b, b→c and c→d, and the initial link weight data corresponding to these 3 links are 0.5, 0.6, and 0.7, respectively, and the corresponding adjustment values ​​are +0.1, -0.3, and +0.1, respectively. Then the link weight data corresponding to these 3 links are 0.6, 0.3, and 0.8, respectively. Then the routing metric value of the transmission path L1 to the destination address M1 = 0.6+0.3+0.8=1.7.

[0106] In this embodiment, when determining the routing metric information within the local area network, the current computing power resource information of the computing power network, the current network topology information of the local area network and the current network status information are comprehensively considered, and combined with the trained neural network model, which is beneficial to improve the accuracy of determining the routing metric information within the local area network.

[0107] In an exemplary embodiment, the above step S202 determines the link weight data within the local area network based on the link weight adjustment matrix, and specifically includes the following contents: obtaining the initial link weight data within the local area network; determining the adjustment value of the initial link weight data within the local area network based on the link weight adjustment matrix; adjusting the initial link weight data based on the adjustment value of the initial link weight data to obtain the link weight data within the local area network.

[0108] The initial link weight data within the LAN includes the initial link weight data of each link in the LAN.

[0109] Exemplarily, the terminal obtains the initial link weight data within the local area network, and determines the initial link weight data of each link in the local area network based on the initial link weight data within the local area network; then queries the link weight adjustment matrix to obtain the adjustment value of the initial link weight data of each link in the local area network; finally, the initial link weight data of each link in the local area network is added to the adjustment value of the initial link weight data to obtain the link weight data of each link in the local area network, and based on the link weight data of each link in the local area network, the link weight data within the local area network is obtained.

[0110] For example, the initial link weight data of the link a→b is 0.5, and the adjustment value is 0.2, then the initial link weight data of the link a→b=0.5+0.2=0.7.

[0111] In this embodiment, the adjustment value of the initial link weight data within the local area network is determined according to the link weight adjustment matrix, and the initial link weight data is adjusted according to the adjustment value of the initial link weight data to obtain the link weight data within the local area network, thereby achieving the purpose of dynamically updating the link weight data within the local area network, which is conducive to making the routing metric information determined based on the link weight data within the local area network more accurate.

[0112] In an exemplary embodiment, Figure 3 As shown, the dynamic routing method provided by the present application also includes a training step of a neural network model of an intelligent agent corresponding to the local area network, specifically including the following steps S301 to S303. Among them:

[0113] Step S301, obtain the first sample observation information of the intelligent agent corresponding to the local area network in the computing power network and the first sample action information corresponding to the first sample observation information, as well as the second sample observation information of the intelligent agent corresponding to other local area networks in the computing power network and the second sample action information corresponding to the second sample observation information.

[0114] Step S302, based on the first sample observation information, the first sample action information, the second sample observation information and the second sample action information, the neural network model to be trained of the intelligent agent corresponding to the local area network is trained to obtain the first initial update parameters of the neural network model to be trained of the intelligent agent corresponding to the local area network, and the first reward information of the intelligent agent corresponding to the local area network.

[0115] Step S303, based on the second initial update parameters of the neural network model to be trained of the intelligent agent corresponding to other local area networks, the second reward information of the intelligent agent corresponding to other local area networks and the first reward information, the first initial update parameters are updated until the preset number of training times is reached to obtain the trained neural network model of the intelligent agent corresponding to the local area network.

[0116] Among them, the neural network models to be trained include a value function evaluation network, a target evaluation network, an action strategy network, and a target action strategy network.

[0117] Among them, the first sample observation information of the intelligent agent corresponding to the local area network includes sample computing power resource information of the computing power network, sample network topology structure information of the local area network and sample network status information.

[0118] The first sample action information refers to the output action of the intelligent agent corresponding to the local area network, such as the sample link weight adjustment matrix within the local area network.

[0119] Among them, other LANs in the computing power network refer to LANs other than the LAN in the computing power network; for example, refer to Figure 5 , the LAN in the computing power network is LAN A, and the other LANs are LAN B... LAN X.

[0120] Among them, the second sample observation information of the intelligent agent corresponding to other local area networks includes sample computing power resource information of the computing power network, sample network topology structure information of other local area networks and sample network status information.

[0121] The second sample action information refers to the output action of the intelligent agent corresponding to other local area networks, such as the sample link weight adjustment matrix within other local area networks.

[0122] Among them, the first initial update parameter of the neural network model to be trained of the intelligent agent corresponding to the local area network refers to the initial model update parameter of the neural network model to be trained of the intelligent agent corresponding to the local area network.

[0123] Among them, the first reward information refers to the return information of the intelligent agent corresponding to the local area network, which is used to represent the average communication delay and packet loss rate of the network.

[0124] Among them, the second initial update parameter of the neural network model to be trained of the intelligent agent corresponding to other local area networks refers to the initial model update parameter of the neural network model to be trained of the intelligent agent corresponding to other local area networks.

[0125] Among them, the second reward information refers to the return information of the intelligent agent corresponding to other local area networks, which is used to indicate the average communication delay and packet loss rate of the network.

[0126] The preset number of training times refers to the total number of training times that is preset.

[0127] It should be noted that when training the neural network model to be trained of the intelligent agent corresponding to the local area network, the first sample observation information, the first sample action information, the second sample observation information and the second sample action information are comprehensively considered.

[0128] Exemplarily, the terminal obtains first sample observation information of an intelligent agent corresponding to a local area network in the computing power network, and second sample observation information of intelligent agents corresponding to other local area networks in the computing power network, and inputs the first sample observation information into the neural network model to be trained of the intelligent agent corresponding to the local area network to obtain first sample action information corresponding to the first sample observation information, and inputs the second sample observation information into the neural network model to be trained of the intelligent agent corresponding to other local area networks to obtain second sample action information corresponding to the second sample observation information; then, when the first sample observation information, the second sample observation information, and the second sample action information are determined, the first sample action information is executed by the intelligent agent corresponding to the local area network to obtain the next state information of the intelligent agent corresponding to the local area network, and the first reward information of the intelligent agent corresponding to the local area network, and then, according to the next state information and the first reward information of the intelligent agent corresponding to the local area network, the neural network model to be trained of the intelligent agent corresponding to the local area network is determined. first initial update parameters; at the same time, when the first sample observation information, the second sample observation information and the first sample action information are determined, the second sample action information is executed by the intelligent agent corresponding to the other local area network to obtain the next state information of the intelligent agent corresponding to the other local area network and the second reward information of the intelligent agent corresponding to the other local area network, and then the second initial update parameters of the neural network model to be trained of the intelligent agent corresponding to the other local area network are determined according to the next state information and the second reward information of the intelligent agent corresponding to the other local area network; finally, according to the second initial update parameters, the second reward information and the first reward information, the first initial update parameters are updated to obtain the updated neural network model of the intelligent agent corresponding to the local area network, and the above training process is repeated until the preset number of training times is reached, then the training is stopped, and the updated neural network model of the intelligent agent corresponding to the local area network when the preset number of training times is reached is used as the neural network model of the trained intelligent agent corresponding to the local area network.

[0129] It should be noted that the training process of the neural network model of the intelligent agent corresponding to other local area networks is consistent with the training process of the neural network model of the intelligent agent corresponding to the local area network, and will not be repeated here.

[0130] For example, for a certain agent in a multi-agent system, the environment explored by the agent itself is constantly changing due to the influence of the behavior of other agents: when the strategies of other agents are different, the law of environmental change is also different. The law of environmental change is expressed as the state transition probability P, the current agent is c, and the corresponding strategy is π c , the total number of agents is N, and the mathematical expression of non-stationarity is as follows. This formula is i ≠π i ' is established.

[0131] P(s′c |s c , a c ,π1,...π c , ...π N )≠P(s′ c |s c , a c ,π′1,...π′ c , ...π′ N )

[0132] Reinforcement learning algorithms are designed to explore the hidden laws of the environment. Traditional reinforcement learning algorithms operate in a stable environment, while in multi-agent algorithms, the exploration object of a single agent is actually a combination of the environment and other agents. If the problem of environmental non-stationarity is not solved, then the quadruple (s i ,a i ,s i ',r) and the current time's four-tuple environmental rules are different, and mechanisms such as experience playback are not applicable.

[0133] There are three main solutions to the problem of environmental non-stationarity: opponent modeling, direct communication, and centralized training and decentralized execution (CTDE). In the opponent modeling method, the agent learns the behavior of other agents while learning the strategy. It has the strongest independence between agents and only needs to know the actions and behavior goals of other agents for training. It occupies less bandwidth and takes a long time to learn. The direct communication method designs the communication channel between agents. The agents communicate with each other through the channel during the training process. The content of the communication needs to be designed according to the specific situation. The algorithm scalability is generally weak. In the centralized training and decentralized execution method, the agents share training data, learn with the information obtained by all agents, make independent decisions during execution, and require the data of all agents for training. It occupies more bandwidth, but the learning efficiency is higher.

[0134] In this method, an intelligent agent is deployed in each private network area, and the intelligent agent manages the network area to which it belongs. The intelligent agent independently makes decisions and is mainly responsible for routing and adjusting network load. The multi-node network environment can be represented by an undirected graph G = {G1,G 2, ...G N}={V1,E1,V2,E2...V N ,E N}, where V i represents the set of nodes in private network area i, E iRepresents the set of links between nodes in the area. The communication behavior of the routing device consists of a series of data packet sending behaviors P(V s ,V d ) sending behavior, where V s is the sending node of the data packet, V d It is the receiving node of the data packet. In the overall network, the data packet sending behavior can occur between any two network nodes.

[0135] The processing capacity of the node is modeled by the node parallel capacity, the size of the node waiting queue, the queue discard strategy and the packet size; the link quality is modeled by the link available computing power and bandwidth. Multi-agent algorithms usually generalize POMDPs (Partially Observable Markov Decision Processes) into a random game model, generally using a tuple (N, S, A1, A2, ... A N ,P,r1,r2,...r N ,γ) description, where N is the total number of agents deployed in the environment, S is the model environment state space, and A i It is used to represent the action space of agent i, and the joint action space is recorded as A = A1×A2×...×A N , P represents the probability of state space transition, the dimension is equal to S×A×S, r i is the reward function of agent i, and γ is the discount factor.

[0136] The characteristic of the random game model is that multiple agents choose their own actions and work together to trigger the state transition of the environment, obtain the next state and reward, and each agent can have an independent reward function, which is directly related to the agent's goal. In a multi-agent environment, the agent's goals may conflict with each other. In the process of maximizing its own benefits, the agent may reduce the benefits of other agents. Therefore, the random game model can be divided into different types such as cooperation and confrontation according to the scenario.

[0137] Agent i takes the network topology and the link quality information of the network area to which it belongs as state observation S i ,The perception of the global network topology is conducive to the algorithm to make reasonable communication decisions: for communication between nodes in its own area, it can make effective path selection; for cross-regional communication, it can be used as a basis for selecting the sending direction. The link weight adjustment matrix of its own area is used as the output action a i The reward function of the agent is the average communication delay and packet loss rate of the network.

[0138] Each agent i contains 4 networks: value function evaluation network Q i (S,A|θi1 ), target evaluation network Q i '(S,A|θ i2 ), Action Strategy Network Act i (S|u i1 ), target action strategy network Act i '(S|u i2 ). The agent i that manages the specified area of ​​the network perceives the network topology and obtains the observation result S i , and use this local observation as the action strategy network Act i (S|u i1 ) input, and obtain the action a selected by the strategy in this time slot i , during the exploration process, add noise of corresponding dimension ou to the action i ,Noise is generated using the OU process.,Each agent selects an action, namely, the weight adjustment matrix of its own network area, and sets the logic of sending data packets by the network nodes in the area according to the matrix.,After the agent performs the action selection, it obtains a set of network link weight data, and then uses the constrained shortest path algorithm to calculate the minimum cost path.

[0139] Each network node performs actions independently without knowing each other, and the overall impact on the environment can be regarded as performing a joint action A = (a i ,a2,...a N ), the overall routing strategy of the environment changes, and a round of communication is performed. After the communication process is completed, agent i perceives again and obtains a new local observation result S i ', and calculate its own reward function r based on the result i , the quaternion (s i ,a i ,s i ',r) to join the experience pool. The MADDPG algorithm adopts the CTDE method, which requires the observations of all agents to form a global information set O as the training input of the evaluation network. The specific elements of the set need to be adjusted according to the characteristics of the algorithm and environment. The simplest information set can be the observation combination of all agents, that is:

[0140] O=(S1,S2,...S N )

[0141] The necessary information can be added to the information set. O constitutes the overall observation of the environment by the Critic network. The richer the environment-related information it contains, the more conducive it is to the neural network's modeling and learning of the potential laws of the environment. The global information O and all agent actions constitute joint data, which are input into the Critic network for training. This is the core idea of ​​MADDPG. For neural networks that can only observe local information and their own actions, the environment of the multi-agent system is non-stationary. For neural networks that can observe the actions of all agents, all factors that cause environmental changes are known, and the environment itself can be considered stable. The MADDPG algorithm deals with the non-stationarity of the environment based on the above theory, so that the multi-agent reinforcement learning problem has Markov decision properties, and can use traditional reinforcement learning training methods such as experience replay. Let the current agent be i and the corresponding strategy be u. i , P represents the state transition probability, and its mathematical expression is:

[0142] P(S i '|S i ,...a i ,...a N )=P(S i '|S i ,...a i ,...a N ,u1,...u i ,...u N )

[0143] =P(S i '|S i ,...a i ,...a N ,u1',...u'2,...u' N )

[0144] When the amount of data in the experience pool of agent i is greater than the preset batch (such as the preset data volume), a batch of experience samples are extracted from it to participate in the training. i 'Input to the target policy network Act i '(S|u i2 ), select the next action a i '. Next S i ' and a i 'Used to estimate the current update target, such as:

[0145] y i =r i +γQ i '(s i ',a i ')|a i '=Acti '(s i '|u2),i∈[1,N]

[0146]

[0147] All agents perform local updates and send the updated parameters and rewards to the server. The server uses a competitive aggregation method to process all parameter updates. The main idea is to normalize the evaluation values, judge the quality of the parameters based on the normalized results, and make more updates in the direction of better results. N )Use Softmax function to normalize (sr1,sr2,...sr N ), and use it as the aggregation weight to update the core network, as follows:

[0148]

[0149] After a certain number of evaluation network updates, the policy gradient is used to update Act i (S|u i1 ) to update. Delayed update can effectively reduce the error accumulation during the update process and reduce invalid updates. The update formula is as follows:

[0150]

[0151] Q in the agent i '(S,A|θ i2 ) i '(S|u i2 ) also uses a delayed update method, compared to the policy network Act i (S|u i1 )Lower update frequency and update parameters using soft update method.

[0152] In this embodiment, a neural network model is iteratively trained for the intelligent agent corresponding to each local area network, which is conducive to the subsequent output of more accurate routing metric information by the trained neural network model, further improving the accuracy of determining the routing metric information.

[0153] In an exemplary embodiment, the above step S303 updates the first initial update parameters according to the second initial update parameters of the neural network model to be trained of the intelligent agent corresponding to other local area networks, the second reward information of the intelligent agent corresponding to other local area networks, and the first reward information, and specifically includes the following contents: normalizing the first reward information and the second reward information to obtain the processed first reward information and the processed second reward information; fusing the difference between the second initial update parameters and the first initial update parameters according to the processed second reward information to obtain the fused update parameters; updating the first initial update parameters according to the fused update parameters to obtain the first target update parameters of the neural network model to be trained of the intelligent agent corresponding to the local area network.

[0154] The processed first reward information refers to the normalized first reward information; the normalized first reward information = the first reward information / (the first reward information + the second reward information). The processed second reward information refers to the normalized second reward information; the normalized second reward information = the second reward information / (the first reward information + the second reward information).

[0155] Wherein, the fusion update parameter = (the second initial update parameter - the first initial update parameter) × the processed second reward information.

[0156] Among them, the first target update parameter of the neural network model to be trained of the intelligent agent corresponding to the local area network = the first initial update parameter + the fusion update parameter.

[0157] Exemplarily, the terminal sends the first initial update parameter, the second initial update parameter, the first reward information, and the second reward information to the server, such as Figure 5 The master control module shown. The server normalizes the first reward information and the second reward information to obtain the processed first reward information and the processed second reward information, then obtains the difference between the second initial update parameter and the first initial update parameter, and sums the difference between the second initial update parameter and the first initial update parameter according to the processed second reward information to obtain the fusion update parameter, then adds the first initial update parameter and the fusion update parameter to obtain the first target update parameter of the neural network model to be trained of the intelligent agent corresponding to the local area network, and finally sends the first target update parameter of the neural network model to be trained of the intelligent agent corresponding to the local area network to the terminal. The terminal updates the model parameters of the neural network model to be trained of the intelligent agent corresponding to the local area network to the first target update parameter to obtain the updated neural network model.

[0158] It should be noted that the process of determining the second target update parameters of the neural network model to be trained of the intelligent agent corresponding to other local area networks is consistent with the process of determining the first target update parameters of the neural network model to be trained of the intelligent agent corresponding to the local area network, and will not be repeated here.

[0159] For example, assuming that there are LAN A, LAN B, and LAN C, which correspond to Agent 1, Agent 2, and Agent 3 respectively, and to reward information 1, reward information 2, and reward information 3 respectively, and to initial update parameters 1, initial update parameters 2, and initial update parameters 3 respectively, then processed reward information 1 = reward information 1 / (reward information 1 + reward information 2 + reward information 3), processed reward information 2 = reward information 1 / (reward information 1 + reward information 2 + reward information 3), processed reward information 3 = reward information 1 / (reward information 1 + reward information 2 + reward information 3); then, the target update parameter of the neural network model to be trained corresponding to Agent 1 = initial update parameter 1 +[processed reward information 2×(initial update parameter 2-initial update parameter 1)+processed reward information 3×(initial update parameter 3-initial update parameter 1)]; The target update parameter of the neural network model to be trained corresponding to agent 2 = initial update parameter 2 + [processed reward information 1×(initial update parameter 1-initial update parameter 2) + processed reward information 3×(initial update parameter 3-initial update parameter 2)]; The target update parameter of the neural network model to be trained corresponding to agent 3 = initial update parameter 3 + [processed reward information 1×(initial update parameter 1-initial update parameter 3) + processed reward information 2×(initial update parameter 2-initial update parameter 3)].

[0160] In this embodiment, when determining the first target update parameters of the neural network model to be trained of the intelligent agent corresponding to the local area network, the second initial update parameters of the neural network model to be trained of the intelligent agent corresponding to other local area networks, the second reward information of the intelligent agent corresponding to other local area networks, the first initial update parameters of the neural network model to be trained of the intelligent agent corresponding to the local area network, and the first reward information of the intelligent agent corresponding to the local area network are comprehensively considered, which is beneficial to improving the accuracy of determining the first target update parameters of the neural network model to be trained of the intelligent agent corresponding to the local area network.

[0161] In an exemplary embodiment, the above-mentioned step S101, before determining the routing metric information inside the local area network based on the observation information of the local area network in the computing power network, also includes the following contents: receiving the current network topology structure information of the local area network perceived by the server, and receiving the current computing power resource information of the computing power network perceived by the computing power routing; obtaining the current network status information of the local area network perceived by the intelligent agent corresponding to the local area network, and combining the current computing power resource information, the current network topology structure information and the current network status information to obtain the observation information of the local area network.

[0162] The server side may refer to the master control side, such as Figure 5 The master control module shown; the server can perceive the current network topology information of each local area network.

[0163] Among them, the computing power router can perceive the current computing power resource information on the resource side of the computing power network, such as Figure 5 The cloud resources and edge resources shown.

[0164] Among them, the intelligent agent corresponding to the local area network can perceive the current network status information of the local area network; for example, refer to Figure 5 , the intelligent agent 1 corresponding to the local area network A can perceive the current network status information of the local area network A, and the intelligent agent 2 corresponding to the local area network B can perceive the current network status information of the local area network B.

[0165] Exemplarily, the server perceives the current network topology information of the local area network, and sends the perceived current network topology information of the local area network to the intelligent agent corresponding to the local area network; at the same time, the computing power routing perceives the current computing power resource information of the computing power network, and sends the perceived current computing power resource information of the computing power network to the intelligent agent corresponding to the local area network; in addition, the intelligent agent corresponding to the local area network can also perceive the current network status information of the local area network. Then, the terminal combines the current network status information of the local area network perceived by the intelligent agent corresponding to the local area network, the current network topology information of the local area network, and the current computing power resource information of the computing power network received by the intelligent agent corresponding to the local area network, to obtain the observation information of the local area network.

[0166] In this embodiment, by obtaining the current computing power resource information of the computing power network, the current network topology information of the local area network and the current network status information, and using them as the observation information of the local area network in the computing power network, it is beneficial to subsequently accurately determine the routing metric information within the local area network based on various observation information of the local area network in the computing power network, and regenerate the routing decision table, thereby achieving the purpose of dynamic routing, and is beneficial to further improve the data transmission efficiency of the computing power network.

[0167] In an exemplary embodiment, Figure 4As shown, another dynamic routing method is provided, which is described by taking the method applied to a terminal as an example, and includes the following steps S401 to S410. Among them:

[0168] Step S401, receiving the current network topology information of the local area network perceived by the server, and receiving the current computing power resource information of the computing power network perceived by the computing power router.

[0169] Step S402, obtain the current network status information of the local area network perceived by the intelligent agent corresponding to the local area network, and combine the current computing resource information, the current network topology structure information and the current network status information to obtain the observation information of the local area network.

[0170] Step S403, inputting the observation information into the trained neural network model of the agent corresponding to the local area network to obtain the link weight adjustment matrix inside the local area network.

[0171] Step S404, obtaining initial link weight data within the local area network; and determining adjustment values ​​of the initial link weight data within the local area network according to the link weight adjustment matrix.

[0172] Step S405, adjusting the initial link weight data according to the adjustment value of the initial link weight data to obtain the link weight data inside the local area network.

[0173] Step S406: Determine routing metric information within the local area network according to the link weight data.

[0174] Step S407: Determine the routing metric information of the transmission path to each destination address according to the routing metric information inside the local area network.

[0175] Step S408 , filtering out transmission paths whose routing metric information meets preset conditions from the transmission paths reaching each destination address respectively as target transmission paths reaching each destination address.

[0176] Step S409, regenerate the routing decision table of the local area network according to the target transmission path to each destination address.

[0177] Step S410: determining a target transmission path corresponding to the data to be transmitted in the local area network according to the regenerated routing decision table.

[0178] In the above-mentioned dynamic routing method, when data is transmitted in the computing power network, the current computing power resource information of the computing power network, the current network topology information of the local area network and the current network status information of the local area network are comprehensively considered, and the routing decision table of the local area network is regenerated based on these observation information, thereby achieving the purpose of dynamically changing the routing decision; then, based on the regenerated routing decision table, the target transmission path corresponding to the data to be transmitted in the local area network is determined, so that the data to be transmitted in the local area network can be transmitted according to the corresponding latest determined target transmission path, so that the data to be transmitted in the local area network can be transmitted according to the adapted target transmission path at all times, thereby achieving the effect of dynamic routing, which is conducive to speeding up the overall data transmission speed of the computing power network, thereby improving the data transmission efficiency of the computing power network, and at the same time avoiding the use of static routing, resulting in slower overall data transmission, and then resulting in the defect of low data transmission efficiency of the computing power network.

[0179] In order to more clearly illustrate the dynamic routing method provided by the embodiment of the present application, the following is a specific embodiment of the dynamic routing method. In an exemplary embodiment, Figure 5 As shown, the present application also provides a distributed intelligent routing system for computing power networks, which applies the multi-agent DDPG algorithm based on machine learning principles to the computing power routing system; considering the current scale of the computing power network, different local area networks are independently modeled as a distributed subsystem, and the objective factors such as the network status in the network are integrated to coordinate the distributed computing resources such as node cloud, edge, and end resource pools and dynamic network resources such as latency, bandwidth, and packet loss rate, and obtain the composition and function of each module in the distributed routing system architecture. Compared with the path selection method in the traditional computing power routing system, the present application proposes to use the multi-agent DDPG algorithm to not only solve the scale problem in the distributed local subnet, but also obtain the routing metric value in the actual router routing process under the condition of dynamic changes in the network topology and network status, thereby changing the routing decision of a large number of concurrent requests in the local area network, and does not need to determine a unique path decision for each request, and does not rely on SDN (Software Defined Network, Software Defined Network), SRv6 (Segment Routing IPv6, Segment Routing based on IPv6 forwarding plane) and other technologies, and the system and algorithm are more universal.

[0180] In view of the computing power resource requirements and computing power scheduling of large-scale users in the current computing power network application background, this application constructs a distributed intelligent routing system, optimizes the path selection method in the routing system, dynamically adjusts the routing metric value in the local subnet, generates a routing decision table, guides the optimal selection of the user request path, and ensures the service quality. Specifically, a distributed intelligent routing system is first constructed, and the computing power perception module perceives the computing power request on the user side and the computing power size on the resource side. The data forwarding template floods and updates the perceived computing power information so that the link state information contains the deployment location, real-time status, load information, and business requirements of the computing power service. The control surface collects a large number of user requests and network status changes, and conducts multi-agent dynamic routing optimization learning. The optimization goal is to reduce the low latency and low packet loss rate of concurrent requests in the local area network, and adjust the routing metric value in the computing power network according to the optimization results, that is, adjust the routing metric value based on the multi-agent reinforcement learning method, so as to optimize the path selection of user requests and achieve the best end-to-end service quality.

[0181] refer to Figure 5 , the distributed intelligent routing system includes:

[0182] User side: including scenarios such as Internet of Vehicles, large models, and image rendering. These applications require a lot of computing resources and may have low latency requirements.

[0183] General control module: 1. Deploy a set of core evaluation neural networks, collect the parameters and reward values ​​of all intelligent agents in the distributed modules for training to obtain evaluations, and then normalize the evaluation values. According to the normalization results, the parameters are evaluated and more updates are made in the direction of better effects; 2. Sense the changes in the network topology structure within the large-scale local area network and send them to the distributed modules.

[0184] Distributed module: 1. Each agent i deploys a training neural network, including a value function evaluation network, a target evaluation network, an action strategy network, and a target action strategy network, and trains the optimal routing decision strategy through neural network learning; 2. The agent i of the designated local area network perceives the network topology and network status, and the computing power routing perceives the computing power resources on the resource side, and obtains the observation results as the input of the neural network; 3. Agent i outputs an action, that is, adjusts the routing metric value within the local area network, and the local area network regenerates the routing decision table according to the routing metric value.

[0185] Resource side: Computing resources are divided into cloud resources and edge resources to meet the computing power requirements of different applications. Cloud resources have powerful computing capabilities and are suitable for complex tasks; edge resources are close to users and are suitable for low-latency tasks.

[0186] The above embodiments can achieve the following technical effects: (1) In a distributed intelligent routing system, the master control module integrates all computing resources and network status, and through intelligent routing learning, dynamically issues the optimal routing decision, thereby ensuring the optimal service quality for users; (2) Through a multi-agent reinforcement learning method, it can solve the high-concurrency requests of large-scale users, ensure the quality of multi-regional network collaborative communication, and optimize the overall average delay and packet loss rate.

[0187] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0188] Based on the same inventive concept, the embodiment of the present application also provides a dynamic routing device for implementing the dynamic routing method involved above. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above method, so the specific limitations in one or more dynamic routing device embodiments provided below can refer to the limitations on the dynamic routing method above, and will not be repeated here.

[0189] In an exemplary embodiment, Figure 6 As shown, a dynamic routing device is provided, including: an information determination module 610, a regeneration module 620 and a path determination module 630, wherein:

[0190] The information determination module 610 is used to determine the routing metric information inside the local area network based on the observation information of the local area network in the computing power network; the observation information at least includes the current computing power resource information of the computing power network, the current network topology structure information of the local area network and the current network status information.

[0191] The regeneration module 620 is used to regenerate the routing decision table of the local area network according to the routing metric information.

[0192] The path determination module 630 is used to determine the target transmission path corresponding to the data to be transmitted in the local area network according to the regenerated routing decision table.

[0193] In an exemplary embodiment, the regeneration module 620 is also used to determine the routing metric information of the transmission path to each destination address based on the routing metric information within the local area network; filter out the transmission path whose routing metric information meets the preset conditions from the transmission path to each destination address, as the target transmission path to each destination address; and regenerate the routing decision table of the local area network based on the target transmission path to each destination address.

[0194] In an exemplary embodiment, the information determination module 610 is also used to input the observation information into the trained neural network model of the intelligent agent corresponding to the local area network to obtain the link weight adjustment matrix within the local area network; determine the link weight data within the local area network based on the link weight adjustment matrix; and determine the routing metric information within the local area network based on the link weight data.

[0195] In an exemplary embodiment, the information determination module 610 is also used to obtain the initial link weight data within the local area network; determine the adjustment value of the initial link weight data within the local area network according to the link weight adjustment matrix; adjust the initial link weight data according to the adjustment value of the initial link weight data to obtain the link weight data within the local area network.

[0196] In an exemplary embodiment, the dynamic routing device also includes a model training module, which is used to obtain first sample observation information of an intelligent agent corresponding to a local area network in the computing power network and first sample action information corresponding to the first sample observation information, as well as second sample observation information of intelligent agents corresponding to other local area networks in the computing power network and second sample action information corresponding to the second sample observation information; according to the first sample observation information, the first sample action information, the second sample observation information and the second sample action information, the neural network model to be trained of the intelligent agent corresponding to the local area network is trained to obtain the first initial update parameter of the neural network model to be trained of the intelligent agent corresponding to the local area network, and the first reward information of the intelligent agent corresponding to the local area network; according to the second initial update parameter of the neural network model to be trained of the intelligent agent corresponding to other local area networks, the second reward information of the intelligent agent corresponding to other local area networks and the first reward information, the first initial update parameter is updated until the preset number of training times is reached to obtain the trained neural network model of the intelligent agent corresponding to the local area network.

[0197] In an exemplary embodiment, the model training module is also used to normalize the first reward information and the second reward information to obtain processed first reward information and processed second reward information; according to the processed second reward information, the difference between the second initial update parameter and the first initial update parameter is fused to obtain fused update parameters; according to the fused update parameters, the first initial update parameters are updated to obtain the first target update parameters of the neural network model to be trained of the intelligent agent corresponding to the local area network.

[0198] In an exemplary embodiment, the dynamic routing device also includes an information acquisition module for receiving the current network topology information of the local area network perceived by the server, and receiving the current computing power resource information of the computing power network perceived by the computing power routing; obtaining the current network status information of the local area network perceived by the intelligent agent corresponding to the local area network, and combining the current computing power resource information, the current network topology information and the current network status information to obtain the observation information of the local area network.

[0199] Each module in the above dynamic routing can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0200] In an exemplary embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as shown in FIG. Figure 7As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and the external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, near field communication (Near Field Communication, NFC) or other technologies. When the computer program is executed by the processor, a dynamic routing method is implemented. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0201] Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0202] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0203] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0204] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0205] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0206] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0207] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0208] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A dynamic routing method, characterized in that: The method comprises: Determine routing metric information within the local area network according to observation information of the local area network in the computing power network; the observation information at least includes current computing power resource information of the computing power network, current network topology information of the local area network, and current network status information; Regenerating a routing decision table of the local area network according to the routing metric information; According to the regenerated routing decision table, a target transmission path corresponding to the data to be transmitted in the local area network is determined.

2. The method according to claim 1, characterized in that The regenerating the routing decision table of the local area network according to the routing metric information comprises: Determining routing metric information of a transmission path to each destination address based on routing metric information within the local area network; Filtering out, from the transmission paths reaching the respective destination addresses, the transmission paths whose routing metric information meets a preset condition as target transmission paths reaching the respective destination addresses; The routing decision table of the local area network is regenerated according to the target transmission path to reach each destination address.

3. The method according to claim 1, characterized in that Determining routing metric information within the local area network according to observation information of the local area network in the computing power network includes: Inputting the observation information into the trained neural network model of the agent corresponding to the local area network to obtain a link weight adjustment matrix within the local area network; Determining link weight data within the local area network according to the link weight adjustment matrix; Routing metric information within the local area network is determined based on the link weight data.

4. The method according to claim 3, characterized in that Determining the link weight data within the local area network according to the link weight adjustment matrix includes: Obtaining initial link weight data within the local area network; Determining, according to the link weight adjustment matrix, an adjustment value of the initial link weight data within the local area network; The initial link weight data is adjusted according to the adjustment value of the initial link weight data to obtain the link weight data within the local area network.

5. The method according to claim 3, characterized in that: The trained neural network model of the intelligent agent corresponding to the local area network is obtained by training in the following way: Obtaining first sample observation information of an intelligent agent corresponding to a local area network in the computing power network and first sample action information corresponding to the first sample observation information, as well as second sample observation information of intelligent agents corresponding to other local area networks in the computing power network and second sample action information corresponding to the second sample observation information; According to the first sample observation information, the first sample action information, the second sample observation information and the second sample action information, the neural network model to be trained of the intelligent agent corresponding to the local area network is trained to obtain a first initial update parameter of the neural network model to be trained of the intelligent agent corresponding to the local area network, and first reward information of the intelligent agent corresponding to the local area network; According to the second initial update parameter of the neural network model to be trained of the intelligent agent corresponding to the other local area network, the second reward information of the intelligent agent corresponding to the other local area network and the first reward information, the first initial update parameter is updated until the preset number of training times is reached to obtain the trained neural network model of the intelligent agent corresponding to the local area network.

6. The method according to claim 5, characterized in that The updating of the first initial update parameter according to the second initial update parameter of the neural network model to be trained of the agent corresponding to the other local area network, the second reward information of the agent corresponding to the other local area network, and the first reward information comprises: Normalizing the first reward information and the second reward information to obtain processed first reward information and processed second reward information; According to the processed second reward information, a fusion process is performed on the difference between the second initial update parameter and the first initial update parameter to obtain a fusion update parameter; The first initial update parameter is updated according to the fusion update parameter to obtain the first target update parameter of the neural network model to be trained of the intelligent agent corresponding to the local area network.

7. The method according to any one of claims 1 to 6, characterized in that: Before determining the routing metric information inside the local area network according to the observation information of the local area network in the computing power network, the method further includes: Receive the current network topology information of the local area network perceived by the server, and receive the current computing power resource information of the computing power network perceived by the computing power router; The current network status information of the local area network perceived by the intelligent agent corresponding to the local area network is obtained, and the current computing resource information, the current network topology structure information and the current network status information are combined to obtain the observation information of the local area network.

8. A dynamic routing device, characterized in that: The device comprises: An information determination module, configured to determine routing metric information within a local area network according to observation information of the local area network in the computing power network; the observation information at least includes current computing power resource information of the computing power network, current network topology information of the local area network, and current network status information; A regeneration module, used to regenerate the routing decision table of the local area network according to the routing metric information; The path determination module is used to determine the target transmission path corresponding to the data to be transmitted in the local area network according to the regenerated routing decision table.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images