Dynamic routing method, device, computer equipment, storage medium and program product

Abstract

The application relates to a dynamic routing method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: determining routing metric information in a local area network in a computing power network according to observation information of the local area network; the observation information at least comprises current computing power resource information of the computing power network, current network topology information and current network state information of the local area network; regenerating a routing decision table of the local area network according to the routing metric information; and determining a target transmission path corresponding to to-be-transmitted data in the local area network according to the regenerated routing decision table. The method can improve the data transmission efficiency of the computing power network.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to a dynamic routing method and device, computer equipment, computer readable storage medium and computer program product. BACKGROUND

[0002] The computing power network is connected based on network connection, and the dynamically distributed computing resources are interconnected. Through flexible scheduling of multi-dimensional resources such as computing, network, and storage, global optimization of connection and computing power in the network is realized, which brings new opportunities for the development of computing industry of users and operators.

[0003] In the prior art, when data transmission is performed in the computing power network, a static routing mode is generally used. However, with the development and progress of network technology, a large number of network devices are connected to the network, including a large number of mobile network access devices, which directly lead to frequent changes in network topology structure. If the static routing mode is still used, the overall data transmission will be slow, and thus the data transmission efficiency of the computing power network is low. SUMMARY

[0004] Therefore, it is necessary to provide a dynamic routing method, device, computer equipment, computer readable storage medium and computer program product capable of improving the data transmission efficiency of the computing power network.

[0005] In a first aspect, the present application provides a dynamic routing method, comprising:

[0006] According to the observation information of the local area network in the computing power network, the routing metric information inside the local area network is determined. The observation information at least includes the current computing power resource information of the computing power network, the current network topology structure information and the current network state information of the local area network;

[0007] According to the routing metric information, the routing decision table of the local area network is regenerated;

[0008] According to the regenerated routing decision table, the target transmission path corresponding to the to-be-transmitted data in the local area network is determined.

[0009] In one embodiment, the routing decision table of the local area network is regenerated according to the routing metric information, comprising:

[0010] According to the routing metric information inside the local area network, the routing metric information of the transmission path to each destination address is determined;

[0011] From the transmission paths to each destination address, the transmission path whose routing metric information meets the preset condition is filtered out as the target transmission path to each destination address.

[0012] re-generate a routing decision table of the local area network according to a target transmission path to each destination address.

[0013] In one of the embodiments, the determining the routing metric information inside the local area network according to the observation information of the local area network in the computing power network comprises:

[0014] inputting the observation information into a trained neural network model of an agent corresponding to the local area network to obtain a link weight adjustment matrix inside the local area network;

[0015] determining link weight data inside the local area network according to the link weight adjustment matrix;

[0016] determining the routing metric information inside the local area network according to the link weight data.

[0017] In one of the embodiments, the determining the link weight data inside the local area network according to the link weight adjustment matrix comprises:

[0018] obtaining initial link weight data inside the local area network;

[0019] determining an adjustment value of the initial link weight data inside the local area network according to the link weight adjustment matrix;

[0020] adjusting the initial link weight data according to the adjustment value of the initial link weight data to obtain the link weight data inside the local area network.

[0021] In one of the embodiments, the trained neural network model of the agent corresponding to the local area network is obtained by training in the following manner:

[0022] obtaining first sample observation information of an agent corresponding to the local area network in the computing power network and first sample action information corresponding to the first sample observation information, and second sample observation information of an agent corresponding to another local area network in the computing power network and second sample action information corresponding to the second sample observation information;

[0023] training a neural network model to be trained of the agent corresponding to the local area network according to the first sample observation information, the first sample action information, the second sample observation information and the second sample action information to obtain a first initial update parameter of the neural network model to be trained of the agent corresponding to the local area network, and first reward information of the agent corresponding to the local area network;

[0024] update the first initial update parameter according to second initial update parameters of a neural network model to be trained of an agent corresponding to another local area network, second reward information of the agent corresponding to the another local area network, and the first reward information, until a preset training number is reached, to obtain a trained neural network model of the agent corresponding to the local area network.

[0025] In one of the embodiments, the updating of the first initial update parameter according to the second initial update parameters of the neural network model to be trained of the agent corresponding to the another local area network, the second reward information of the agent corresponding to the another local area network, and the first reward information comprises:

[0026] normalizing the first reward information and the second reward information to obtain processed first reward information and processed second reward information;

[0027] performing fusion processing on a difference between the second initial update parameter and the first initial update parameter according to the processed second reward information to obtain a fusion update parameter;

[0028] updating the first initial update parameter according to the fusion update parameter to obtain a first target update parameter of the neural network model to be trained of the agent corresponding to the local area network.

[0029] In one of the embodiments, before determining the routing metric information inside the local area network according to observation information of the local area network in the computing power network, the method further comprises:

[0030] receiving current network topology information of the local area network perceived by a server and receiving current computing power resource information of the computing power network perceived by a computing power routing;

[0031] obtaining current network state information of the local area network perceived by an agent corresponding to the local area network, and combining the current computing power resource information, the current network topology information, and the current network state information to obtain the observation information of the local area network

[0032] In a second aspect, the application further provides a dynamic routing device, comprising:

[0033] an information determining module configured to determine routing metric information inside a local area network in a computing power network according to observation information of the local area network; the observation information at least includes current computing power resource information of the computing power network, current network topology information of the local area network, and current network state information of the local area network;

[0034] a regenerating module configured to regenerate a routing decision table of the local area network according to the routing metric information;

[0035] determine, according to the regenerated routing decision table, a target transmission path corresponding to to-be-transmitted data in the local area network.

[0036] In a third aspect, the present application further provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0037] determine routing metric information inside the local area network according to observation information of the local area network in the computing power network; the observation information at least comprises current computing power resource information of the computing power network, current network topology structure information and current network state information of the local area network;

[0038] regenerate a routing decision table of the local area network according to the routing metric information;

[0039] determine, according to the regenerated routing decision table, a target transmission path corresponding to to-be-transmitted data in the local area network.

[0040] In a fourth aspect, the present application further provides a computer readable storage medium, which stores a computer program, and the computer program implements the following steps when executed by a processor:

[0041] determine routing metric information inside the local area network according to observation information of the local area network in the computing power network; the observation information at least comprises current computing power resource information of the computing power network, current network topology structure information and current network state information of the local area network;

[0042] regenerate a routing decision table of the local area network according to the routing metric information;

[0043] determine, according to the regenerated routing decision table, a target transmission path corresponding to to-be-transmitted data in the local area network.

[0044] In a fifth aspect, the present application further provides a computer program product, comprising a computer program, and the computer program implements the following steps when executed by a processor:

[0045] determine routing metric information inside the local area network according to observation information of the local area network in the computing power network; the observation information at least comprises current computing power resource information of the computing power network, current network topology structure information and current network state information of the local area network;

[0046] regenerate a routing decision table of the local area network according to the routing metric information;

[0047] determine, according to the regenerated routing decision table, a target transmission path corresponding to to-be-transmitted data in the local area network.

[0048] The dynamic routing method, device, computer equipment, computer readable storage medium and computer program product determine routing metric information inside the local area network according to observation information of the local area network in the computing power network; the observation information at least includes current computing power resource information of the computing power network, current network topology structure information and current network state information of the local area network; then, the routing decision table of the local area network is regenerated according to the routing metric information, and finally, the target transmission path corresponding to the to-be-transmitted data in the local area network is determined according to the regenerated routing decision table. In this way, when data transmission is performed in the computing power network, the current computing power resource information of the computing power network, the current network topology structure information of the local area network and the current network state information of the local area network are comprehensively considered, and the routing decision table of the local area network is regenerated based on the observation information, so that the purpose of dynamically changing the routing decision is achieved; then, the target transmission path corresponding to the to-be-transmitted data in the local area network is determined based on the regenerated routing decision table, so that the to-be-transmitted data in the local area network can be transmitted according to the corresponding target transmission path determined at the latest, so that the to-be-transmitted data in the local area network can be transmitted according to the adaptive target transmission path at each moment, and the effect of dynamic routing is achieved, which is beneficial to accelerating the overall data transmission speed of the computing power network, thereby improving the data transmission efficiency of the computing power network, and at the same time, the defect that the overall data transmission is slow and the data transmission efficiency of the computing power network is low caused by the static routing mode is avoided. BRIEF DESCRIPTION OF DRAWINGS

[0049] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related art, the drawings needed to be used in the description of the embodiments of the present application or the related art will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other related drawings can be obtained by those skilled in the art without creative labor.

[0050] Figure 1 A flowchart of a dynamic routing method in an embodiment;

[0051] Figure 2 A flowchart of a step of determining routing metric information inside a local area network in an embodiment;

[0052] Figure 3 A flowchart of a training step of a neural network model of an agent corresponding to a local area network in an embodiment;

[0053] Figure 4 A flowchart of a dynamic routing method in another embodiment;

[0054] Figure 5A structural block diagram of a distributed intelligent routing system for a computing power network in an embodiment;

[0055] Figure 6 A structural block diagram of a dynamic routing device in an embodiment;

[0056] Figure 7 An internal structural diagram of a computer device in an embodiment. DETAILED DESCRIPTION

[0057] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0058] The computing power demand of users is becoming more and more rich and large-scale as the computing power network develops. At the same time, a large number of heterogeneous computing power resources also participate in the construction of modern new computing power network environment. In this process, operators need to provide good network service quality to users. Although the standard of network service quality needs to be determined according to specific scenarios, routing is always an important part of influencing network service quality.

[0059] The main content of routing is to select a suitable path for the data packets passing through the network. Static routing allocation is the most original routing method, as long as the network topology structure is stable, the static routing table is stable. However, with the development and progress of network technology, a large number of network devices are connected to the network, including a large number of mobile network access devices, which directly lead to the rapid growth of network scale and the frequent change of network topology, causing great difficulties to the application of traditional static routing method, and dynamic routing optimization method emerges as the times require.

[0060] The dynamic routing optimization method not only needs to consider the dynamic change of network structure, but also needs to consider the network state (such as jitter, bandwidth, packet loss rate, etc.) and computing power resource (such as CPU, GPU, memory) change, wherein the network structure can be perceived by the master control module, the network state change in the local area network can be perceived by the distributed module, and the computing power resource change can be perceived by the computing power routing. The dynamic routing optimization method is to dynamically change the routing decision according to the above changes to provide good network service quality. Based on this, the present application provides a dynamic routing method, which generates a routing decision table dynamically to improve the data transmission efficiency of the computing power network.

[0061] In order to make the dynamic routing method provided by the embodiments of the present application clearer, the following is an explanation of some terms:

[0062] Multi-Agent DDPG: Also known as MADDPG (Muli-Agent Deep Deterministic Policy Gradient), is a reinforcement learning algorithm designed to enable multiple agents to optimize their strategies while sharing global information through a centralized training and decentralized execution architecture. During the training process, agents have access to the states and actions of other agents to evaluate their behavior through a centralized Critic network (part of the Q-Learning algorithm based on value functions); while in execution, each agent only relies on its own local observations to make decisions. MADDPG effectively solves the problem of multi-agent cooperation or competition by preserving the independence between individual agents and ignoring unnecessary complex interactions, making it suitable for complex tasks in high-dimensional continuous action spaces.

[0063] CTDE (Centralized Training with Decentralized Execution): A multi-agent reinforcement learning architecture that allows agents to share global information during the training phase to improve the efficiency of policy learning; while in the execution phase, each agent only relies on its own local observations to make independent decisions, thus adapting to the needs of distributed and dynamic environments. CTDE enhances the flexibility and robustness of the system while ensuring the collaboration ability of the agents, making it an effective method for solving cooperation and competition problems in multi-agent systems.

[0064] In an exemplary embodiment, as shown in Figure 1 , a dynamic routing method is provided, and this embodiment is exemplified by applying the method to a terminal; it can be understood that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. The terminal can be, but is not limited to, various personal computers, notebook computers, smartphones, tablet computers, etc.; the server can be a standalone physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. In this embodiment, the following steps S101 to S103 are included. Among them:

[0065] Step S101, according to the observation information of the local area network in the computing power network, determine the routing metric information inside the local area network; the observation information at least includes the current computing power resource information of the computing power network, the current network topology structure information and the current network state information of the local area network.

[0066] Among them, the computing power network includes multiple local area networks; for example, referring to Figure 5 , the computing power network includes local area network A, local area network B, …, and local area network X.

[0067] The observation information of the local area network includes various information, which are the current computing power resource information of the computing power network, the current network topology structure information and the current network state information of the local area network, and the like.

[0068] The current computing power resource information of the computing power network refers to the computing power resource information of the computing power network at the current moment, and specifically includes CPU (Central Processing Unit), GPU (Graphics Processing Unit), memory, and the like.

[0069] The current network state information of the local area network refers to the network state information of the local area network at the current moment, and specifically includes jitter, bandwidth, packet loss rate, time delay, and the like.

[0070] The current network topology structure information of the local area network refers to the network topology structure information of the local area network at the current moment.

[0071] The routing metric information refers to the routing metric value of the transmission path inside the local area network, each transmission path corresponds to a routing metric value, and is obtained by re-determining according to the observation information of the local area network.

[0072] The routing metric value represents the cost of the transmission path inside the local area network, and specifically refers to the cost of reaching the destination address indicated by the transmission path, which is an important basis in the routing selection process. In actual scenarios, the smaller the routing metric value of the transmission path, the more it is preferred; and for multiple transmission paths reaching the same destination address, the transmission path with the smallest corresponding routing metric value will be added to the corresponding routing decision table of the local area network.

[0073] The routing metric value can be represented by cost, hop count, bandwidth, time delay, load, reliability, maximum transmission unit, and the like.

[0074] For example, a terminal (such as a control device in the local area network) obtains the observation information of the local area network in the computing power network, such as the current computing power resource information of the computing power network, the current network topology structure information and the current network state information of the local area network, and the like; then inputs the observation information into the trained neural network model deployed by the agent of the local area network, processes the observation information through the neural network model, obtains the link weight adjustment matrix inside the local area network, and finally obtains the routing metric information inside the local area network according to the link weight adjustment matrix inside the local area network, thereby achieving the purpose of adjusting the routing metric information.

[0075] In step S102, the routing decision table of the local area network is re-generated according to the routing metric information.

[0076] Each local area network corresponds to a routing decision table. The routing decision table of the local area network includes target transmission paths (i.e., optimal transmission paths) to different destination addresses, such as an optimal transmission path L1 to a destination address M1, an optimal transmission path L2 to a destination address M2, an optimal transmission path L3 to a destination address M3, and the like.

[0077] It should be noted that when the observation information of the local area network changes, the terminal will regenerate the routing decision table of the local area network, thereby achieving the purpose of dynamic routing. Moreover, based on the observation information of each local area network, the routing metric information within each local area network can be determined, and then the routing decision table of each local area network is regenerated according to the routing metric information within each local area network.

[0078] For example, the terminal determines the target transmission paths to different destination addresses according to the routing metric information within the local area network, and regenerates the routing decision table of the local area network according to the target transmission paths to different destination addresses.

[0079] In step S103, the target transmission path corresponding to the to-be-transmitted data in the local area network is determined according to the regenerated routing decision table.

[0080] The to-be-transmitted data carries a destination address, and according to the destination address, the corresponding target transmission path can be found in the routing decision table.

[0081] The target transmission path corresponding to the to-be-transmitted data refers to the target transmission path to the destination address carried in the to-be-transmitted data.

[0082] For example, the terminal queries the regenerated routing decision table according to the destination address carried in the to-be-transmitted data in the local area network, obtains the target transmission path to the destination address, and takes it as the target transmission path corresponding to the to-be-transmitted data. Further, the terminal can also transmit the to-be-transmitted data according to the target transmission path corresponding to the to-be-transmitted data, so that the to-be-transmitted data is transmitted to the destination address through the target transmission path.

[0083] For example, assuming that the to-be-transmitted data in the local area network carries a destination address M1, and the regenerated routing decision table has a target transmission path L1 to the destination address M1, then the target transmission path corresponding to the to-be-transmitted data is L1.

[0084] In the aforementioned dynamic routing method, routing metric information within the local area network (LAN) is determined based on observation information from the LAN within the computing power network. The observation information includes at least the current computing power resource information of the computing power network, the current network topology information of the LAN, and the current network status information. Then, based on the routing metric information, the routing decision table of the LAN is regenerated. Finally, based on the regenerated routing decision table, the target transmission path corresponding to the data to be transmitted in the LAN is determined. In this way, when data transmission occurs in the computing power network, the current computing power resource information, the current network topology information, and the current network status information of the local area network are comprehensively considered. Based on these observations, the routing decision table of the local area network is regenerated, achieving the goal of dynamically changing the routing decision. Then, based on the regenerated routing decision table, the target transmission path corresponding to the data to be transmitted in the local area network is determined, so that the data to be transmitted in the local area network can be transmitted according to the newly determined target transmission path. This ensures that the data to be transmitted in the local area network can be transmitted according to the appropriate target transmission path at all times, thus achieving the effect of dynamic routing. This helps to speed up the overall data transmission speed of the computing power network, thereby improving the data transmission efficiency of the computing power network. At the same time, it avoids the shortcomings of using static routing, which leads to slow overall data transmission and thus low data transmission efficiency of the computing power network.

[0085] In an exemplary embodiment, step S102 above, which regenerates the routing decision table of the local area network based on the routing metric information, specifically includes the following: determining the routing metric information of the transmission path to each destination address based on the routing metric information within the local area network; selecting transmission paths whose routing metric information meets preset conditions from the transmission paths to each destination address, and using them as target transmission paths to each destination address; and regenerating the routing decision table of the local area network based on the target transmission paths to each destination address.

[0086] The routing metric information within the local area network includes the routing metric information of the transmission paths to each destination address.

[0087] There are multiple transmission paths to the same destination address. For example, the transmission paths to destination address M1 include L1, L4, L5, L6, etc.

[0088] Among them, the routing metric information meets the preset condition when the value corresponding to the routing metric information (i.e., the routing metric value) is the smallest.

[0089] For each local area network, the regenerated routing decision table includes the target transmission path to each destination address.

[0090] Exemplarily, the terminal determines the routing metric values of the transmission paths to each destination address according to the routing metric values within the local area network, then respectively filters the transmission path with the minimum routing metric value from the transmission paths to each destination address as the target transmission path to each destination address, and finally replaces the historical target transmission paths to each destination address in the historical routing decision table of the local area network with the target transmission paths to each destination address to obtain a new routing decision table of the local area network.

[0091] For example, the transmission path to the destination address M1 includes L1, L4, L5 and L6, and the routing metric value of the transmission path L1 is the minimum, which indicates that the optimal transmission path to the destination address M1 is L1; by analogy, the optimal transmission path L2 to the destination address M2 and the optimal transmission path L3 to the destination address M3 can be obtained; based on the optimal transmission path L1 to the destination address M1, the optimal transmission path L2 to the destination address M2 and the optimal transmission path L3 to the destination address M3, the routing decision table of the local area network can be regenerated.

[0092] In this embodiment, the routing decision table of the local area network is regenerated according to the routing metric information within the local area network, which achieves the purpose of dynamically changing the routing decision and is beneficial to improving the determination accuracy of the routing decision table, so that the to-be-transmitted data in the local area network can be transmitted according to the adaptive target transmission path at each moment, thereby accelerating the overall data transmission speed of the computing power network.

[0093] In one exemplary embodiment, as shown in Figure 2 The step S101 of determining the routing metric information within the local area network according to the observation information of the local area network in the computing power network includes the following steps S201 to S203. Wherein:

[0094] The step S201 inputs the observation information into the trained neural network model of the agent corresponding to the local area network to obtain a link weight adjustment matrix within the local area network.

[0095] The step S202 determines the link weight data within the local area network according to the link weight adjustment matrix.

[0096] The step S203 determines the routing metric information within the local area network according to the link weight data.

[0097] Wherein, each local area network corresponds to an agent, and each agent is deployed with a trained neural network model, such as a reinforcement learning model; the trained neural network model includes a value function evaluation network, a target evaluation network, an action policy network and a target action policy network. For example, refer to Figure 5, the local area network A corresponds to the agent 1, the local area network B corresponds to the agent 2, and the local area network X corresponds to the agent N, and the agent 1, the agent 2, and the agent N are all deployed with a trained neural network model.

[0098] The trained neural network model refers to a network model that outputs a link weight adjustment matrix in the local area network based on the input observation information of the local area network.

[0099] The local area network includes a plurality of links, and each link corresponds to an initial link weight data, such as 0.5, 0.6, etc., which is used to represent the cost or overhead of each link.

[0100] The link weight adjustment matrix includes the adjustment value of the initial link weight data of each link in the local area network, such as 0.1, 0.2, etc.

[0101] The link weight data in the local area network includes the link weight data of each link in the local area network; the link weight data of each link in the local area network is used to represent the adjusted cost or adjusted overhead of each link.

[0102] In the local area network, each transmission path includes a plurality of links, such as the transmission path L1 to the destination address M1 a→b→c→d, which includes 3 links, a→b, b→c and c→d.

[0103] In the local area network, the routing metric information of the transmission path to the destination address is determined based on the link weight data corresponding to the links included in the transmission path, such as the routing metric information of the transmission path to the destination address being equal to the sum of the link weight data corresponding to the links included in the transmission path.

[0104] Exemplarily, the terminal inputs the observation information into the trained neural network model of the intelligent agent corresponding to the local area network, performs prediction processing on the observation information through the trained neural network model, obtains the corresponding output action, that is, the link weight adjustment matrix inside the local area network, then determines the adjustment value of the initial link weight data of the link inside the local area network according to the link weight adjustment matrix inside the local area network, and adjusts the initial link weight data of the link inside the local area network according to the adjustment value of the initial link weight data of the link inside the local area network, to obtain the link weight data corresponding to the link inside the local area network. Finally, the link weight data corresponding to the link included in the transmission path to each destination address is filtered from the link weight data corresponding to the link inside the local area network, and the link weight data corresponding to the link included in the transmission path to each destination address is added respectively to obtain the weight data of the transmission path to each destination address as the routing metric information of the transmission path to each destination address, and the routing metric information of the local area network is obtained based on the routing metric information of the transmission path to each destination address.

[0105] For example, the transmission path L1 to the destination address M1 is a→b→c→d, which includes a total of 3 links, that is, a→b, b→c and c→d, and the initial link weight data corresponding to the 3 links is 0.5, 0.6 and 0.7 respectively, and the adjustment value is +0.1, -0.3 and +0.1 respectively, so the link weight data corresponding to the 3 links is 0.6, 0.3 and 0.8 respectively, and the routing metric value of the transmission path L1 to the destination address M1 is 0.6+0.3+0.8=1.7.

[0106] In this embodiment, when determining the routing metric information inside the local area network, the current computing power resource information of the computing power network, the current network topology structure information and the current network state information of the local area network are comprehensively considered, and the trained neural network model is combined, which is beneficial to improve the determination accuracy of the routing metric information inside the local area network.

[0107] In an exemplary embodiment, the step S202 of determining the link weight data inside the local area network according to the link weight adjustment matrix specifically includes the following contents: obtaining the initial link weight data inside the local area network; determining the adjustment value of the initial link weight data inside the local area network according to the link weight adjustment matrix; and adjusting the initial link weight data according to the adjustment value of the initial link weight data to obtain the link weight data inside the local area network.

[0108] The initial link weight data inside the local area network includes the initial link weight data of each link in the local area network

[0109] Exemplarily, the terminal obtains initial link weight data in the local area network, determines initial link weight data of each link in the local area network based on the initial link weight data in the local area network, then queries the link weight adjustment matrix to obtain an adjustment value of the initial link weight data of each link in the local area network, and finally adds the initial link weight data of each link in the local area network and the adjustment value of the initial link weight data to obtain link weight data of each link in the local area network, and obtains the link weight data in the local area network based on the link weight data of each link in the local area network.

[0110] For example, the initial link weight data of the link a→b is 0.5, and the adjustment value is 0.2, so the initial link weight data of the link a→b is 0.5+0.2=0.7.

[0111] In this embodiment, the adjustment value of the initial link weight data in the local area network is determined according to the link weight adjustment matrix, and the initial link weight data is adjusted according to the adjustment value of the initial link weight data to obtain the link weight data in the local area network, thereby achieving the purpose of dynamically updating the link weight data in the local area network, and facilitating the determination of more accurate routing metric information based on the link weight data in the local area network.

[0112] In one exemplary embodiment, as shown in Figure 3 The dynamic routing method provided by the application further includes a training step of a trained neural network model of an agent corresponding to a local area network of the computing power network, specifically including the following steps S301 to S303. Wherein:

[0113] Step S301, obtaining first sample observation information and first sample action information corresponding to the first sample observation information of an agent corresponding to a local area network in the computing power network, and second sample observation information and second sample action information corresponding to the second sample observation information of an agent corresponding to another local area network in the computing power network.

[0114] Step S302, training the neural network model to be trained of the agent corresponding to the local area network according to the first sample observation information, the first sample action information, the second sample observation information and the second sample action information, obtaining the first initial update parameter of the neural network model to be trained of the agent corresponding to the local area network, and the first reward information of the agent corresponding to the local area network.

[0115] Step S303, updating the first initial update parameter according to the second initial update parameter of the neural network model to be trained of the agent corresponding to the other local area network, the second reward information of the agent corresponding to the other local area network and the first reward information, until a preset training number is reached, to obtain the trained neural network model of the agent corresponding to the local area network.

[0116] The neural network model to be trained includes a value function evaluation network, a target evaluation network, an action policy network, and a target action policy network.

[0117] The first sample observation information of the agent corresponding to the local area network includes sample computing power resource information of the computing power network, sample network topology structure information of the local area network, and sample network state information.

[0118] The first sample action information refers to the output action of the agent corresponding to the local area network, such as a sample link weight adjustment matrix inside the local area network.

[0119] The other local area network in the computing power network refers to a local area network other than the local area network in the computing power network. Figure 5 For example, if the local area network in the computing power network is local area network A, the other local area networks are local area networks B, …, and X.

[0120] The second sample observation information of the agent corresponding to the other local area network includes sample computing power resource information of the computing power network, sample network topology structure information of the other local area network, and sample network state information.

[0121] The second sample action information refers to the output action of the agent corresponding to the other local area network, such as a sample link weight adjustment matrix inside the other local area network.

[0122] The first initial update parameter of the neural network model to be trained of the agent corresponding to the local area network refers to the initial model update parameter of the neural network model to be trained of the agent corresponding to the local area network.

[0123] The first reward information refers to the reward information of the agent corresponding to the local area network, and is used to represent the network average communication delay and packet loss rate.

[0124] The second initial update parameter of the neural network model to be trained of the agent corresponding to the other local area network refers to the initial model update parameter of the neural network model to be trained of the agent corresponding to the other local area network.

[0125] The second reward information refers to the reward information of the agent corresponding to the other local area network, and is used to represent the network average communication delay and packet loss rate.

[0126] The preset training number refers to the total training number set in advance.

[0127] It should be noted that when the neural network model to be trained of the agent corresponding to the local area network is trained, the first sample observation information, the first sample action information, the second sample observation information, and the second sample action information are comprehensively considered.

[0128] Exemplarily, the terminal obtains first sample observation information of an agent corresponding to a local area network in the computing power network and second sample observation information of an agent corresponding to another local area network in the computing power network, inputs the first sample observation information into a neural network model to be trained of the agent corresponding to the local area network, obtains first sample action information corresponding to the first sample observation information, and inputs the second sample observation information into a neural network model to be trained of the agent corresponding to the other local area network, obtains second sample action information corresponding to the second sample observation information; then, in a case where the first sample observation information, the second sample observation information and the second sample action information are determined, the first sample action information is executed by the agent corresponding to the local area network to obtain next state information of the agent corresponding to the local area network and first reward information of the agent corresponding to the local area network, and then first initial update parameters of the neural network model to be trained of the agent corresponding to the local area network are determined according to the next state information of the agent corresponding to the local area network and the first reward information; meanwhile, in a case where the first sample observation information, the second sample observation information and the first sample action information are determined, the second sample action information is executed by the agent corresponding to the other local area network to obtain next state information of the agent corresponding to the other local area network and second reward information of the agent corresponding to the other local area network, and then second initial update parameters of the neural network model to be trained of the agent corresponding to the other local area network are determined according to the next state information of the agent corresponding to the other local area network and the second reward information; finally, the first initial update parameters are updated according to the second initial update parameters, the second reward information and the first reward information to obtain an updated neural network model of the agent corresponding to the local area network, and the above training process is continuously repeated until a preset training number is reached, the training is stopped, and the updated neural network model of the agent corresponding to the local area network when the preset training number is reached is taken as a trained neural network model of the agent corresponding to the local area network.

[0129] It should be noted that the training process of the trained neural network model of the agent corresponding to the other local area network is consistent with the training process of the trained neural network model of the agent corresponding to the local area network, which will not be described here.

[0130] For example, for an agent in a multi-agent system, the environment explored by the agent itself changes constantly due to the influence of the behaviors of other agents: when the strategies of other agents are different, the change law of the environment is also different. The change law of the environment is expressed as state transition probability P, the current agent is denoted as c, the corresponding strategy is denoted as π c , the total number of agents is N, and the mathematical expression of non-stationarity is as follows: the formula is established in the case where π i ≠ π i '.

[0131] P(s′c |s c a c ,π1,...π c ,...π N )≠P(s′ c |s c a c ,π′1,...π′ c ,...π′ N )

[0132] Reinforcement learning algorithms aim to explore hidden patterns in the environment. Traditional reinforcement learning algorithms operate in stable environments, while in multi-agent algorithms, the exploration object of a single agent is actually the combination of the environment and the patterns formed by other agents. If the problem of environmental non-stationarity is not addressed, then the quadruplets (s) performed by a single agent at different times will become problematic. i ,a i ,s i The environment rules for the quadruple ',r) are different from those for the current time, so mechanisms such as experience replay are not applicable.

[0133] To address the issue of environmental non-stationarity, three main solutions exist: Opponent Modeling, Direct Communication, and Centralized Training and Decentralized Execution (CTDE). In Opponent Modeling, the agent learns the behavior of other agents simultaneously with the policy, exhibiting the strongest inter-agent independence. Training only requires knowledge of the actions and behavioral goals of other agents, consuming less bandwidth but with a longer learning time. Direct Communication involves designing communication channels between agents, allowing them to exchange information during training. The content of this communication needs to be designed according to specific circumstances, resulting in generally weak algorithm scalability. In Centralized Training and Decentralized Execution, agents share training data, combining information from all agents for learning, and making independent decisions during execution. Training requires data from all agents, consuming more bandwidth but offering higher learning efficiency.

[0134] This method deploys one agent in each private network area, which manages the network area it belongs to. The agent independently executes decisions, mainly responsible for routing and adjusting network load. Multi-node network environments can be represented using an undirected graph G = {G1, G2}. 2, ...G N}={V1,E1,V2,E2...V N E N Let} represent, where V i E represents the set of nodes in private network region i. irepresents the set of links existing between nodes of the area. The communication behavior of a routing device is described by a sequence of packet sending behaviors P(V s ,V d ) where V s is the sending node of the packet and V d is the receiving node of the packet. Packet sending behaviors can occur between any two nodes in the overall network.

[0135] The processing capability of a node is modeled by the node parallelism, the node queue size, the queue discard policy and the packet size; the link quality is modeled by the link available computing power and the bandwidth. Multi-agent algorithms usually generalize POMDPs (Partially Observable Markov Decision Processes) to stochastic game models, which are generally described by the tuple (N, S, A1, A2,... A N , P, r1, r2,... r N , γ), where N is the total number of agents deployed in the environment, S is the model environment state space, A i is used to represent the action space of agent i, the joint action space is denoted as A = A1 x A2 x... x A N , P represents the probability of state space transition, the dimension is equal to S x A x S, r i is the reward function of agent i, and γ is the discount factor.

[0136] The characteristic of the stochastic game model is that multiple agents each select their own actions, which jointly act on the environment to cause the state transition of the environment, obtain the next state and reward, and each agent can have an independent reward function directly related to the agent's goal. In a multi-agent environment, the goals of agents can be in conflict, and agents in the process of maximizing their own benefits can cause the benefits of other agents to decrease, so the stochastic game model can be divided into different types such as cooperation and confrontation according to the scene.

[0137] Agent i takes the network topology and the link quality information of the network area to which it belongs as the state observation S i , and the perception of the global network topology is conducive to the algorithm to make reasonable communication decisions: for the communication between nodes in its own area, effective path selection can be made; for cross-area communication, it can be used as a basis for selecting a sending direction. The link weight adjustment matrix of its own area is taken as the output action a i . The reward function of the agent is the average communication delay and packet loss rate of the network.

[0138] Each agent i contains 4 networks: the value function evaluation network Q i (S, A | θi1 ), target evaluation network Q i '(S, A | 0 i2 ), action policy network Act i (S | u i1 ), target action policy network Act i '(S | u i2 ). The management network specifies the agent i in the region to perceive the network topology to obtain the observation S i , and takes this local observation as the input of the action policy network Act i (S | u i1 ) to obtain the action a i selected by the policy in this time slot. In the exploration process, noise of the corresponding dimension is added to the action ou i , and the noise is generated using the OU process. Each agent selects an action, i.e., the weight adjustment matrix of its own network region, and sets the sending logic of the network nodes in this region according to the matrix. After the agent executes the action selection, a set of network link weight data is obtained, and a constrained shortest path algorithm is used to calculate the minimum cost path.

[0139] Each network node executes the action independently without knowing each other, and the overall impact on the environment can be regarded as executing the joint action A = (a i , a2,...a N ). The overall routing strategy of the environment changes, and a round of communication is performed. After the communication process is completed, the agent i performs perception again to obtain a new local observation S i ', and calculates its own reward function r i according to the result. The four-tuple (s i , a i , s i ', r) is added to the experience pool. The MADDPG algorithm uses the CTDE method, which needs to combine the observations of all agents to form a global information set O as the training input of the evaluation network. The specific elements of the set need to be adjusted according to the algorithm and the characteristics of the environment. The simplest information set can be the joint observation of all agents, i.e.,

[0140] O = (S1, S2,...S N )

[0141] The required information can be additionally added in the information set, and O constitutes the overall observation of the Critic network to the environment. The richer the environment-related information contained, the more conducive to the neural network modeling and learning of the potential laws of the environment. The joint data of global information O and all agent actions are input into the Critic network for training, which is the core idea of MADDPG; for the neural network that can only observe local information and its own action, the environment of the multi-agent system is non-stationary; and for the neural network that can observe all agent actions, all factors leading to the change of the environment are known, and the environment itself can be considered as stationary. The MADDPG algorithm deals with the non-stationarity of the environment based on the above theory, so that the multi-agent reinforcement learning problem has the Markov decision property, and the traditional reinforcement learning experience replay training method can be used. Let the current agent be i, and the corresponding strategy be u i , P represents the state transition probability, and its mathematical expression is:

[0142] P(S i '|S i ,...a i ,...a N )=P(S i '|S i ,...a i ,...a N ,u1,...u i ,...u N )

[0143] =P(S i '|S i ,...a i ,...a N ,u1',...u'2,...u' N )

[0144] When the data amount in the experience pool of the agent i is greater than the preset batch (such as the preset data amount), the batch number of experience samples are extracted from the experience pool for training. S i ' in the sample is input into the target policy network Act i '(S|u i2 ), and the next time action a i ' is selected. The next S i ' and a i ' are used to estimate the current update target, such as:

[0145] y i =r i +γQ i '(s i ',a i ')|a i '=Acti '(s i '|u2),i∈[1,N]

[0146]

[0147] All agents update locally, and send the updated parameters and rewards to the Server. The Server uses a competitive aggregation method to update all parameters. The main idea is to normalize the evaluation value, and then judge the pros and cons of the parameters according to the normalized result. More updates are made in the direction of better effect. The reward function value sequence (r1, r2,... r N ) is normalized using the Softmax function to obtain (sr1, sr2,... sr N ), which is used as the aggregation weight for updating the core network, as follows:

[0148]

[0149] After a certain number of evaluation network updates, the strategy gradient is used to update Act i (S|u i1 ). Delayed updating can effectively reduce error accumulation during updating and reduce invalid updates, which is updated as follows:

[0150]

[0151] The update of Act i '(S,A|θ i2 ) in the agent is also delayed, and the update of Act i '(S|u i2 ) is lower than that of the strategy network Act i (S|u i1 ), and the parameters are updated in a soft update manner.

[0152] In this embodiment, a neural network model is iteratively trained for each agent corresponding to a local area network, which is conducive to outputting more accurate routing metric information through the trained neural network model, and further improves the determination accuracy of the routing metric information.

[0153] In an exemplary embodiment, the step S303 updates the first initial update parameter according to the second initial update parameter of the neural network model to be trained of the agent corresponding to the other local area network, the second reward information of the agent corresponding to the other local area network, and the first reward information, and specifically includes the following contents: normalizing the first reward information and the second reward information to obtain processed first reward information and processed second reward information; fusing the difference between the second initial update parameter and the first initial update parameter according to the processed second reward information to obtain a fusion update parameter; and updating the first initial update parameter according to the fusion update parameter to obtain the first target update parameter of the neural network model to be trained of the agent corresponding to the local area network.

[0154] The processed first reward information refers to the normalized first reward information, and the normalized first reward information = the first reward information / (the first reward information + the second reward information). The processed second reward information refers to the normalized second reward information, and the normalized second reward information = the second reward information / (the first reward information + the second reward information).

[0155] The fusion update parameter = (the second initial update parameter - the first initial update parameter) x the processed second reward information.

[0156] The first target update parameter of the neural network model to be trained of the agent corresponding to the local area network = the first initial update parameter + the fusion update parameter.

[0157] Exemplarily, the terminal sends the first initial update parameter, the second initial update parameter, the first reward information, and the second reward information to the server, such as the total control module shown in FIG. 8. Figure 5 The server normalizes the first reward information and the second reward information to obtain the processed first reward information and the processed second reward information, then obtains the difference between the second initial update parameter and the first initial update parameter, and sums the difference between the second initial update parameter and the first initial update parameter according to the processed second reward information to obtain the fusion update parameter, then adds the first initial update parameter and the fusion update parameter to obtain the first target update parameter of the neural network model to be trained of the agent corresponding to the local area network, and finally sends the first target update parameter of the neural network model to be trained of the agent corresponding to the local area network to the terminal. The terminal updates the model parameter of the neural network model to be trained of the agent corresponding to the local area network to the first target update parameter to obtain the updated neural network model.

[0158] It should be noted that the determination process of the second target update parameter of the neural network model to be trained of the agent corresponding to the other local area network is consistent with the determination process of the first target update parameter of the neural network model to be trained of the agent corresponding to the local area network, and details are not repeated here.

[0159] For example, assuming that there are local area network A, local area network B and local area network C, which correspond to agent 1, agent 2 and agent 3 respectively, reward information 1, reward information 2 and reward information 3 respectively, initial update parameter 1, initial update parameter 2 and initial update parameter 3 respectively, then the processed reward information 1 = reward information 1 / (reward information 1+reward information 2+reward information 3), the processed reward information 2 = reward information 1 / (reward information 1+reward information 2+reward information 3), and the processed reward information 3 = reward information 1 / (reward information 1+reward information 2+reward information 3); then, the target update parameter of the neural network model to be trained corresponding to the agent 1 = initial update parameter 1+[processed reward information 2×(initial update parameter 2-initial update parameter 1)+processed reward information 3×(initial update parameter 3-initial update parameter 1)]; the target update parameter of the neural network model to be trained corresponding to the agent 2 = initial update parameter 2+[processed reward information 1×(initial update parameter 1-initial update parameter 2)+processed reward information 3×(initial update parameter 3-initial update parameter 2)]; and the target update parameter of the neural network model to be trained corresponding to the agent 3 = initial update parameter 3+[processed reward information 1×(initial update parameter 1-initial update parameter 3)+processed reward information 2×(initial update parameter 2-initial update parameter 3)].

[0160] In the embodiment, when determining the first target update parameter of the neural network model to be trained of the agent corresponding to the local area network, the second initial update parameter of the neural network model to be trained of the agent corresponding to the other local area network, the second reward information of the agent corresponding to the other local area network, the first initial update parameter of the neural network model to be trained of the agent corresponding to the local area network, and the first reward information of the agent corresponding to the local area network are comprehensively considered, which is beneficial to improve the determination accuracy of the first target update parameter of the neural network model to be trained of the agent corresponding to the local area network.

[0161] In an example embodiment, before determining the routing metric information inside the local area network according to the observation information of the local area network in the computing power network, the step S101 further includes the following contents: receiving the current network topology information of the local area network perceived by the server, and receiving the current computing power resource information of the computing power network perceived by the computing power routing; obtaining the current network state information of the local area network perceived by the agent corresponding to the local area network, and combining the current computing power resource information, the current network topology information and the current network state information to obtain the observation information of the local area network.

[0162] The server can be a total control end, such as the total control module shown in Figure 5 The server can perceive the current network topology information of each local area network.

[0163] The computing power routing can perceive the current computing power resource information of the resource side of the computing power network, such as the cloud resources and edge resources shown in Figure 5

[0164] The agent corresponding to the local area network can perceive the current network state information of the local area network. For example, referring to Figure 4 The agent 1 corresponding to the local area network A can perceive the current network state information of the local area network A, and the agent 2 corresponding to the local area network B can perceive the current network state information of the local area network B.

[0165] For example, the server perceives the current network topology information of the local area network, and sends the perceived current network topology information of the local area network to the agent corresponding to the local area network. At the same time, the computing power routing perceives the current computing power resource information of the computing power network, and sends the perceived current computing power resource information of the computing power network to the agent corresponding to the local area network. In addition, the agent corresponding to the local area network can also perceive the current network state information of the local area network. Then, the terminal combines the current network state information of the local area network perceived by the agent corresponding to the local area network, and the current network topology information of the local area network and the current computing power resource information of the computing power network received by the agent corresponding to the local area network to obtain the observation information of the local area network.

[0166] In this embodiment, by obtaining the current computing power resource information of the computing power network, the current network topology information and the current network state information of the local area network, and taking them as the observation information of the local area network in the computing power network, it is beneficial to subsequently accurately determine the routing metric information inside the local area network based on the various observation information of the local area network in the computing power network, and regenerate the routing decision table, achieving the purpose of dynamic routing, and further improving the data transmission efficiency of the computing power network.

[0167] In an example embodiment, as Figure 5 ​As shown, another dynamic routing method is provided, which is described by taking a terminal as an example, and includes the following steps S401 to S410. Among them:

[0168] Step S401, receiving the current network topology information of the local area network perceived by the service end, and receiving the current computing power resource information of the computing power network perceived by the computing power routing.

[0169] Step S402, obtaining the current network state information of the local area network perceived by the agent corresponding to the local area network, and combining the current computing power resource information, the current network topology information and the current network state information to obtain the observation information of the local area network.

[0170] Step S403, inputting the observation information into the trained neural network model of the agent corresponding to the local area network to obtain the link weight adjustment matrix inside the local area network.

[0171] Step S404, obtaining the initial link weight data inside the local area network; determining the adjustment value of the initial link weight data inside the local area network according to the link weight adjustment matrix.

[0172] Step S405, adjusting the initial link weight data according to the adjustment value of the initial link weight data to obtain the link weight data inside the local area network.

[0173] Step S406, determining the routing metric information inside the local area network according to the link weight data.

[0174] Step S407, determining the routing metric information of the transmission path to each destination address according to the routing metric information inside the local area network.

[0175] Step S408, respectively filtering out the transmission path whose routing metric information meets the preset condition from the transmission path to each destination address as the target transmission path to each destination address.

[0176] Step S409, regenerating the routing decision table of the local area network according to the target transmission path to each destination address.

[0177] Step S410, determining the target transmission path corresponding to the to-be-transmitted data in the local area network according to the regenerated routing decision table.

[0178] In the aforementioned dynamic routing method, when data transmission occurs within the computing power network, the current computing power resource information, the current network topology information, and the current network status information of the local area network (LAN) are comprehensively considered. Based on these observations, a new routing decision table for the LAN is generated, achieving the goal of dynamically changing routing decisions. Then, based on the regenerated routing decision table, the target transmission path corresponding to the data to be transmitted in the LAN is determined, ensuring that the data to be transmitted in the LAN can be transmitted according to the newly determined target transmission path. This ensures that the data to be transmitted in the LAN can be transmitted along the appropriate target transmission path at all times, thus achieving the effect of dynamic routing. This is beneficial for accelerating the overall data transmission speed of the computing power network, thereby improving the data transmission efficiency of the computing power network. Simultaneously, it avoids the drawback of using static routing, which leads to slower overall data transmission and consequently lower data transmission efficiency in the computing power network.

[0179] To more clearly illustrate the dynamic routing method provided in the embodiments of this application, the following describes the dynamic routing method in detail with a specific embodiment. In an exemplary embodiment, such as Figure 5 As shown, this application also provides a distributed intelligent routing system for computing power networks, which centrally applies the multi-agent DDPG algorithm based on machine learning principles to the computing power routing system. Considering the current scale of computing power networks, different local area networks are independently modeled as a distributed subsystem, and objective factors such as network status are integrated, and distributed computing resources such as node cloud, edge, and terminal resource pools, as well as dynamic network resources such as latency, bandwidth, and packet loss rate are coordinated to obtain the composition and function of each module in the distributed routing system architecture. Compared with the path selection method in traditional computing power routing systems, this application proposes to use the multi-agent DDPG algorithm to not only solve the scaling problem in distributed local area subnets, but also to obtain the routing metric value in the actual router routing process under dynamic changes in network topology and network status, thereby changing the routing decision for a large number of concurrent requests in the local area network. It does not require determining a unique path decision for each request, and does not rely on technologies such as SDN (Software Defined Network) and SRv6 (Segment Routing IPv6), making the system and algorithm more universal.

[0180] In view of the computing power resource demand and computing power scheduling of a large number of users in the current computing power network application background, the application constructs a distributed intelligent routing system, optimizes the path selection method in the routing system, dynamically adjusts the routing metric value in the local area subnet, generates a routing decision table, and guides the optimal selection of the user request path to ensure the quality of service. Specifically, first, a distributed intelligent routing system is constructed, and the computing power perception module perceives the user side computing power request and the resource side computing power size. The data forwarding template updates the perceived computing power information by flooding, so that the link state information contains the deployment location, real-time state, load information and service demand of the computing power service. The control plane collects a large number of user requests and network state changes, and performs multi-agent dynamic routing optimization learning. The optimization goal is to reduce the low latency and low packet loss rate of concurrent requests in the local area network. According to the optimization result, the routing metric value in the computing power network is adjusted, that is, the routing metric value is adjusted based on the multi-agent reinforcement learning method, so as to optimize the path selection of the user request and realize the optimal end-to-end service quality.

[0181] Reference Figure 6 The distributed intelligent routing system comprises:

[0182] User side: including vehicle networking, large model, image rendering and other scenes, these applications need a large amount of computing resources, and may have low delay demand.

[0183] Total control module: 1, deploy a group of core evaluation neural networks, train the evaluation by collecting the parameters and reward values of all agents in the distributed module, then normalize the evaluation value, and evaluate the pros and cons of the parameters according to the normalization result, and update more in the direction of better effect; 2, perceive the network topology structure change in the large-scale local area network and issue to the distributed module.

[0184] Distributed module: 1, each agent i deploys a training neural network, including a value function evaluation network, a target evaluation network, an action policy network and a target action policy network, and the optimal routing decision strategy is trained through neural network learning; 2, the agent i specified in the local area network perceives the network topology and network state, and perceives the resource side computing power resource together to obtain the observation result as the input of the neural network; 3, the agent i outputs the action, that is, adjusts the routing metric value in the local area network, and the local area network regenerates the routing decision table according to the routing metric value.

[0185] Resource side: the computing resources are divided into cloud resources and edge resources to meet the computing power demand of different applications. The cloud resources have strong computing power and are suitable for complex tasks; the edge resources are close to the user and are suitable for low delay tasks.

[0186] The above embodiments can achieve the following technical effects: (1) In the distributed intelligent routing system, the total control module comprehensively considers all computing power resources and network states, and dynamically issues optimal routing decisions through intelligent routing learning, thereby ensuring the optimal service quality of users; (2) Through the multi-agent reinforcement learning method, the high-concurrency request of a large number of users is solved, the quality of collaborative communication of multi-region networks is ensured, and the overall average time delay and packet loss rate are optimized.

[0187] It should be understood that, although each step in the flowchart involved in each embodiment as described above is shown in sequence according to the arrow, these steps are not necessarily executed in sequence according to the arrow. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other sequences. Moreover, at least part of the steps in the flowchart involved in each embodiment as described above can include multiple steps or stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution sequence of these steps or stages is not necessarily sequential, but can be executed in rotation or alternation with at least part of other steps or steps or stages in other steps.

[0188] Based on the same inventive concept, the embodiments of the present application also provide a dynamic routing device for implementing the above-mentioned dynamic routing method. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme described in the above method, so the specific limitations in one or more dynamic routing device embodiments provided below can refer to the limitations of the dynamic routing method in the above text, which will not be repeated here.

[0189] In one exemplary embodiment, as shown in Figure 7 a dynamic routing device is provided, comprising: an information determination module 610, a regeneration module 620 and a path determination module 630, wherein:

[0190] The information determination module 610 is configured to determine routing metric information inside a local area network according to observation information of the local area network in a computing power network; the observation information at least includes current computing power resource information of the computing power network, current network topology information and current network state information of the local area network.

[0191] The regeneration module 620 is configured to regenerate a routing decision table of the local area network according to the routing metric information.

[0192] The path determination module 630 is configured to determine a target transmission path corresponding to to-be-transmitted data in the local area network according to the regenerated routing decision table.

[0193] In an example embodiment, the regeneration module 620 is further configured to determine routing metric information of transmission paths to each destination address according to the routing metric information inside the local area network; filter transmission paths whose routing metric information satisfies preset conditions from the transmission paths to each destination address respectively, as target transmission paths to each destination address; and regenerate the routing decision table of the local area network according to the target transmission paths to each destination address.

[0194] In an example embodiment, the information determination module 610 is further configured to input the observation information into the trained neural network model of the agent corresponding to the local area network to obtain a link weight adjustment matrix inside the local area network; determine link weight data inside the local area network according to the link weight adjustment matrix; and determine routing metric information inside the local area network according to the link weight data.

[0195] In an example embodiment, the information determination module 610 is further configured to obtain initial link weight data inside the local area network; determine adjustment values of the initial link weight data inside the local area network according to the link weight adjustment matrix; and adjust the initial link weight data according to the adjustment values of the initial link weight data to obtain the link weight data inside the local area network.

[0196] In an example embodiment, the dynamic routing device further includes a model training module configured to obtain first sample observation information of an agent corresponding to a local area network in a computing power network and first sample action information corresponding to the first sample observation information, and second sample observation information of an agent corresponding to another local area network in the computing power network and second sample action information corresponding to the second sample observation information; train a neural network model to be trained of the agent corresponding to the local area network according to the first sample observation information, the first sample action information, the second sample observation information and the second sample action information, to obtain first initial update parameters of the trained neural network model of the agent corresponding to the local area network, and first reward information of the agent corresponding to the local area network; update the first initial update parameters according to second initial update parameters of a neural network model to be trained of the agent corresponding to the other local area network, second reward information of the agent corresponding to the other local area network and the first reward information, until a preset training number is reached, to obtain the trained neural network model of the agent corresponding to the local area network.

[0197] In an example embodiment, the model training module is further configured to normalize the first reward information and the second reward information to obtain processed first reward information and processed second reward information; perform fusion processing on a difference between the second initial update parameter and the first initial update parameter according to the processed second reward information to obtain a fusion update parameter; and update the first initial update parameter according to the fusion update parameter to obtain a first target update parameter of the neural network model to be trained of the agent corresponding to the local area network.

[0198] In an example embodiment, the dynamic routing device further includes an information acquisition module configured to receive current network topology information of the local area network sensed by the server, and receive current computing power resource information of the computing power network sensed by the computing power routing; acquire current network state information of the local area network sensed by the agent corresponding to the local area network, and combine the current computing power resource information, the current network topology information, and the current network state information to obtain observation information of the local area network.

[0199] The above modules in the dynamic routing device can be all or partially implemented by software, hardware, and combinations thereof. The above modules can be embedded in or independent of a processor in a computer device in hardware form, or can be stored in a memory in the computer device in software form, so as to be called and executed by a processor to perform operations corresponding to the above modules.

[0200] In an example embodiment, a computer device is provided, which can be a terminal, and an internal structure diagram of the computer device can be as shown in Figure 7The computer device shown in the figure includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capability. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The input / output interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with external terminals in a wired or wireless manner. The wireless manner can be realized through WIFI, mobile cellular network, near field communication (Near Field Communication, NFC) or other technologies. The computer program is executed by the processor to realize a dynamic routing method. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer overlaid on the display screen, or a key, trackball or touchpad arranged on the shell of the computer device, or an external keyboard, touchpad or mouse, etc.

[0201] Those skilled in the art can understand that, ​ The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0202] In one embodiment, a computer device is also provided, including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to realize the steps in each of the above method embodiments.

[0203] In one embodiment, a computer readable storage medium is provided, which stores a computer program, and the computer program is executed by a processor to realize the steps in each of the above method embodiments.

[0204] In one embodiment, a computer program product is provided, including a computer program, and the computer program is executed by a processor to realize the steps in each of the above method embodiments.

[0205] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant regulations.

[0206] It can be understood by those skilled in the art that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing related hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments of each method. In the embodiments provided in the present application, any reference to memory, database or other medium can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (Read-Only Memory, ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (Resistive Random Access Memory, ReRAM), magnetoresistive random access memory (Magnetoresistive Random Access Memory, MRAM), ferroelectric memory (Ferroelectric Random Access Memory, FRAM), phase change memory (Phase Change Memory, PCM), graphene memory, etc. Volatile memory can include random access memory (Random Access Memory, RAM) or external cache memory, etc. As an illustration but not limitation, RAM can be in various forms, such as static random access memory (Static Random Access Memory, SRAM) or dynamic random access memory (Dynamic Random Access Memory, DRAM), etc. The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without being limited thereto. The processor involved in the embodiments provided in the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (Artificial Intelligence, AI) processor, etc., without being limited thereto.

[0207] Any technical features in the above embodiments can be combined, and for the sake of brevity, not all possible combinations are described above, however, any combination of these technical features is deemed to be within the scope of the present application.

[0208] The above embodiments only express several implementation manners of the present application, and the description is relatively specific and detailed, but it should not be understood as a limitation on the patent scope of the present application. It should be pointed out that, for ordinary skilled persons in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A dynamic routing method, characterized by, The method comprises: According to the observation information of the local area network in the computing power network, the routing metric information inside the local area network is determined; the observation information at least includes the current computing power resource information of the computing power network, the current network topology information and the current network state information of the local area network; According to the routing metric information, the routing decision table of the local area network is regenerated; According to the regenerated routing decision table, the target transmission path corresponding to the to-be-transmitted data in the local area network is determined; The routing metric information is determined by the trained neural network model of the agent corresponding to the local area network based on the observation information; The trained neural network model of the agent corresponding to the local area network is obtained by training in the following way: Obtain the first sample observation information of the agent corresponding to the local area network in the computing power network and the first sample action information corresponding to the first sample observation information, and the second sample observation information of the agent corresponding to the other local area network in the computing power network and the second sample action information corresponding to the second sample observation information; According to the first sample observation information, the first sample action information, the second sample observation information and the second sample action information, the to-be-trained neural network model of the agent corresponding to the local area network is trained to obtain the first initial update parameter of the to-be-trained neural network model of the agent corresponding to the local area network, and the first reward information of the agent corresponding to the local area network; According to the second initial update parameter of the to-be-trained neural network model of the agent corresponding to the other local area network, the second reward information of the agent corresponding to the other local area network and the first reward information, the first initial update parameter is updated until a preset training number is reached, and the trained neural network model of the agent corresponding to the local area network is obtained; the first reward information and the second reward information are used to represent the network average communication delay and the packet loss rate.

2. The method of claim 1, wherein, The regenerated routing decision table of the local area network according to the routing metric information comprises: According to the routing metric information inside the local area network, the routing metric information of the transmission path to each destination address is determined; Respectively, from the transmission path to each destination address, the transmission path whose routing metric information satisfies the preset condition is selected as the target transmission path to each destination address; According to the target transmission path to each destination address, the routing decision table of the local area network is regenerated.

3. The method of claim 1, wherein, The routing metric information inside the local area network is determined according to the observation information of the local area network in the computing power network, which comprises: The observation information is input into the trained neural network model of the agent corresponding to the local area network to obtain the link weight adjustment matrix inside the local area network; According to the link weight adjustment matrix, the link weight data inside the local area network is determined; According to the link weight data, the routing metric information inside the local area network is determined.

4. The method of claim 3, wherein, According to the link weight adjustment matrix, the link weight data inside the local area network is determined, which comprises: Obtain the initial link weight data inside the local area network; According to the link weight adjustment matrix, an adjustment value of initial link weight data inside the local area network is determined; According to the adjustment value of the initial link weight data, the initial link weight data is adjusted to obtain the link weight data inside the local area network.

5. The method of claim 1, wherein, The updating of the first initial update parameter according to the second initial update parameter of the neural network model to be trained of the agent corresponding to the other local area network, the second reward information of the agent corresponding to the other local area network and the first reward information comprises: The first reward information and the second reward information are normalized to obtain processed first reward information and processed second reward information; According to the processed second reward information, the difference between the second initial update parameter and the first initial update parameter is fused to obtain a fused update parameter; According to the fused update parameter, the first initial update parameter is updated to obtain the first target update parameter of the neural network model to be trained of the agent corresponding to the local area network.

6. The method according to any one of claims 1 to 5, characterized in that, Before determining the routing metric information inside the local area network according to the observation information of the local area network in the computing power network, the method further comprises: Receiving current network topology information of the local area network perceived by the server, and receiving current computing power resource information of the computing power network perceived by the computing power routing; Obtaining current network state information of the local area network perceived by the agent corresponding to the local area network, and combining the current computing power resource information, the current network topology information and the current network state information to obtain the observation information of the local area network.

7. A dynamic routing device, characterized by The device comprises: An information determination module is configured to determine routing metric information inside a local area network according to observation information of the local area network in a computing power network; the observation information at least comprises current computing power resource information of the computing power network, current network topology information and current network state information of the local area network; A regeneration module is configured to regenerate a routing decision table of the local area network according to the routing metric information; A path determination module is configured to determine a target transmission path corresponding to to-be-transmitted data in the local area network according to the regenerated routing decision table. The routing metric information is determined by a trained neural network model of an agent corresponding to the local area network based on the observation information; the device further comprises a model training module configured to: obtain first sample observation information of an agent corresponding to the local area network in the computing power network and first sample action information corresponding to the first sample observation information, and second sample observation information of an agent corresponding to another local area network in the computing power network and second sample action information corresponding to the second sample observation information; train the to-be-trained neural network model of the agent corresponding to the local area network according to the first sample observation information, the first sample action information, the second sample observation information and the second sample action information, to obtain first initial update parameters of the to-be-trained neural network model of the agent corresponding to the local area network and first reward information of the agent corresponding to the local area network; update the first initial update parameters according to second initial update parameters of a to-be-trained neural network model of the agent corresponding to the other local area network, second reward information of the agent corresponding to the other local area network and the first reward information, until a preset training number is reached, to obtain the trained neural network model of the agent corresponding to the local area network; and the first reward information and the second reward information are both used to represent network average communication delay and packet loss rate.

8. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that, The processor, when executing the computer program, implements the steps of the method of any one of claims 1 to 6.

9. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program, when executed by the processor, implements the steps of the method of any one of claims 1 to 6.

10. A computer program product comprising a computer program, characterized in that, The computer program, when executed by the processor, implements the steps of the method of any one of claims 1 to 6. The computer program, when executed by the processor, implements the steps of the method of any one of claims 1 to 6.

Images