Routing decision model training method, device, computer equipment and storage medium

Abstract

The embodiment of the present application discloses a method, device, computer equipment and storage medium for training a routing decision model. A first node is determined from multiple nodes in a dynamic network, and a first network state and a first network action corresponding to the first node are determined; the first network state and the first network action are input into an evaluation sub-model, and an evaluation value of each first network action is output; the target first network action with the largest evaluation value and the second node of the target first network action are determined; the reward value of the second network action, the second network state and the target first network action of the second node pair is input into the target sub-model, and the target value of the target first network action is output; the model loss of the evaluation sub-model is determined according to the target value and the largest evaluation value, the model parameters of the evaluation sub-model are adjusted according to the model loss, and the execution is returned to determine the first node from multiple nodes in the dynamic network until the model loss meets the preset loss condition, and the trained routing decision model is obtained.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, apparatus, computer equipment and storage medium for training a routing decision model. Background Art

[0002] With the development of communication technology, multiple network devices can be used as network nodes, and network nodes can form a network, thereby realizing data transmission through the network. However, the location of network nodes will change, so it is necessary to determine the network node with the highest reliability among multiple network nodes for data transmission.

[0003] In the related art, one method is to obtain data changes within a fixed network structure, such as the data flow of data packets in the network or changes in the status of network nodes, and then use the corresponding algorithm to determine the path of data transmission. However, these changes do not include changes in the location of network nodes, so it is impossible to accurately determine the optimal network path for data transmission. Another method is to obtain routing tables corresponding to multiple network nodes, and then dynamically select routing protocols through the routing tables. However, in a dynamically changing network composed of multiple network devices whose locations change dynamically, this method has problems such as slow response speed, high maintenance and management costs of routing tables, and packet transmission loss, and cannot accurately determine the optimal network path for data transmission.

[0004] Therefore, in the related art, when faced with a dynamically changing network composed of multiple network devices whose locations change dynamically, it is impossible to accurately determine the optimal network path for data transmission. Summary of the invention

[0005] The embodiments of the present application provide a routing decision model training method, apparatus, computer equipment and storage medium, which can accurately determine the most reliable next hop node for data transmission in a dynamic network through a trained routing decision model.

[0006] According to one aspect of the present application, an embodiment of the present application provides a method for training a routing decision model, wherein the routing decision model includes an evaluation sub-model and a target sub-model, and the method includes:

[0007] Determine a first node from among multiple nodes of a dynamic network, and determine a first network state and a first network action corresponding to the first node;

[0008] Inputting the first network state and the first network action into the evaluation sub-model, and outputting an evaluation value corresponding to each of the first network actions;

[0009] Determine a target first network action corresponding to the largest evaluation value and a second node corresponding to the target first network action;

[0010] Inputting the second network action corresponding to the second node, the second network state and the reward value corresponding to the target first network action into the target sub-model, and outputting the target value corresponding to the target first network action;

[0011] The model loss corresponding to the evaluation sub-model is determined according to the target value and the maximum evaluation value, and the model parameters corresponding to the evaluation sub-model are adjusted according to the model loss. The execution is returned to determine the first node among multiple nodes in the dynamic network until the model loss meets the preset loss condition, thereby obtaining a trained routing decision model.

[0012] According to one aspect of the present application, an embodiment of the present application provides a routing decision model training device, wherein the routing decision model includes an evaluation sub-model and a target sub-model, and the device includes:

[0013] A first determination module, configured to determine a first node from among a plurality of nodes in a dynamic network, and determine a first network state and a first network action corresponding to the first node;

[0014] A first input module, configured to input the first network state and the first network action into the evaluation sub-model, and output an evaluation value corresponding to each of the first network actions;

[0015] A second determination module, used to determine a target first network action corresponding to the maximum evaluation value and a second node corresponding to the target first network action;

[0016] A second input module, configured to input a second network action corresponding to the second node, a second network state, and a reward value corresponding to the target first network action into the target sub-model, and output a target value corresponding to the target first network action;

[0017] The loss module is used to determine the model loss corresponding to the evaluation sub-model according to the target value and the maximum evaluation value, and adjust the model parameters corresponding to the evaluation sub-model according to the model loss, and return to execute to determine the first node among multiple nodes in the dynamic network until the model loss meets the preset loss condition, thereby obtaining a trained routing decision model.

[0018] In some embodiments, the first input module includes a first sub-input module, a second sub-input module, and a first determination sub-module;

[0019] A first sub-input module, used for inputting the first network state into a first neural network, and outputting a state value corresponding to each of the first network actions;

[0020] A second sub-input module, configured to input the first network state and the first network action into a second neural network, and output an advantage value corresponding to each of the first network actions;

[0021] The first determining submodule is used to determine the evaluation value corresponding to each of the first network actions according to the state value and the advantage value.

[0022] In some implementations, the determining submodule is configured to determine the number of all the first network actions, and to obtain an average advantage value by adding up all the advantage values ​​and dividing the sum by the number;

[0023] Subtract the average advantage value from each advantage value to obtain a target advantage value corresponding to each of the first network actions;

[0024] The target advantage value is added to the state value to obtain an evaluation value corresponding to each of the first network actions.

[0025] In some embodiments, a loss module is used to obtain a difference by subtracting the maximum evaluation value from the target value;

[0026] The model loss corresponding to the evaluation sub-model is determined according to the square value of the difference.

[0027] In some embodiments, the routing decision model training device further includes a feature acquisition module, a third determination module, and a fourth determination module;

[0028] Before determining the first node among the multiple nodes in the dynamic network, a feature acquisition module is used to acquire node features corresponding to each node in the dynamic network;

[0029] A third determination module is used to determine the neighbor nodes corresponding to each of the nodes, and the neighbor node features corresponding to the neighbor nodes;

[0030] The fourth determination module is used to determine the network status of each node relative to other nodes according to the node characteristics corresponding to each node and the corresponding neighbor node characteristics.

[0031] In some embodiments, the fourth determination module includes an attention submodule, a second determination submodule, and a third determination submodule;

[0032] An attention submodule, configured to determine an attention score between the node feature and each of the neighboring node features;

[0033] A second determination submodule is used to determine the target node feature corresponding to each of the nodes according to the attention score;

[0034] The third determination submodule is used to determine the network status of each of the nodes relative to other nodes according to the target node characteristics.

[0035] In some implementations, the second determination submodule is used to determine the value vector weight corresponding to each of the neighbor nodes;

[0036] The value vector weight corresponding to each of the neighbor nodes, the attention score and the neighbor node feature are multiplied to obtain a target result, and the target results are summed to obtain a target node feature corresponding to each of the nodes.

[0037] In some embodiments, the third determination submodule is used to perform one-hot encoding on the target node feature of each node according to the connection status of each node with other nodes in the network to generate a node structure feature;

[0038] Performing an OR operation on the node structure feature of each node and the node structure feature of the corresponding neighboring node to obtain an OR operation result;

[0039] Determine the destination transmission node corresponding to each of the nodes and the node structure characteristics of the destination transmission node, perform an OR operation on the OR operation result and the node structure characteristics of the destination transmission node, and obtain the network status of each of the nodes relative to other nodes.

[0040] In some embodiments, the loss module is further used to determine the number of iterations corresponding to the evaluation sub-model after adjusting the model parameters corresponding to the evaluation sub-model according to the model loss;

[0041] When the number of iterations reaches a preset number of iterations, the model parameters of the evaluation sub-model are obtained, and the model parameters are updated to the target sub-model.

[0042] In some embodiments, the loss module is further configured to obtain experience data corresponding to the evaluation sub-model after each training after determining the model loss corresponding to the evaluation sub-model according to the target value and the maximum evaluation value, wherein the experience data includes the first network state, the target first network action, the reward value, and the second network state;

[0043] The experience data is stored in a training data set, and the training data in the training data set is used to train the routing decision model.

[0044] According to one aspect of the present application, an embodiment of the present application provides a computer-readable storage medium, which stores multiple instructions, and the instructions are suitable for a processor to load to execute the routing decision model training method provided by the embodiment of the present application.

[0045] According to one aspect of the present application, an embodiment of the present application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the routing decision model training method provided in the embodiment of the present application is implemented.

[0046] In an embodiment of the present application, a routing decision model includes an evaluation sub-model and a target sub-model, by determining a first node from a plurality of nodes in a dynamic network, and determining a first network state and a first network action corresponding to the first node; inputting the first network state and the first network action into the evaluation sub-model, and outputting an evaluation value corresponding to each first network action; determining a target first network action corresponding to the maximum evaluation value and a second node corresponding to the target first network action; inputting a second network action corresponding to the second node, a second network state, and a reward value corresponding to the target first network action into the target sub-model, and outputting a target value corresponding to the target first network action; determining a model loss corresponding to the evaluation sub-model according to the target value and the maximum evaluation value, and adjusting the model parameters corresponding to the evaluation sub-model according to the model loss, returning to execute the first node determined from a plurality of nodes in the dynamic network until the model loss meets a preset loss condition, and obtaining a trained routing decision model. In this way, the target first network action with the largest evaluation value is determined among all executable first network actions of the first node through the evaluation sub-model, and the target value corresponding to the target first network action is determined through the target sub-model. The target value is determined based on the reward value obtained after the execution of the target first network action. The reward value is a real value, so the target value is more authentic than the evaluation value. The model loss calculated by taking the target value as a label and the evaluation value is more accurate, so that the model parameters of the evaluation sub-model can be adjusted more accurately, so that the evaluation sub-model can be continuously iterated according to the network state changes of the nodes in the dynamic network to adapt to the dynamic network until the trained routing decision model is obtained. Compared with the related art that cannot determine the optimal transmission path of data in the dynamic network, the present application can accurately determine the most reliable next hop node for data transmission in the dynamic network through the trained routing decision model to achieve data transmission.

[0047] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0049] Figure 1 is a schematic diagram of a dynamic network scenario provided by an embodiment of the present application;

[0050] Figure 2 It is a flowchart of a routing decision model training method provided in an embodiment of the present application;

[0051] Figure 3 is a schematic diagram of a flow chart of obtaining a network status provided in an embodiment of the present application;

[0052] Figure 4 is a schematic diagram of obtaining target node features of each node provided in an embodiment of the present application;

[0053] Figure 5 is a schematic diagram of a routing decision model provided in an embodiment of the present application;

[0054] Figure 6 is a flowchart of step 120 provided in an embodiment of the present application;

[0055] Figure 7 is a schematic diagram of the structure of the evaluation sub-model provided in the embodiment of the present application;

[0056] Figure 8 It is another flowchart of the routing decision model training method provided in an embodiment of the present application;

[0057] Fig. 9 It is a structural diagram of a routing decision model training device provided in an embodiment of the present application;

[0058] Fig.10 It is a schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0059] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0060] It can be understood that in the specific implementation of this application, related data such as network status data is involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards.

[0061] It should be noted that in some processes described in the specification, claims and the above-mentioned drawings, multiple steps appearing in a specific order are included, but it should be clearly understood that these steps may not be executed in the order in which they appear in this document or may be executed in parallel. The step numbers are only used to distinguish different steps, and the numbers themselves do not represent any execution order. In addition, descriptions such as "first", "second" or "target" in this document are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0062] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0063] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0064] Before further describing the embodiments of the present disclosure in detail, the nouns and terms involved in the embodiments of the present disclosure are described. The nouns and terms involved in the embodiments of the present disclosure are subject to the following interpretations:

[0065] Deep-Q-Network (DQN) is a reinforcement learning algorithm that uses deep neural networks to estimate the Q-value function, which can handle problems with high-dimensional state space. DQN avoids the problem of Q-table storage and updating in traditional Q learning by using deep neural networks to approximate the Q function. It obtains an accurate Q function by training deep neural networks to minimize the gap between the predicted Q value and the true Q value.

[0066] In reinforcement learning, the Q function (also called Q value or action value function) is a core concept used to describe the expected long-term return of taking an action in a given state. The Q function is usually represented by Q(s, a), where s represents the current state and a represents the action taken in the current state.

[0067] Graph Attention Network (GAT) is a variant of Graph Neural Network (GNN) designed for processing graph structured data. It introduces an attention mechanism to assign weights between different nodes in the graph, thereby capturing the dependencies between adjacent nodes in the graph.

[0068] Rectified Linear Unit (ReLU), also known as the rectified linear unit, is an activation function commonly used in artificial neural networks, usually referring to nonlinear functions represented by ramp functions and their variants.

[0069] One-hot encoding, also known as one-bit effective encoding, uses an N-bit state register to encode N states, each state has its own independent register bit, and at any time, only one of them is valid. One-hot encoding is a method of converting categorical variables into a format that is easy for machine learning algorithms to handle. Its core idea is to create a new binary feature for each possible value of each categorical feature, where only one feature is activated (marked as 1) at any given time, and all other features are marked as 0.

[0070] A Markov decision process (MDP) is a mathematical model used to describe decision-making problems in a fully observable environment, where each decision may affect future states and rewards. It consists of the following core parts: States: various situations that the system may be in. Actions: the actions that can be chosen in each state. Transition Probabilities: the probability of transitioning to another state after performing an action. Reward Function: the immediate reward obtained after performing an action. Decision Rule: The decision rule or policy defines which action should be chosen in a given state. The goal is to find a policy to maximize the long-term reward calculated in some way.

[0071] First, let’s describe the technical problems existing in the relevant technologies:

[0072] With the development of communication technology, multiple network devices can be used as network nodes, and network nodes can form a network, thereby realizing data transmission through the network. However, the location of network nodes will change, so it is necessary to determine the network node with the highest reliability among multiple network nodes for data transmission.

[0073] In the related art, one method is to obtain data changes within a fixed network structure, such as the data flow of data packets in the network or changes in the status of network nodes, and then use the corresponding algorithm to determine the path of data transmission. However, these changes do not include changes in the location of network nodes, so it is impossible to accurately determine the optimal network path for data transmission. Another method is to obtain routing tables corresponding to multiple network nodes, and then dynamically select routing protocols through the routing tables. However, in a dynamically changing network composed of multiple network devices whose locations change dynamically, this method has problems such as slow response speed, high maintenance and management costs of routing tables, and packet transmission loss, and cannot accurately determine the optimal network path for data transmission.

[0074] Therefore, in the related art, when faced with a dynamically changing network composed of multiple network devices whose locations change dynamically, it is impossible to accurately determine the optimal network path for data transmission.

[0075] In order to solve the above technical problems, in an embodiment of the present application, a routing decision model is trained, and the routing decision model is used to determine the next node for data transmission that is most reliable for the current node in the dynamic network, thereby realizing data transmission. The routing decision model includes an evaluation sub-model and a target sub-model. The evaluation sub-model is used to determine the target first network action with the largest evaluation value among all executable first network actions of the first node, and the target value corresponding to the target first network action is determined by the target sub-model. The target value is determined based on the reward value obtained after the execution of the target first network action. The reward value is a real value, and the target value is more authentic than the evaluation value. The model loss calculated by using the target value as a label and the evaluation value is more accurate, so that the model parameters of the evaluation sub-model can be adjusted more accurately, so that the evaluation sub-model can be continuously iterated according to the network state changes of the nodes in the dynamic network to adapt to the dynamic network until the trained routing decision model is obtained. Compared with the related art that cannot determine the optimal transmission path of data in the dynamic network, the present application can accurately determine the next hop node with the most reliable data transmission in the dynamic network through the trained routing decision model to realize data transmission.

[0076] The routing decision model training method, apparatus, computer equipment and storage medium provided in the embodiments of the present application are described in detail below.

[0077] The embodiments of the present application provide a method, device, computer equipment and storage medium for training a routing decision model. Specifically, the embodiments of the present application will be described from the dimension of a routing decision model training device, which can be integrated in a computer device, which can be a terminal or a server. Among them, the terminal can be a drone, an unmanned ship, a mobile phone, a computer, a mobile device, etc. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The embodiments of the present application can be applied to various scenarios, including but not limited to data transmission, security warning, information collection and other scenarios.

[0078] See also Figure 1 , Figure 1 It is a schematic diagram of a scenario of a dynamic network provided in an embodiment of the present application.

[0079] In the embodiments of the present application, a dynamic network refers to a network in which nodes change position. For example, each node corresponds to a network device, and the network device can move in the real physical space. For example, drones, unmanned ships, robots, etc. can move physically. A network composed of multiple network devices that move physically is a dynamic network.

[0080] like Figure 1 As shown, in a sea area, there are multiple unmanned ships, which move on the sea surface with the waves and sea breeze. At the same time, the unmanned ships themselves also move. Multiple unmanned ships can communicate with each other. Each unmanned ship serves as a network node, and the network composed of multiple unmanned ships is a dynamic network.

[0081] It is understandable that in this dynamic network, since the nodes are dynamically changing, if a node needs to transmit data, it is necessary to accurately find the most reliable next-hop node among multiple nodes, and then the next-hop node accurately finds the most reliable transmission node, thus achieving high-quality data transmission, thereby avoiding packet loss, high latency and other phenomena during data transmission.

[0082] In the embodiment of the present application, the trained routing decision model can be applied to each node. Taking a certain node as an example, the network status corresponding to the node and the executable network action corresponding to each node can be obtained, and then the network status and the executable network action can be input into the trained routing decision model. According to the output result of the trained routing decision model, the output result with the largest predicted value is found, and finally the network action corresponding to the output result with the largest predicted value is executed, so as to realize the transmission of data from the node to the most reliable next-hop node, such as the next node with the best communication quality, so as to transmit the data to the node.

[0083] Therefore, the trained routing decision model in the embodiment of the present application can accurately determine the most reliable next hop node for data transmission in a dynamic network to achieve data transmission.

[0084] See also Figure 2 , Figure 2 1 is a flow chart of a method for training a routing decision model provided by an embodiment of the present application. The method for training a routing decision model can be performed by a routing decision model training device, and the routing decision model training device can be set in a computer device. The routing decision model includes an evaluation sub-model and a target sub-model, and the method for training a routing decision model can include the following steps:

[0085] Step 110: Determine a first node from among multiple nodes in the dynamic network, and determine a first network state and a first network action corresponding to the first node;

[0086] Step 120: input the first network state and the first network action into the evaluation sub-model, and output the evaluation value corresponding to each first network action;

[0087] Step 130: determine the target first network action corresponding to the maximum evaluation value and the second node corresponding to the target first network action;

[0088] Step 140: input the second network action corresponding to the second node, the second network state, and the reward value corresponding to the target first network action into the target sub-model, and output the target value corresponding to the target first network action;

[0089] Step 150, determine the model loss corresponding to the evaluation sub-model according to the target value and the maximum evaluation value, and adjust the model parameters corresponding to the evaluation sub-model according to the model loss, return to execute to determine the first node among multiple nodes in the dynamic network until the model loss meets the preset loss condition, and obtain the trained routing decision model.

[0090] Before describing steps 110 to 150 in detail, the acquisition of training data for the routing decision model is first described.

[0091] The routing decision model corresponds to a training data set, which includes multiple groups of training data. Each group of training data includes the network state corresponding to the node at the current time, the network action corresponding to the node at the current time, the reward value corresponding to the node after executing the network action at the current time, and the network state corresponding to the node at the next time.

[0092] The network state includes the network congestion state between the node and other nodes. The network action is the action that the node can transmit data to the next node, which can be specifically represented by the next node that the node can transmit to. The reward value is the feedback value received by the node from the environment after executing the network action, which is used to evaluate the quality of the network action.

[0093] Please also read Figure 3 , Figure 3 is a flow chart of obtaining network status provided by an embodiment of the present application. In some implementations, before determining the first node among multiple nodes of the dynamic network, the following steps are also included:

[0094] Step 201: Obtain node features corresponding to each node in the dynamic network;

[0095] Step 202: Determine the neighbor nodes corresponding to each node, and the neighbor node features corresponding to the neighbor nodes;

[0096] Step 203: Determine the network status of each node relative to other nodes based on the node characteristics corresponding to each node and the corresponding neighbor node characteristics.

[0097] Steps 201 to 203 are described in detail below.

[0098] In step 201, node features corresponding to each node in the dynamic network are obtained.

[0099] In a dynamic network, there are multiple nodes. Multiple nodes can be set in a space. The node location information of each node can be determined in the space. The node location information can be the relative location information between nodes. The data transmission busyness of each node can also be obtained. For example, the amount of data that each node can transmit per unit time is limited. The data transmission busyness of each node can be determined based on the maximum data transmission amount of each node and the average data transmission amount of each node in the recent period of time. The node location information and / or data transmission busyness of each node can be encoded to generate the node characteristics of each node.

[0100] It should be noted that the above is only one way to determine the node feature of a node, and is not limited here. The node feature can reflect the unique features of the node relative to other nodes in the dynamic network, and can be a vector. The node feature of each node in the dynamic network is in the same feature space dimension.

[0101] In step 202, the neighbor nodes corresponding to each node and the neighbor node features corresponding to the neighbor nodes are determined.

[0102] In a dynamic network, each node corresponds to a neighbor node. The neighbor node can be understood as the next hop node of the node. The neighbor nodes of each node can be determined, and then each node and its neighbor nodes are regarded as a group of nodes, which can be specifically expressed as: .in is a set of node features of a group of nodes, is the node feature corresponding to each node in this group of nodes, N is the number of nodes in a group of nodes, is a node in a set of nodes, is a space with feature dimension F.

[0103] In step 203, the network state of each node relative to other nodes is determined based on the node characteristics corresponding to each node and the corresponding neighbor node characteristics.

[0104] For example, the intermediate network state of each node relative to its neighboring nodes can be calculated by first using the node features corresponding to each node and the features of its neighboring nodes. Then, an OR operation can be performed between the intermediate network state and the features corresponding to the destination transmission node to be transmitted, to obtain the network state of each node relative to other nodes.

[0105] In some embodiments, the node features corresponding to each node and the corresponding neighbor node features can be input into a graph attention network, which includes at least one graph attention layer, so as to output the target node features corresponding to each node, thereby reflecting the network congestion situation from each node in the network to the next hop node through the target node features.

[0106] Specifically, according to the node characteristics corresponding to each node and the corresponding neighbor node characteristics, the network state of each node relative to other nodes is determined, including:

[0107] (1.1) Determine the attention score between the node feature and each neighbor node feature;

[0108] (1.2) Determine the target node features corresponding to each node based on the attention score;

[0109] (1.3) Determine the network status of each node relative to other nodes based on the characteristics of the target node.

[0110] Among them, the graph attention network has an attention mechanism. In order to describe the attention mechanism, the attention mechanism can be imagined as an attention module, in which the attention module corresponds to three channels: Q, K, and V, namely query, key, and value. The Q channel is used for the node features of the current node, the K channel is used to input the neighbor node features of the neighbor nodes of the current node, and the V channel is used to input the value vector corresponding to each neighbor node. The final output of the attention module is the target node feature corresponding to each node.

[0111] Among them, it is necessary to determine the attention score between the node feature and each neighbor node feature. The specific calculation method is as follows:

[0112] .

[0113] Where i is the current node, j is any neighbor node of the current node, exp is the exponential function, τ is the scaling factor, and LeakyReLU is the activation function. is the node feature of the current node, is the neighbor node feature of the neighbor node, is the weight matrix that transforms node features into query vectors, Weight matrix used to transform neighbor node features into key vectors.

[0114] The attention score between the current node and each neighbor node is obtained by this calculation method, which can be a vector. The attention score can be used to measure the relationship between the current node and each neighbor node, such as the connection relationship between the current node and the neighbor node, and also represents the congestion between the current node and the neighbor node. The attention score can be used to measure which neighbor node's queue length has a greater impact on the current node.

[0115] After obtaining the attention score, the target node feature corresponding to the current node can be determined based on the attention score.

[0116] In some implementations, determining the target node feature corresponding to each node according to the attention score includes:

[0117] (1.2.1) Determine the value vector weight corresponding to each neighbor node;

[0118] (1.2.2) Multiply the value vector weight, attention score and neighbor node features corresponding to each neighbor node to obtain the target result, and sum the target results to obtain the target node features corresponding to each node.

[0119] For example, among multiple neighbor nodes of the current node, the value vector weight corresponding to each neighbor node, the attention score and the neighbor node feature can be determined to multiply and obtain the target result, and then all the target results are added together to obtain the target node feature corresponding to the current node.

[0120] The specific calculation method is as follows:

[0121] .

[0122] Among them, the graph attention network contains at least one graph attention layer. is the output feature vector after the first graph attention layer. It is the value vector weight corresponding to each neighbor node, which can be a weight matrix, which transforms the state features corresponding to neighbor node j into a value vector and captures its basic features. is the activation function, is the attention score of neighbor node j to node i, indicating the importance of neighbor node j to the current node.

[0123] The target node feature may be a feature obtained by aggregating the network features of neighboring nodes of the current node, and may reflect the network status of the current node, such as congestion.

[0124] Please combine Figure 4 , Figure 4 It is a schematic diagram of obtaining target node features of each node provided in an embodiment of the present application.

[0125] Among them, Y1 is the node feature of the current node, Y2, Y3, Y4, and Y5 are the neighbor node features of the neighbor nodes of the current node respectively. After the node feature and the neighbor node feature are input into the attention module, the attention module can perform calculations and processing in the above manner to output the target node feature Z1 corresponding to the current node.

[0126] In some implementations, after obtaining the target node features, the target node features can be further processed, for example, they can be input into the next graph attention layer for processing to obtain updated target node features. This operation can achieve the role of aggregating second-order neighbor nodes and expand the local perception range of the node. Finally, a feature set composed of updated target node features can be obtained, for example, it can be expressed as: .in is the updated target node feature corresponding to each node, is the dimensional space corresponding to the updated target node features. The updated target node features are more conducive to subsequent applications, such as training decision models, because the updated target node features can extract and retain important information about the nodes and their neighboring nodes in the network, allowing the model to learn and predict more effectively.

[0127] In some implementations, determining the network state of each node relative to other nodes based on the target node characteristics includes:

[0128] (1.3.1) Based on the connection between each node and other nodes in the network, the target node features of each node are uniquely encoded to generate node structure features;

[0129] (1.3.2) Perform an OR operation on the node structure characteristics of each node and the node structure characteristics of the corresponding neighboring node to obtain an OR operation result;

[0130] (1.3.3) Determine the destination transmission node corresponding to each node and the node structure characteristics of the destination transmission node, perform an OR operation on the OR operation result and the node structure characteristics of the destination transmission node, and obtain the network status of each node relative to other nodes.

[0131] Each target node feature can be uniquely encoded according to the connection between each node and other nodes in the network, thereby converting it into a binary node structure feature, which is more conducive to subsequent data processing.

[0132] The updated target node features can also be used as target node features, and then the target node features of each node are uniquely encoded according to the connection between each node and other nodes in the network to generate node structure features.

[0133] Then, the node structure features of each node that aggregates the unique-hot encoding of neighboring nodes and the node structure features of the destination transmission node that ultimately needs to be transmitted are ORed to obtain the network status of each node relative to other nodes. In other words, each corresponding bit between the node structure features can be ORed.

[0134] The OR operation can be specifically expressed as:

[0135] .

[0136] Among them, if S1 and S2 can be expressed as two different node structure features, i is the i-th position corresponding to the node structure feature. When S1 is the result of OR operation of the node structure characteristics of each node and the node structure characteristics of the corresponding neighboring node, S2 is the node structure characteristics of the destination transmission node that needs to be transmitted. is the final OR operation result, which can reflect the network status of each node relative to other nodes.

[0137] Through OR operations, a method that can directly calculate and analyze the geometric relationship between nodes can construct a logical topology view without a deep neural network structure, and ultimately reflect the connection status between nodes through the logical topology view. For example, each activation bit will be activated in the operation result to encode the relative position between nodes, and at the same time collect which nodes are the next hop reachable nodes of the current node.

[0138] Therefore, the node structure characteristics of each node are ORed with the node structure characteristics of other nodes to obtain the network status of each node relative to other nodes, such as the nodes to which data can be transmitted directly or indirectly, etc. The network status can be represented by corresponding values ​​or vectors.

[0139] It can be seen from step 201 to step 203 that by aggregating the node features of the current node and the neighbor node features of the neighbor nodes, the network state corresponding to the current node can be more accurately reflected, which is helpful to improve the training accuracy of the routing decision model in the future.

[0140] It should be noted that the above method can obtain the network state corresponding to the current node at each moment in the dynamic network and the network action corresponding to the current node at each moment, and through the interaction between the current node and the dynamic network, the reward value corresponding to the current node after executing the network action at each moment can be obtained. Finally, the network state, network action, and reward value data are saved as training data in the training data set. In the subsequent training process of the routing decision model, the training data in the training data set needs to be used.

[0141] Please also read Figure 5 , Figure 5 It is a schematic diagram of a routing decision model provided in an embodiment of the present application.

[0142] Regarding the routing decision model provided in the present application, the routing decision model includes an evaluation sub-model and a target sub-model, wherein the evaluation sub-model and the target sub-model have the same network structure, but the evaluation sub-model and the target sub-model correspond to different model parameters. Among them, after each round of training iteration of the evaluation sub-model, the model parameters need to be updated, and after the evaluation sub-model is updated and iterated to a certain preset number of iterations, the target sub-model migrates the latest model parameters of the evaluation sub-model to the target sub-model.

[0143] The evaluation sub-model is used to output the evaluation value, that is, the predicted value, and the target sub-model is used to output the target value, that is, the label value. The model loss is calculated by the evaluation value and the target value, and then the model parameters of the evaluation sub-model are adjusted according to the model loss. The evaluation sub-model is trained until the model loss meets the preset loss condition. The training of the evaluation sub-model is completed, indicating that the training of the routing decision model is completed. The routing decision model can be used to make routing decisions subsequently. Specifically, predictions are made through the trained evaluation sub-model, and the corresponding routing decisions are determined according to the prediction results.

[0144] The routing decision model can be set in each node in the dynamic network. Each node can act as an intelligent agent and can accurately determine the most reliable next hop node for data transmission in the dynamic network through the locally stored trained routing decision model to achieve data transmission. The routing decision model can be a model improved based on the deep Q network model.

[0145] Steps 110 to 150 will be described in detail below.

[0146] In step 110, a first node is determined from among a plurality of nodes in a dynamic network, and a first network state and a first network action corresponding to the first node are determined.

[0147] Among them, the first node can be the current node, and the first network state and the first network action corresponding to the first node at the target time can be obtained from the training data set. The first network state can be understood as the network state of the first node relative to other nodes, such as the network congestion state corresponding to the first node. The first network action can be understood as the action of the current node transmitting data to the next hop node. If the current node has 10 next hop nodes, then it has 10 first network actions.

[0148] The first network state can be expressed as , the first network action can be expressed as , t is the target time, which can be understood as the time when the first node needs to transmit the next-hop data. The first network action corresponds to a set, such as ,in is the total number of first network actions.

[0149] The first network state and the first network action are a batch of data. For example, the first network state has , each first network state corresponds to a corresponding first network action.

[0150] In step 120, the first network state and the first network action are input into the evaluation sub-model, and the evaluation value corresponding to each first network action is output.

[0151] The first network state and the first network action can be input into the evaluation sub-model, and the evaluation sub-model can output the evaluation value corresponding to each first network action according to the first network state and the first network action. With the corresponding evaluation value, the first network action Have corresponding evaluation values.

[0152] For example, the evaluation sub-model corresponds to a strategy After the first network state and the first network action are input, in the strategy The corresponding evaluation value is output below.

[0153] See also Figure 6 , Figure 6 1 is a flow chart of step 120 provided in an embodiment of the present application. In some embodiments, the evaluation sub-model includes a first neural network and a second neural network, the first network state and the first network action are input into the evaluation sub-model, and the evaluation value corresponding to each first network action is output, including:

[0154] Step 301: input the first network state into the first neural network, and output the state value corresponding to each first network action;

[0155] Step 302: input the first network state and the first network action into the second neural network, and output the advantage value corresponding to each first network action;

[0156] Step 303: Determine an evaluation value corresponding to each first network action according to the state value and the advantage value.

[0157] Steps 301 to 303 are described in detail below.

[0158] In step 301, a first network state and a first network action are input into a first neural network, and a state value corresponding to each first network action is output.

[0159] Please combine Figure 7 , Figure 7 1 is a schematic diagram of the structure of the evaluation sub-model provided in the embodiment of the present application. The evaluation sub-model includes a first neural network and a second neural network. The first neural network and the second neural network can be linear neural networks.

[0160] The first network state and the first network action can be input into the first neural network, and the state value corresponding to each first network action can be output. The state value can be understood as the result output by the state value function. The state value represents the expected return that the agent can obtain by following a certain strategy from a given state, and is used to calculate the value corresponding to the first network state.

[0161] That is to say, the state value can be understood as a characteristic representation of the current first network state.

[0162] In step 302, the first network state and the first network action are input into the second neural network, and the advantage value corresponding to each first network action is output.

[0163] The first network state and the first network action can be input into the second neural network, and the advantage value corresponding to each first network action can be output, and the advantage value is used to measure the value difference of taking a certain first network action relative to the average network action of all first network actions under a given first network state. In other words, the advantage value indicates how better a certain first network action is than the average network action. The advantage value is used to calculate the advantage of each first network action relative to the state value.

[0164] In other words, the advantage value can distinguish which first network actions are more valuable, thereby guiding the agent to learn a better strategy. In this way, it is possible to allow the first network actions of the first node not to be all in one action space, and the advantage value can still be used to reflect which first network actions are more valuable. In a specific application scenario, for example, when the first node transmits data for the last time, the action space of its corresponding first network action is R1, which includes some first network actions. As the position of the node in the dynamic network changes, the next time the first node transmits data, the action space of its corresponding first network action is R2, in which the first network action has changed, that is, the node that can be transmitted to the next hop has changed, but these do not affect the agent's learning of a better strategy, that is, the first node can still learn the value of each first network action through the first network actions in different action spaces, which is conducive to updating the strategy corresponding to the routing decision model.

[0165] In step 303, an evaluation value corresponding to each first network action is determined according to the state value and the advantage value.

[0166] The state value and the advantage value jointly determine the evaluation value corresponding to each first network action, that is, the evaluation value of the first network action is determined by the output results of different parts of the neural network, which can avoid the problem of overestimation of the evaluation value and make the output evaluation value more accurate.

[0167] In some implementations, determining the evaluation value corresponding to each first network action according to the state value and the advantage value includes:

[0168] (1.1) Determine the number of all first network actions, add up all advantage values ​​and divide by the number to get the average advantage value;

[0169] (1.2) Subtract the average advantage value from each advantage value to obtain the target advantage value corresponding to each first network action;

[0170] (1.3) Add the target advantage value to the state value to obtain the evaluation value corresponding to each first network action.

[0171] Among them, by normalizing the evaluation values ​​corresponding to different first network actions, the calculation of the evaluation values ​​can be made more accurate.

[0172] The specific calculation method is as follows:

[0173] .

[0174] Where Q is the Q value function, V is the state value function, A is the advantage value function, s is the first network state, a is the first network action, which can be a first network action in the first network action space (set), To evaluate the model parameters of the sub-model, are the model parameters of the first neural network, are the model parameters of the second neural network model, is a first network action among all first network actions, is the number of all first network actions. The average advantage is , the target advantage value is , is the status value, is the evaluation value.

[0175] It can be seen from step 301 to step 303 that for the evaluation values ​​corresponding to the first network state and the first network action, by respectively calculating the corresponding advantage value and state value, the evaluation value jointly determined by the advantage value and the state value is more accurate, avoiding the problem of overestimation of the evaluation value. At the same time, by introducing the advantage value, the first network actions in different action spaces can be used to help the evaluation sub-model to update the corresponding parameters. The evaluation sub-model can flexibly handle the situation where the positions of the surrounding nodes change, that is, if the action changes, its advantage value is recalculated, and then the corresponding action evaluation value is obtained, without the need to train the model, thereby improving the efficiency of the entire routing decision model in a changing environment.

[0176] In step 130, the target first network action corresponding to the maximum evaluation value and the second node corresponding to the target first network action are determined.

[0177] Among them, the target first network action corresponding to the largest evaluation value can be selected from multiple first network actions, that is, the first node can transmit data to the second node corresponding to the target first network action through the target first network action. This can improve the model training efficiency of the routing decision model and avoid learning the first network action with a lower evaluation value.

[0178] In step 140, the second network action corresponding to the second node, the second network state and the reward value corresponding to the target first network action are input into the target sub-model, and the target value corresponding to the target first network action is output.

[0179] It can be understood that since the training of the routing decision model is based on existing data, after determining the target first network action, the second node corresponding to the execution of the target first network action, that is, the next hop node of the first node, and the second network action and the second network state corresponding to the second node can be determined through the existing data (training data set). The second network action is an action of the second node that can execute the next hop data transmission. The method for determining the second network state is the same as the method for determining the network state described above.

[0180] After executing the target first network action, a reward value corresponding to executing the target first network action may be obtained. The reward value may be understood as a reward value generated based on a result after executing the target first network action under a reward standard.

[0181] Among them, for the calculation based on the target value, the Bellman equation can be used for calculation, and the specific calculation method is as follows:

[0182] .

[0183] in, is the target value, r is the reward value, is the second network state, For the second network action, are the model parameters in the target sub-model, The discount factor can be preset, usually between 0 and 1, and is used to reduce the weight of future rewards.

[0184] That is to say, the reward value is a real value, which is the value generated after executing the first network action of the target, and the evaluation value is determined based on the reward value. Therefore, the evaluation value has a more real reference value, and the evaluation value can be determined as the label of the routing decision model, that is, the label corresponding to the evaluation value output by the evaluation sub-model.

[0185] It should be noted that the target sub-model and the evaluation sub-model have the same network structure, but the model parameters are different.

[0186] In step 150, the model loss corresponding to the evaluation sub-model is determined according to the target value and the maximum evaluation value, and the model parameters corresponding to the evaluation sub-model are adjusted according to the model loss. The execution is returned to determine the first node among multiple nodes in the dynamic network until the model loss meets the preset loss condition, thereby obtaining a trained routing decision model.

[0187] The model loss corresponding to the evaluation sub-model can be determined according to the target value and the maximum evaluation value, and the model loss is used to guide the training direction of the routing decision model, specifically the model training direction of the evaluation sub-model, so as to adjust the model parameters of the routing decision model, such as the model parameters of the evaluation sub-model.

[0188] In some implementations, determining the model loss corresponding to the evaluation sub-model according to the target value and the maximum evaluation value includes:

[0189] (1.1) Subtract the maximum evaluation value from the target value to get the difference;

[0190] (1.2) The model loss corresponding to the evaluation sub-model is determined based on the square value of the difference.

[0191] The specific calculation method is as follows:

[0192] .

[0193] in, is the model loss, is the target value, is the evaluation value.

[0194] If the routing decision model is trained with a batch of training data, the model loss can be calculated by combining the loss corresponding to each training data with the mathematical expectation method. The specific calculation method is as follows:

[0195] .in, is the mathematical expectation.

[0196] In some implementations, after adjusting the model parameters corresponding to the evaluation sub-model according to the model loss, the method further includes:

[0197] (1.1) Determine the number of iterations corresponding to the evaluation sub-model;

[0198] (1.2) When the number of iterations reaches the preset number of iterations, the model parameters of the evaluation sub-model are obtained, and the model parameters are updated to the target sub-model.

[0199] The evaluation sub-model is iteratively updated each time it is trained, that is, its model parameters are updated. If the number of iterations reaches the preset number of iterations, the model parameters of the evaluation sub-model are obtained and updated to the target sub-model, so that the model parameters of the target sub-model are also updated.

[0200] In the subsequent process of training the routing decision model, the target sub-model will determine the target value based on the new model parameters, thereby continuing to guide the iterative training of the evaluation sub-model, so that the entire routing decision model is continuously iterated and adapted to the dynamic network.

[0201] The trained routing decision model can accurately determine the next hop node with the most reliable data transmission to achieve data transmission.

[0202] In some implementations, after determining the model loss corresponding to the evaluation sub-model according to the target value and the maximum evaluation value, the method further includes:

[0203] (2.1) Obtaining the experience data corresponding to the evaluation sub-model after each training, the experience data including the first network state, the target first network action, the reward value and the second network state;

[0204] (2.2) The empirical data is saved in a training data set, and the training data in the training data set is used to train the routing decision model.

[0205] Among them, the first network state, the target first network action, the reward value, and the second network state are data generated after the first node interacts with the dynamic network, such as vectors or values. Experience data, such as these vectors or values, can be saved in a training data set, and the training data in the training data set is used to train the routing decision model. This can avoid the waste of previous interaction experience data, and the routing decision model can be trained again later. These training data not only reflect the historical changes in node and network states, but also record the effects of interactions between different nodes. The trained routing decision model can absorb historical experience and accurately determine the most reliable next-hop node for data transmission to achieve data transmission.

[0206] In some implementations, when the model loss satisfies a preset loss condition, a trained routing decision model is obtained. For example, when the model loss is less than a preset loss value, and the model loss does not change within a few time steps, it is determined that the routing decision model training is completed, and the trained routing decision model is obtained. Alternatively, when the number of times the routing decision model is trained reaches a certain number, it is considered that the routing decision model training is completed, and the trained routing decision model is obtained.

[0207] As can be seen from the above, in an embodiment of the present application, the routing decision model includes an evaluation sub-model and a target sub-model, by determining a first node from multiple nodes in a dynamic network, and determining a first network state and a first network action corresponding to the first node; inputting the first network state and the first network action into the evaluation sub-model, and outputting an evaluation value corresponding to each first network action; determining a target first network action corresponding to the maximum evaluation value and a second node corresponding to the target first network action; inputting the reward value corresponding to the second network action, the second network state and the target first network action corresponding to the second node into the target sub-model, and outputting a target value corresponding to the target first network action; determining a model loss corresponding to the evaluation sub-model according to the target value and the maximum evaluation value, and adjusting the model parameters corresponding to the evaluation sub-model according to the model loss, returning to execute to determine a first node from multiple nodes in the dynamic network until the model loss meets the preset loss condition, and obtaining a trained routing decision model. In this way, the target first network action with the largest evaluation value is determined among all executable first network actions of the first node through the evaluation sub-model, and the target value corresponding to the target first network action is determined through the target sub-model. The target value is determined based on the reward value obtained after the execution of the target first network action. The reward value is a real value, so the target value is more authentic than the evaluation value. The model loss calculated by taking the target value as a label and the evaluation value is more accurate, so that the model parameters of the evaluation sub-model can be adjusted more accurately, so that the evaluation sub-model can be continuously iterated according to the network state changes of the nodes in the dynamic network to adapt to the dynamic network until the trained routing decision model is obtained. Compared with the related art that cannot determine the optimal transmission path of data in the dynamic network, the present application can accurately determine the most reliable next hop node for data transmission in the dynamic network through the trained routing decision model to achieve data transmission.

[0208] See also Figure 8 , Figure 8 1 is another flow chart of the routing decision model training method provided by the embodiment of the present application. The routing decision model training method may include the following steps:

[0209] Step 401: Obtain node features corresponding to each node in the dynamic network;

[0210] Step 402: Determine the neighbor nodes corresponding to each node, and the neighbor node features corresponding to the neighbor nodes;

[0211] Step 403: Determine the network status of each node relative to other nodes according to the node characteristics corresponding to each node and the corresponding neighbor node characteristics;

[0212] Step 404: determine a first node from among multiple nodes in the dynamic network, and determine a first network state and a first network action corresponding to the first node;

[0213] Step 405: input the first network state into the first neural network, and output the state value corresponding to each first network action;

[0214] Step 406: input the first network state and the first network action into the second neural network, and output the advantage value corresponding to each first network action;

[0215] Step 407: Determine an evaluation value corresponding to each first network action according to the state value and the advantage value;

[0216] Step 408: Determine the target first network action corresponding to the maximum evaluation value and the second node corresponding to the target first network action;

[0217] Step 409: input the second network action corresponding to the second node, the second network state and the reward value corresponding to the target first network action into the target sub-model, and output the target value corresponding to the target first network action;

[0218] Step 410, determine the model loss corresponding to the evaluation sub-model according to the target value and the maximum evaluation value, and adjust the model parameters corresponding to the evaluation sub-model according to the model loss, return to execute to determine the first node among multiple nodes in the dynamic network until the model loss meets the preset loss condition, and obtain the trained routing decision model.

[0219] In the above embodiments, the description of each embodiment has its own focus. For the part that is not described in detail in a certain embodiment, please refer to the detailed description of the routing decision model training method above, which will not be repeated here.

[0220] See 9, Fig. 9 1 is a schematic diagram of the structure of a routing decision model training device provided in an embodiment of the present application. The routing decision model training device is used to execute the routing decision model training method described above.

[0221] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories), or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0222] The routing decision model training device 500 may include:

[0223] A first determination module 510, configured to determine a first node from among a plurality of nodes in a dynamic network, and determine a first network state and a first network action corresponding to the first node;

[0224] A first input module 520, configured to input a first network state and a first network action into an evaluation sub-model, and output an evaluation value corresponding to each first network action;

[0225] A second determination module 530, configured to determine a target first network action corresponding to the maximum evaluation value and a second node corresponding to the target first network action;

[0226] A second input module 540 is used to input the second network action corresponding to the second node, the second network state and the reward value corresponding to the target first network action into the target sub-model, and output the target value corresponding to the target first network action;

[0227] The loss module 550 is used to determine the model loss corresponding to the evaluation sub-model according to the target value and the maximum evaluation value, and adjust the model parameters corresponding to the evaluation sub-model according to the model loss, and return to execute to determine the first node among multiple nodes in the dynamic network until the model loss meets the preset loss condition to obtain the trained routing decision model.

[0228] In some embodiments, the first input module 520 includes a first sub-input module, a second sub-input module, and a first determination sub-module;

[0229] A first sub-input module, used to input the first network state into the first neural network, and output a state value corresponding to each first network action;

[0230] A second sub-input module, used for inputting the first network state and the first network action into the second neural network, and outputting an advantage value corresponding to each first network action;

[0231] The first determination submodule is used to determine the evaluation value corresponding to each first network action according to the state value and the advantage value.

[0232] In some embodiments, the determination submodule is used to determine the number of all first network actions, and to add up all advantage values ​​and divide by the number to obtain an average advantage value;

[0233] Subtract the average advantage value from each advantage value to obtain the target advantage value corresponding to each first network action;

[0234] The target advantage value is added to the state value to obtain the evaluation value corresponding to each first network action.

[0235] In some embodiments, the loss module 550 is used to obtain a difference by subtracting the maximum evaluation value from the target value;

[0236] The model loss corresponding to the evaluation sub-model is determined based on the square value of the difference.

[0237] In some embodiments, the routing decision model training device further includes a feature acquisition module, a third determination module, and a fourth determination module;

[0238] Before determining the first node among the multiple nodes in the dynamic network, a feature acquisition module is used to acquire node features corresponding to each node in the dynamic network;

[0239] A third determination module is used to determine the neighbor nodes corresponding to each node, and the neighbor node features corresponding to the neighbor nodes;

[0240] The fourth determination module is used to determine the network status of each node relative to other nodes according to the node characteristics corresponding to each node and the corresponding neighbor node characteristics.

[0241] In some embodiments, the fourth determination module includes an attention submodule, a second determination submodule, and a third determination submodule;

[0242] The attention submodule is used to determine the attention score between the node feature and each neighbor node feature;

[0243] The second determination submodule is used to determine the target node feature corresponding to each node according to the attention score;

[0244] The third determination submodule is used to determine the network status of each node relative to other nodes according to the target node characteristics.

[0245] In some implementations, the second determination submodule is used to determine the value vector weight corresponding to each neighbor node;

[0246] Multiply the value vector weight, attention score and neighbor node features corresponding to each neighbor node to obtain the target result, and sum the target results to obtain the target node features corresponding to each node.

[0247] In some embodiments, the third determination submodule is used to perform one-hot encoding on the target node feature of each node according to the connection status of each node with other nodes in the network to generate a node structure feature;

[0248] Perform an OR operation on the node structure feature of each node and the node structure feature of the corresponding neighboring node to obtain an OR operation result;

[0249] The destination transmission node corresponding to each node and the node structure characteristics of the destination transmission node are determined, and the OR operation result and the node structure characteristics of the destination transmission node are ORed to obtain the network status of each node relative to other nodes.

[0250] In some embodiments, the loss module 550 is further configured to determine the number of iterations corresponding to the evaluation sub-model after adjusting the model parameters corresponding to the evaluation sub-model according to the model loss;

[0251] When the number of iterations reaches the preset number of iterations, the model parameters of the evaluation sub-model are obtained, and the model parameters are updated to the target sub-model.

[0252] In some embodiments, the loss module 550 is further configured to obtain experience data corresponding to the evaluation sub-model after each training after determining the model loss corresponding to the evaluation sub-model according to the target value and the maximum evaluation value, where the experience data includes the first network state, the target first network action, the reward value, and the second network state;

[0253] The experience data is saved in a training data set, and the training data in the training data set is used to train the routing decision model.

[0254] In an embodiment of the present application, the routing decision model includes an evaluation sub-model and a target sub-model. The first determination module 510 determines the first node among multiple nodes of the dynamic network, and determines the first network state and the first network action corresponding to the first node; the first input module 520 inputs the first network state and the first network action into the evaluation sub-model, and outputs the evaluation value corresponding to each first network action; the second determination module 530 determines the target first network action corresponding to the maximum evaluation value and the second node corresponding to the target first network action; the second input module 540 inputs the second network action corresponding to the second node, the second network state and the reward value corresponding to the target first network action into the target sub-model, and outputs the target value corresponding to the target first network action; the loss module 550 determines the model loss corresponding to the evaluation sub-model according to the target value and the maximum evaluation value, and adjusts the model parameters corresponding to the evaluation sub-model according to the model loss, and returns to execute to determine the first node among multiple nodes of the dynamic network until the model loss meets the preset loss condition, and obtains the trained routing decision model. In this way, the target first network action with the largest evaluation value is determined among all executable first network actions of the first node through the evaluation sub-model, and the target value corresponding to the target first network action is determined through the target sub-model. The target value is determined based on the reward value obtained after the execution of the target first network action. The reward value is a real value, so the target value is more authentic than the evaluation value. The model loss calculated by taking the target value as a label and the evaluation value is more accurate, so that the model parameters of the evaluation sub-model can be adjusted more accurately, so that the evaluation sub-model can be continuously iterated according to the network state changes of the nodes in the dynamic network to adapt to the dynamic network until the trained routing decision model is obtained. Compared with the related art that cannot determine the optimal transmission path of data in the dynamic network, the present application can accurately determine the most reliable next hop node for data transmission in the dynamic network through the trained routing decision model to achieve data transmission.

[0255] In the above embodiments, the description of each embodiment has its own focus. For the part that is not described in detail in a certain embodiment, please refer to the detailed description of the routing decision model training method above, which will not be repeated here.

[0256] The present application also provides a computer device, which may be a terminal. Fig.10 As shown, it shows a schematic diagram of the structure of the terminal involved in the embodiment of the present application, specifically:

[0257] The computer device may include a radio frequency (RF) circuit 601, a memory 602 including one or more computer-readable storage media, an input unit 603, a display unit 604, a sensor 605, an audio circuit 606, a wireless fidelity (WiFi) module 607, a processor 608 including one or more processing cores, and a power supply 609. Those skilled in the art will appreciate that Fig.10 The terminal structure shown in the figure does not constitute a limitation on the terminal, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0258] The RF circuit 601 can be used for receiving and sending signals during the process of sending and receiving information or making calls. In particular, after receiving the downlink information of the base station, it is handed over to one or more processors 608 for processing; in addition, the data related to the uplink is sent to the base station. Generally, the RF circuit 601 includes but is not limited to an antenna, at least one amplifier, a tuner, one or more oscillators, a Subscriber Identity Module (SIM) card, a transceiver, a coupler, a low noise amplifier (LNA), a duplexer, etc. In addition, the RF circuit 601 can also communicate with the network and other devices through wireless communication. Wireless communication can use any communication standard or protocol, including but not limited to Global System of Mobile Communication (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), email, Short Messaging Service (SMS), etc.

[0259] The memory 602 can be used to store software programs and modules. The processor 608 executes various functional applications and information retrieval by running the software programs and modules stored in the memory 602. The memory 602 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the terminal (such as audio data, a phone book, etc.), etc. In addition, the memory 602 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 602 may also include a memory controller to provide the processor 608 and the input unit 603 with access to the memory 602.

[0260] The input unit 603 can be used to receive input digital or character information, and generate keyboard, mouse, joystick, optical or trackball signal input related to object setting and function control. Specifically, in a specific embodiment, the input unit 603 may include a touch-sensitive surface and other input devices. The touch-sensitive surface, also known as a touch display screen or a touch pad, can collect touch operations of an object on or near it (such as operations of an object using a finger, a stylus or any other suitable object or accessory on or near the touch-sensitive surface), and drive the corresponding connection device according to a pre-set program. Optionally, the touch-sensitive surface may include a touch detection device and a touch controller. Among them, the touch detection device detects the touch orientation of the object, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into the touch point coordinates, and then sends it to the processor 608, and can receive and execute commands sent by the processor 608. In addition, the touch-sensitive surface can be implemented using multiple types such as resistive, capacitive, infrared and surface acoustic waves. In addition to the touch-sensitive surface, the input unit 603 may also include other input devices. Specifically, other input devices may include, but are not limited to, one or more of a physical keyboard, function keys (such as a volume control key, a switch key, etc.), a trackball, a mouse, a joystick, and the like.

[0261] The display unit 604 can be used to display information input by the object or information provided to the object and various graphic object interfaces of the terminal, which can be composed of graphics, text, icons, videos and any combination thereof. The display unit 604 may include a display panel. Optionally, the display panel can be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), etc. Further, the touch-sensitive surface can cover the display panel. When the touch-sensitive surface detects a touch operation on or near it, it is transmitted to the processor 608 to determine the type of touch event, and then the processor 608 provides a corresponding visual output on the display panel according to the type of touch event.

[0262] The terminal may also include at least one sensor 605, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor may adjust the brightness of the display panel according to the brightness of the ambient light, and the proximity sensor may turn off the display panel and / or backlight when the terminal is moved to the ear. As a type of motion sensor, the gravity acceleration sensor can detect the magnitude of acceleration in each direction (generally three axes), and can detect the magnitude and direction of gravity when stationary. It can be used for applications that identify the posture of the mobile phone (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc.; As for other sensors that can be configured in the terminal, such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc., they will not be repeated here.

[0263] The audio circuit 606, the speaker, and the microphone can provide an audio interface between the object and the terminal. The audio circuit 606 can transmit the electrical signal converted from the received audio data to the speaker, which is converted into a sound signal for output; on the other hand, the microphone converts the collected sound signal into an electrical signal, which is received by the audio circuit 606 and converted into audio data, and then the audio data is processed by the output processor 608 and sent to, for example, another terminal through the RF circuit 601, or the audio data is output to the memory 602 for further processing. The audio circuit 606 may also include an earplug jack to provide communication between an external headset and the terminal.

[0264] WiFi is a short-range wireless transmission technology. The terminal can help the object to send and receive emails, browse web pages and access streaming media through the WiFi module 607. It provides wireless broadband Internet access for the object. Fig.10 A WiFi module 607 is shown, but it is understandable that it is not an essential component of the terminal and can be omitted as required without changing the essence of the invention.

[0265] The processor 608 is the control center of the terminal. It uses various interfaces and lines to connect various parts of the entire mobile phone. By running or executing software programs and / or modules stored in the memory 602 and calling data stored in the memory 602, it executes various functions of the terminal and processes data, thereby monitoring the mobile phone as a whole. Optionally, the processor 608 may include one or more processing cores; preferably, the processor 608 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, object interface and application programs, etc., and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 608.

[0266] The terminal also includes a power supply 609 (such as a battery) for supplying power to each component. Preferably, the power supply can be logically connected to the processor 608 through a power management system, so that the power management system can manage charging, discharging, and power consumption management. The power supply 609 can also include one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0267] Although not shown, the terminal may also include a camera, a Bluetooth module, etc., which will not be described in detail here. Specifically in this embodiment, the processor 608 in the terminal will load the executable files corresponding to the processes of one or more applications into the memory 602 according to the following instructions, and the processor 608 will run the applications stored in the memory 602 to implement various functions:

[0268] Determine a first node from among multiple nodes of the dynamic network, and determine a first network state and a first network action corresponding to the first node;

[0269] Inputting the first network state and the first network action into the evaluation sub-model, and outputting the evaluation value corresponding to each first network action;

[0270] Determine a target first network action corresponding to the maximum evaluation value and a second node corresponding to the target first network action;

[0271] Input the second network action corresponding to the second node, the second network state, and the reward value corresponding to the target first network action into the target sub-model, and output the target value corresponding to the target first network action;

[0272] The model loss corresponding to the evaluation sub-model is determined according to the target value and the maximum evaluation value, and the model parameters corresponding to the evaluation sub-model are adjusted according to the model loss. The execution is returned to determine the first node among multiple nodes in the dynamic network until the model loss meets the preset loss condition, thereby obtaining the trained routing decision model.

[0273] In the above embodiments, the description of each embodiment has its own focus. For the part that is not described in detail in a certain embodiment, please refer to the detailed description of the routing decision model training method above, which will not be repeated here.

[0274] As can be seen from the above, in an embodiment of the present application, the routing decision model includes an evaluation sub-model and a target sub-model, by determining a first node from multiple nodes in a dynamic network, and determining a first network state and a first network action corresponding to the first node; inputting the first network state and the first network action into the evaluation sub-model, and outputting an evaluation value corresponding to each first network action; determining a target first network action corresponding to the maximum evaluation value and a second node corresponding to the target first network action; inputting the reward value corresponding to the second network action, the second network state and the target first network action corresponding to the second node into the target sub-model, and outputting a target value corresponding to the target first network action; determining a model loss corresponding to the evaluation sub-model according to the target value and the maximum evaluation value, and adjusting the model parameters corresponding to the evaluation sub-model according to the model loss, returning to execute to determine a first node from multiple nodes in the dynamic network until the model loss meets the preset loss condition, and obtaining a trained routing decision model. In this way, the target first network action with the largest evaluation value is determined among all executable first network actions of the first node through the evaluation sub-model, and the target value corresponding to the target first network action is determined through the target sub-model. The target value is determined based on the reward value obtained after the execution of the target first network action. The reward value is a real value, so the target value is more authentic than the evaluation value. The model loss calculated by taking the target value as a label and the evaluation value is more accurate, so that the model parameters of the evaluation sub-model can be adjusted more accurately, so that the evaluation sub-model can be continuously iterated according to the network state changes of the nodes in the dynamic network to adapt to the dynamic network until the trained routing decision model is obtained. Compared with the related art that cannot determine the optimal transmission path of data in the dynamic network, the present application can accurately determine the most reliable next hop node for data transmission in the dynamic network through the trained routing decision model to achieve data transmission.

[0275] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0276] To this end, an embodiment of the present application provides a computer-readable storage medium, which stores multiple instructions, which can be loaded by a processor to execute the steps in any of the routing decision model training methods provided in the embodiments of the present application. For example, the instructions can execute the following steps:

[0277] Determine a first node from among multiple nodes of the dynamic network, and determine a first network state and a first network action corresponding to the first node;

[0278] Inputting the first network state and the first network action into the evaluation sub-model, and outputting the evaluation value corresponding to each first network action;

[0279] Determine a target first network action corresponding to the maximum evaluation value and a second node corresponding to the target first network action;

[0280] Input the second network action corresponding to the second node, the second network state, and the reward value corresponding to the target first network action into the target sub-model, and output the target value corresponding to the target first network action;

[0281] The model loss corresponding to the evaluation sub-model is determined according to the target value and the maximum evaluation value, and the model parameters corresponding to the evaluation sub-model are adjusted according to the model loss. The execution is returned to determine the first node among multiple nodes in the dynamic network until the model loss meets the preset loss condition, thereby obtaining the trained routing decision model.

[0282] According to one aspect of the present application, a computer program product or a computer program is provided, the computer program product or the computer program including computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the routing decision model training method provided in various optional implementations provided in the above embodiments.

[0283] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0284] The computer-readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0285] Since the instructions stored in the computer-readable storage medium can execute the steps in any one of the routing decision model training methods provided in the embodiments of the present application, the beneficial effects that can be achieved by any one of the routing decision model training methods provided in the embodiments of the present application can be achieved. For details, please refer to the previous embodiments and will not be repeated here.

[0286] The above is a detailed introduction to a routing decision model training method, device, computer equipment and storage medium provided in an embodiment of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A routing decision model training method, characterized in that: The routing decision model includes an evaluation sub-model and a target sub-model, and the method includes: Obtaining node characteristics corresponding to each node in a dynamic network, wherein each node is a network device and the dynamic network includes multiple nodes; Determine neighbor nodes corresponding to each of the nodes, and neighbor node features corresponding to the neighbor nodes; Determining an attention score between the node feature and each of the neighboring node features; Determine a target node feature corresponding to each of the nodes according to the attention score; Input the target node feature to the next graph attention layer for processing to obtain an updated target node feature; According to the connection status of each node with other nodes in the network, the updated target node feature of each node is one-hot encoded to generate a node structure feature; Performing an OR operation on the node structure feature of each node and the node structure feature of the corresponding neighboring node to obtain an OR operation result; Determine the destination transmission node corresponding to each of the nodes and the node structure characteristics of the destination transmission node, perform an OR operation on the OR operation result and the node structure characteristics of the destination transmission node, and obtain the network status of each of the nodes relative to other nodes; Determine a first node from among multiple nodes of a dynamic network, and determine a first network state and a first network action corresponding to the first node; Inputting the first network state and the first network action into the evaluation sub-model, and outputting an evaluation value corresponding to each of the first network actions; Determine a target first network action corresponding to the largest evaluation value and a second node corresponding to the target first network action; Inputting the second network action corresponding to the second node, the second network state and the reward value corresponding to the target first network action into the target sub-model, and outputting the target value corresponding to the target first network action; The model loss corresponding to the evaluation sub-model is determined according to the target value and the maximum evaluation value, and the model parameters corresponding to the evaluation sub-model are adjusted according to the model loss. The execution is returned to determine the first node among multiple nodes in the dynamic network until the model loss meets the preset loss condition, and the trained routing decision model is obtained. The trained routing decision model is used to determine the next hop node with the most reliable data transmission in the dynamic network.

2. The routing decision model training method according to claim 1, characterized in that: The evaluation sub-model includes a first neural network and a second neural network, and the first network state and the first network action are input into the evaluation sub-model, and the evaluation value corresponding to each first network action is output, including: Inputting the first network state into a first neural network, and outputting a state value corresponding to each of the first network actions; Inputting the first network state and the first network action into a second neural network, and outputting an advantage value corresponding to each of the first network actions; An evaluation value corresponding to each of the first network actions is determined according to the state value and the advantage value.

3. The routing decision model training method according to claim 2, characterized in that: The determining, according to the state value and the advantage value, an evaluation value corresponding to each of the first network actions includes: Determine the number of all the first network actions, and add up all the advantage values ​​and divide by the number to obtain an average advantage value; Subtract the average advantage value from each advantage value to obtain a target advantage value corresponding to each of the first network actions; The target advantage value is added to the state value to obtain an evaluation value corresponding to each of the first network actions.

4. The routing decision model training method according to claim 1, characterized in that: The determining the model loss corresponding to the evaluation sub-model according to the target value and the maximum evaluation value includes: Subtract the maximum evaluation value from the target value to obtain a difference; The model loss corresponding to the evaluation sub-model is determined according to the square value of the difference.

5. The routing decision model training method according to claim 1, characterized in that: Determining the target node feature corresponding to each of the nodes according to the attention score includes: Determine the value vector weight corresponding to each of the neighbor nodes; The value vector weight corresponding to each of the neighbor nodes, the attention score and the neighbor node feature are multiplied to obtain a target result, and the target results are summed to obtain a target node feature corresponding to each of the nodes.

6. The routing decision model training method according to claim 1, characterized in that: After adjusting the model parameters corresponding to the evaluation sub-model according to the model loss, the method further includes: Determine the number of iterations corresponding to the evaluation sub-model; When the number of iterations reaches a preset number of iterations, the model parameters of the evaluation sub-model are obtained, and the model parameters are updated to the target sub-model.

7. The routing decision model training method according to claim 1, characterized in that: After determining the model loss corresponding to the evaluation sub-model according to the target value and the maximum evaluation value, the method further includes: Acquire experience data corresponding to the evaluation sub-model after each training, the experience data including the first network state, the target first network action, the reward value and the second network state; The experience data is stored in a training data set, and the training data in the training data set is used to train the routing decision model.

8. A routing decision model training device, characterized in that: The routing decision model includes an evaluation sub-model and a target sub-model, and the device includes: A feature acquisition module, used to acquire node features corresponding to each node in a dynamic network, each node being a network device, and the dynamic network comprising a plurality of nodes; A third determination module is used to determine the neighbor nodes corresponding to each of the nodes, and the neighbor node features corresponding to the neighbor nodes; An attention submodule, configured to determine an attention score between the node feature and each of the neighboring node features; A second determination submodule is used to determine the target node feature corresponding to each of the nodes according to the attention score; input the target node feature to the next graph attention layer for processing to obtain an updated target node feature; A third determination submodule is used to perform one-hot encoding on the updated target node features of each node according to the connection status of each node with other nodes in the network to generate node structure features; Performing an OR operation on the node structure feature of each node and the node structure feature of the corresponding neighboring node to obtain an OR operation result; Determine the destination transmission node corresponding to each of the nodes and the node structure characteristics of the destination transmission node, perform an OR operation on the OR operation result and the node structure characteristics of the destination transmission node, and obtain the network status of each of the nodes relative to other nodes; A first determination module, configured to determine a first node from among a plurality of nodes in a dynamic network, and determine a first network state and a first network action corresponding to the first node; A first input module, configured to input the first network state and the first network action into the evaluation sub-model, and output an evaluation value corresponding to each of the first network actions; A second determination module, used to determine a target first network action corresponding to the maximum evaluation value and a second node corresponding to the target first network action; A second input module, configured to input a second network action corresponding to the second node, a second network state, and a reward value corresponding to the target first network action into the target sub-model, and output a target value corresponding to the target first network action; A loss module is used to determine the model loss corresponding to the evaluation sub-model according to the target value and the maximum evaluation value, and adjust the model parameters corresponding to the evaluation sub-model according to the model loss, and return to execute to determine the first node among multiple nodes in the dynamic network until the model loss meets the preset loss condition, so as to obtain a trained routing decision model, and the trained routing decision model is used to determine the next hop node with the most reliable data transmission in the dynamic network.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor to execute the routing decision model training method according to any one of claims 1 to 7.

10. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the routing decision model training method described in any one of claims 1 to 7 is implemented.

Images