Route selection method, system, device and equipment suitable for electric power communication service

Abstract

The invention relates to the technical field of electric power communication, in particular to a route selection method, system, device and equipment suitable for an electric power communication service, and the method comprises the steps: obtaining the node occupancy rate and the node bandwidth of each route node in the electric power communication service, and the communication delay between the route nodes; obtaining a routing node coordinate set; analyzing the similarity between a vector formed by the starting points of the analog links and a vector formed by the routing nodes to obtain channel similarity; analyzing the importance degree of each simulation link to obtain a random process weight; obtaining a channel learning rate; constructing an R table by combining distribution of node occupancy rates among the routing nodes according to a transmission condition among the routing nodes, replacing an original learning rate in a Q learning algorithm with the channel learning rate, and updating a Q table; and selecting the route at each moment according to the updated Q table at each moment and the channel similarity. The invention aims to make routing selection in the electric power communication service more reasonable and improve the communication quality.

Description

Technical Field

[0001] The present application relates to the field of power communication technology, and in particular to a routing selection method, system, device and equipment applicable to power communication services. Background Art

[0002] The power communication service is a key service used to transmit control information in the power grid system. It relies on the collaborative work with the power grid to build the industrial Internet and realize information transmission. This type of service has very high requirements for reliability, security and latency consistency to ensure that the power system can operate stably and efficiently. At present, the power communication network is mainly based on optical fiber communication. In this process, different routers need to be controlled as communication transfer stations to ensure the high reliability of the communication process.

[0003] When two routing nodes in the power system are communicating by routing, a secure and reliable router link needs to be selected for communication. Traditional methods usually aim to optimize the balance and efficiency of the entire communication network, and use the Q learning algorithm to determine the router link to complete data communication. This method uses the node utilization as the evaluation criterion and obtains the final path selection Q table by running the Q learning algorithm. During the generation of the Q table, the system randomly controls the router to simulate the formation of the link, and updates and generates the Q table by simulating the communication effect of the link.

[0004] In this process, the simulated link does not fully consider the communication direction between routers, resulting in a large difference between the simulated link generated by the Q learning algorithm and the actual link when it is executed. This gap reduces the convergence speed of Q learning, thereby slowing down the algorithm's response to changes in router network status. This situation may lead to unreasonable routing selection, which in turn affects the communication quality of power communication services. Summary of the invention

[0005] In view of the above, it is necessary to provide a routing selection method, system, device and equipment suitable for power communication business to solve the problem that when the Q learning algorithm is executed in routing selection, the reinforcement learning efficiency is low due to the large difference between the simulation data and the actual scheme habits, which makes the algorithm respond slowly to the changes in the routing network state, resulting in unreasonable routing selection results.

[0006] In a first aspect, the present application provides a routing selection method applicable to a power communication service, the method comprising: Obtain the node occupancy rate and node bandwidth of each routing node used for data transmission in the power communication service at each time, as well as the communication delay between routing nodes; A graph search algorithm is used to calculate the communication delay between routing nodes to obtain the shortest communication delay between any two routing nodes and obtain a distance matrix. The distance matrix is ​​reduced in dimension to obtain a set of routing node coordinates. In the routing node coordinate set, the vector formed by the starting point of each simulated link is used as the communication direction vector, and the similarity between the vector formed by each routing node and the routing node directly transmitted by it and the communication direction vector is combined with the Q learning algorithm to obtain the channel similarity of each routing node corresponding to the channel based on each simulated link; All routing nodes except for each simulated link are recorded as bandwidth distribution nodes of each simulated link, and the similarity between the vector from the starting point of each simulated link to each bandwidth distribution node and the communication direction vector is analyzed to obtain each communication distribution weight, and the random process weight of each simulated link at each moment is obtained by combining the node bandwidth of all bandwidth distribution nodes at each moment; Based on the Q learning algorithm, according to the channel similarity of the corresponding channels of each routing node selected by each simulated link, combined with the random process weight, the channel learning rate of each channel of each simulated link at each moment is obtained; whether direct transmission between routing nodes can be used as a judgment standard, combined with the distribution of node occupancy between routing nodes, an R table is constructed, the channel learning rate is used to replace the original learning rate in the Q learning algorithm, and the Q table is updated; The route at each moment is selected according to the Q table updated at each moment and the channel similarity.

[0007] The channel corresponding to each routing node is obtained based on the channel similarity of each simulated link, specifically: Get all routing nodes that directly transmit to each routing node, and record them as the communication nodes of each routing node; The vector from the routing node to each of its communication nodes is recorded as each communication vector; the normalized value of the cosine similarity between each communication vector and the communication direction vector corresponding to each simulated link is used as the rationality of the communication direction of each communication vector based on each simulated link; The communication direction rationality is used to replace the node selection probability in the Q learning algorithm; for each simulated link, the communication direction rationality corresponding to the channel selected by each routing node based on the Q learning algorithm is used as the channel similarity of each routing node based on each simulated link.

[0008] The communication distribution weights are obtained as follows: The vectors from the starting point of each simulated link to each bandwidth distribution node are recorded as each communication distribution vector; The normalized value of the cosine similarity between each communication distribution vector and the communication direction vector of each simulated link is used as each communication distribution weight.

[0009] The random process weight of each simulated link at each time is obtained as follows: For each simulated link, the node bandwidth proportion of each bandwidth distribution node at each time is obtained; based on each communication distribution weight, the corresponding node bandwidth proportion is weighted and summed to obtain the random process weight of each simulated link at each time.

[0010] The specific formula for obtaining the channel learning rate of each channel of each simulated link at each moment is: ;in, is the simulated link at time t The channel learning rate of the g-th channel, is the simulated link at time t The random process weights, It is an analog link The channel similarity of the g-th channel.

[0011] The construction of the R table is specifically as follows: If direct transmission is possible from routing node s to routing node a, the average of the node occupancy rates of the two routing nodes is calculated, and the difference between the first preset value and the average is used as the element of the sth row and ath column of the R table; if direct transmission is not possible, the second preset value is used as the element of the sth row and ath column of the R table; Among them, the first preset value is greater than or equal to 1; the second preset value is less than 0.

[0012] The selecting of the route at each moment according to the Q table updated at each moment and the channel similarity comprises: For any routing node at each moment, obtain the Q value with other routing nodes on the Q table, merge the negative correlation mapping results with the channel similarity of other routing nodes, and calculate the routing control judgment value; The routing node with the smallest routing judgment value with any of the routing nodes is selected as the next routing node for information transmission.

[0013] In a second aspect, the present application provides a routing selection system applicable to electric power communication services, the system comprising: A router status acquisition module is used to obtain the node occupancy rate and node bandwidth of each routing node used for data transmission in the power communication business at each moment, as well as the communication delay between routing nodes; A graph search algorithm is used to calculate the communication delay between routing nodes to obtain the shortest communication delay between any two routing nodes and obtain a distance matrix. The distance matrix is ​​reduced in dimension to obtain a set of routing node coordinates. A Q table generation module is used to use the vector formed by the starting points of each simulated link as the communication direction vector in the routing node coordinate set, and to obtain the channel similarity of each routing node corresponding to the channel based on each simulated link by combining the Q learning algorithm with the similarity between the vector formed by each routing node and the routing node directly transmitted to it and the communication direction vector; All routing nodes except for each simulated link are recorded as bandwidth distribution nodes of each simulated link, and the similarity between the vector from the starting point of each simulated link to each bandwidth distribution node and the communication direction vector is analyzed to obtain each communication distribution weight, and the random process weight of each simulated link at each moment is obtained by combining the node bandwidth of all bandwidth distribution nodes at each moment; Based on the Q learning algorithm, according to the channel similarity of the corresponding channels of each routing node selected by each simulated link, combined with the random process weight, the channel learning rate of each channel of each simulated link at each moment is obtained; whether direct transmission between routing nodes can be used as a judgment standard, combined with the distribution of node occupancy between routing nodes, an R table is constructed, the channel learning rate is used to replace the original learning rate in the Q learning algorithm, and the Q table is updated; The route selection module is used to select the route at each moment according to the Q table updated at each moment and the channel similarity.

[0014] In a third aspect, the present application provides a routing selection device suitable for power communication services, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor implements the steps of any one of the above methods when executing the computer program.

[0015] In a fourth aspect, the present application provides a routing selection device suitable for power communication services, wherein a computer program is stored in the device, and when the computer program is executed by a processor, the method described in any one of the above items is implemented.

[0016] In the above scheme, firstly, the node occupancy rate, node bandwidth and communication delay between routing nodes of each routing node used for data transmission in the power communication business are obtained at each moment; since two routing nodes cannot communicate directly, a graph search algorithm is used for the communication delay between routing nodes to obtain the shortest communication delay between any two routing nodes and obtain a distance matrix, which has the beneficial effect of characterizing the distance between routing nodes through the shortest communication delay; the distance matrix is ​​reduced in dimension to obtain a set of routing node coordinates, which is convenient for subsequent judgment of the directional difference between different channels and the final transmission target through the coordinates of the routing nodes, guiding the selection of the simulation operation scheme of the Q learning algorithm, and obtaining the channel similarity of each routing node corresponding to the channel based on each simulation link, which has the beneficial effect of improving the generation quality of simulation data, thereby improving the response speed of the Q learning algorithm to changes in the routing network state.

[0017] Furthermore, the random process weight is calculated through the node bandwidth and the routing node coordinate set to determine the similarity between the simulated data and the actual communication situation, and to screen out the simulated data that is close to the actual transmission process; the channel learning rate is calculated by combining the channel similarity and the random process weight. The higher the channel learning rate, the more it conforms to the channel control result of the actual communication link plan. When updating the Q table, the learning rate of the corresponding channel is increased, so that the Q learning algorithm can learn more quickly to conform to the actual communication link plan; finally, the route selection is performed through the Q table, and the simulated data is given a differentiated learning rate according to the actual router information transmission habits, which speeds up the Q learning algorithm's learning rate and response speed to changes in the routing network state, making the selected router communication link more reasonable and improving the communication quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 A flowchart of the steps of the routing selection method applicable to the power communication service provided in this application; Figure 2 A network diagram provided for this application; Figure 3 A block diagram of a routing system suitable for power communication services provided in this application. DETAILED DESCRIPTION

[0019] In the description of the embodiments of the present application, words such as "exemplary", "or", "for example" and the like are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary", "or", "for example" and the like is intended to present related concepts in a concrete manner.

[0020] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art in the present application. The terms used in the specification of the present application are only for the purpose of describing specific embodiments and are not intended to limit the present application.

[0021] It should also be noted that the terms "first" and "second" in this application and the accompanying drawings are used to distinguish similar objects, rather than to describe a specific order or sequence. The method disclosed in the embodiments of the present application or the method shown in the flowchart includes one or more steps for implementing the method. Without departing from the scope of the present application, the execution order of multiple steps can be interchanged with each other, and some steps can also be deleted.

[0022] See also Figure 1, which shows a flowchart of a route selection method applicable to a power communication service provided by an embodiment of the present application, the method comprising the following steps: Step 1: Obtain the node occupancy rate, node bandwidth, and communication delay between routing nodes of each routing node used for data transmission in the power communication service at each moment.

[0023] This application adopts the Q learning algorithm as the algorithm for routing control in the power communication business. In this process, it is necessary to obtain the status of the router network in real time, build a corresponding R table according to the router network status, and use the R table as input to generate a Q table using the Q learning algorithm, and select the corresponding communication link through the Q table. The routing network status information obtained in this embodiment includes: node occupancy rate, node bandwidth, and communication delay between nodes.

[0024] In this embodiment, there are a total of N routing nodes used for data transmission in the power communication service. The communication delay obtained between the routing nodes through the power communication system is recorded. The communication delay between routing node A and routing node B is recorded as In particular, two routers may be too far away to communicate directly, and the communication delay between the two routing nodes cannot be directly obtained.

[0025] For the nth routing node, the node occupancy rate is obtained at time t through the power communication system, which is recorded as the node occupancy rate , get the bandwidth of the nth routing node, recorded as node bandwidth .

[0026] Therefore, for N routing nodes in the routing network, they constitute a network composed of N routing nodes. The nth routing node in the network represents the nth router. Each routing node has its node occupancy rate. At the same time, there is a connection between nodes, which means that the two routing nodes can communicate with each other. The attribute value of the connection is the communication delay, which represents the time it takes for two routing nodes to communicate. The network diagram is as follows: Figure 2 As shown, Figure 2 It is a network composed of 6 routing nodes. A circle in the figure represents a routing node with attribute values. If the nodes can communicate directly, there will be a connection, and the attribute value of the connection is the communication delay.

[0027] Finally, we obtain the occupancy rates and node bandwidths of N routing nodes, as well as several communication delays. The number of communication delays is determined by the number of routers that can directly communicate with each other. It should be noted that the node occupancy rate and bandwidth will change with the state of the routing network, while the communication delay is the communication delay obtained when the routing network is operating normally and will not change over time.

[0028] Step 2: Use a graph search algorithm to calculate the communication delay between routing nodes, obtain the shortest communication delay between any two routing nodes, and obtain a distance matrix; perform dimensionality reduction on the distance matrix to obtain a set of routing node coordinates.

[0029] When the traditional Q learning algorithm is applied to the routing selection of the power communication service, two routing nodes are randomly selected from N routing nodes as the simulated communication tasks. In this embodiment, the node BG and the node ED are taken as examples to simulate the communication process from BG to ED.

[0030] In actual communication link schemes, a communication link scheme with a shorter total delay time is usually selected. Therefore, this embodiment obtains the shortest communication delay between the communication link scheme and two routing nodes, and compares the shortest communication delay with the communication delay of the simulated link to determine whether a communication link scheme conforms to the actual communication link scheme.

[0031] This application takes routing node A and routing node B as an example for analysis: First, if the two routing nodes A and B can communicate directly, their shortest communication delay is . If the two routing nodes A and B cannot communicate directly, the communication structure data set is used as input, and the Dijkstra algorithm is used for calculation. The parameters are A and B, the two connected routing nodes, and the output is the shortest communication delay between the two routing nodes. Finally, there is a shortest communication delay between any two routing nodes among the N routing nodes, and a total of H shortest communication delays are obtained. Among them, the Dijkstra algorithm is a graph search algorithm, which is a well-known technology, and this application will not repeat it.

[0032] It should be understood that for any simulated link, the closer its communication delay is to the shortest communication delay, the more it conforms to the actual communication link solution, and the simulated data generated by it can make the Q table converge faster, thereby improving the response speed of the Q learning algorithm to changes in the routing network state.

[0033] First, the manifold learning method is used to process the shortest communication delay, and the shortest communication delay and the corresponding routing node are mapped to the coordinate system, that is, H shortest communication delays form a distance matrix, as shown below: in, , Both represent the shortest communication delay between the first routing node and the second routing node; , Both represent the shortest communication delay between the Nth routing node and the 1st routing node; , Both represent the shortest communication delay between the Nth routing node and the second routing node.

[0034] It should be understood that the distance matrix is ​​a square matrix with a side length of N. The element in the nth row and the ith column of the distance matrix represents the shortest communication delay between the nth routing node and the ith routing node. The elements on the main axis of the matrix are the shortest communication delay between the node and itself, so the value is 0. In particular, the elements symmetrical along the main diagonal of the distance matrix are equal, for example = .

[0035] Furthermore, with the distance matrix as input, the Multiple Dimensional Scaling (MDS) algorithm is used to obtain a two-dimensional coordinate point set, which is recorded as the routing node coordinate set. The routing node coordinate set contains N points, where the nth point corresponds to the nth routing node. Among them, the MDS algorithm is a common technology in the field of data processing. It is a dimensionality reduction algorithm for manifold learning, which can map nodes to a two-dimensional coordinate system through node distance information.

[0036] Step 3: In the routing node coordinate set, the vector formed by the starting points of each simulated link is used as the communication direction vector. The similarity between the vector formed by each routing node and the routing node to which it directly transmits and the communication direction vector is combined with the Q learning algorithm to obtain the channel similarity of each routing node corresponding to the channel based on each simulated link.

[0037] This application aims to reduce the generated communication links that do not conform to reality by intervening in the scheme generation process of the simulated link, thereby improving the simulation data quality of Q learning and accelerating its convergence speed. Therefore, this application needs to evaluate the impact of different generation paths on the rationality of the final communication link during the simulation link generation process.

[0038] When using the shortest communication delay to determine whether a simulated link is consistent with an actual communication link, the evaluation can only be performed on the entire simulated link, and the performance of a specific channel cannot be evaluated separately. To this end, this application maps the shortest communication delay and the node to a coordinate system, and determines the rationality of a single channel by comparing the direction of a single channel with the direction of the simulated link. The specific process is as follows: This application uses routing node BG to routing node ED as a simulated link for analysis. In the routing node coordinate set, routing node BG is used as the starting point and routing node ED is used as the end point. The constructed vector is recorded as the communication direction vector.

[0039] When the Q learning algorithm randomly generates a simulated link between routing node BG and routing node ED, routing node BG is used as the first node with information, and then other nodes that can directly transmit with routing node BG are searched for. Among these nodes that can directly transmit, a routing node is randomly selected with the same probability as the second node with information; further, the third node with information is obtained through the second node with information in the same way, and so on, until the next routing node found is node ED.

[0040] In this application, a node C with information is taken as an example for analysis, and the routing node that can directly transmit with the node C with information is recorded as a communication node e. There are a total of E communication nodes that can directly transmit with the node C, so as to describe the process of generating a simulated link.

[0041] When the Q learning algorithm randomly selects a communication node from the E communication nodes of node C to form a simulated channel with node C, the probability of selecting a node from the E communication nodes is equal, but the E communication nodes are not all nodes that conform to communication habits; therefore, the method of selecting nodes with the same selection probability will cause the simulated link to not conform to the actual communication link scheme, resulting in a decrease in the quality of the generated simulated data. In this regard, the present application takes node C as the starting point and E communication nodes as the end points to construct E communication vectors; the cosine similarity of the e-th communication vector and the communication direction vector is calculated. The larger the cosine similarity, the more the single channel formed by node C and its e-th communication node conforms to the actual communication link scheme.

[0042] Furthermore, the E cosine similarities of node C are normalized. In this embodiment, the normalization method is as follows: first, 1 is added to each of the E cosine similarities to complete the non-negative processing; first, the cosine similarities after the non-negative processing are summed up, and then the cosine similarities after the non-negative processing are divided by the sum to obtain the cosine similarities after the normalization processing; the cosine similarities after the normalization processing are recorded as the rationality of the communication direction. In other embodiments, the maximum and minimum method can be used for normalization. It should be understood that the greater the rationality of the communication direction, the more the single channel formed by the node C and the e-th communication node conforms to the actual communication link scheme.

[0043] When the Q learning algorithm randomly selects a routing node from E communication nodes at node C to form a channel, the method described here is used to calculate the rationality of the communication direction. When selecting a node, the rationality of the e-th communication direction is used as the selection probability of the e-th communication node, replacing the same selection probability of E communication nodes in the original Q learning algorithm, and then the channel is selected to obtain a simulated link. Compared with the traditional simulation communication method, the simulation link construction scheme described in this embodiment can make the generated simulation link conform to the actual communication link scheme, improve the quality of simulation data, and improve the convergence speed of the Q learning algorithm; at the same time, for the channel selected by the Q learning algorithm based on the simulated link, the corresponding communication direction rationality is obtained, which is recorded as the channel similarity.

[0044] Step 4: record all routing nodes except each simulated link as the bandwidth distribution nodes of each simulated link, analyze the similarity between the vector from the starting point of each simulated link to each bandwidth distribution node and the communication direction vector, obtain each communication distribution weight, and combine the node bandwidth of all bandwidth distribution nodes at each moment to obtain the random process weight of each simulated link at each moment.

[0045] In the Q-learning algorithm, the link selection process from routing node BG to routing node ED is random, but in the actual routing network, the links for communication are not random, but there are obvious tendencies. For example, most of the nodes that issue communication requests and the final communication nodes are fixed, and some routers only serve as transit nodes.

[0046] Therefore, for the selection of routing nodes BG and routing nodes ED, this application characterizes whether the selection of routing nodes conforms to the actual communication situation by calculating the random process weight. The specific method is as follows: There are N routing nodes in the routing node coordinate set. The nodes other than routing node BG and routing node ED are recorded as simulated links. bandwidth distribution node; taking the routing node BG as the starting point and the mth bandwidth distribution node as the end point, the mth communication distribution vector is obtained; the mth cosine similarity of the mth communication distribution vector and the communication direction vector is calculated and normalized to obtain the mth communication distribution weight. In this embodiment, the method for normalizing the cosine similarity of the communication distribution vector and the communication direction vector is: adding 1 to the cosine similarity and then dividing it by 2 to obtain the mth communication distribution weight; the implementer may also use other normalization methods, which are not limited in this application.

[0047] It should be understood that the smaller the cosine similarity between the mth communication distribution vector and the communication direction vector, the closer the mth routing node is to the connecting line segment between routing node BG and routing node ED. When routing node BG and routing node ED are selected for simulated communication, the simulated link is performed through the mth bandwidth distribution node. The more communication can achieve analog link The lower the latency of the communication, the more it conforms to the communication habits of the nodes, and the more likely the mth bandwidth distribution node is to participate in the communication between the two. Therefore, the greater the mth communication distribution weight, the more likely the mth bandwidth distribution node is to participate in the communication between the two routing nodes, which means that the bandwidth distribution node is more likely to become one of the routing nodes in the communication link connecting the two routing nodes.

[0048] Finally, taking the communication distribution weight as the weight, the node bandwidth of all bandwidth distribution nodes is weighted and summed, and then divided by the sum of the bandwidth of all bandwidth distribution nodes to obtain the random process weight. The larger the value, the more the bandwidth of the entire routing network is distributed in the simulated link. The more frequently the routing nodes BG and ED communicate in the current routing network, the more important the simulated data they generate. In particular, the random process weight is related to the time t at which the node bandwidth is obtained. The random process weight is recorded as .

[0049] Step 5: Based on the Q learning algorithm, the channel similarity of the channels corresponding to the routing nodes selected for each simulated link is combined with the random process weight to obtain the channel learning rate of each channel of each simulated link at each moment; whether direct transmission between routing nodes can be used as a judgment criterion, combined with the distribution of node occupancy between routing nodes, an R table is constructed, the channel learning rate is used to replace the original learning rate in the Q learning algorithm, and the Q table is updated.

[0050] After the Q-learning algorithm generates simulated links, each channel can be regarded as a state-action pair in the Q-table. In each simulation operation, the calculation result updates the value of the corresponding element in the Q-table, so that the system can learn how to optimize the selection of communication links based on reward feedback. In this way, the R-table can be used to record reward information and help the system optimize in the process of trial and error. When the Q value in the Q-table converges, it means that Q-learning has successfully learned the optimal strategy, and routing control can be performed through the Q-table.

[0051] In order to speed up the convergence of the R table, this application calculates the channel learning rate. The more the channel conforms to the actual routing control, the higher the channel learning rate is, and the higher the weight is when constructing the R table. The calculation method is: Still using analog links For analysis, assume that this simulation process obtains a total of G channels, and for the g-th channel, obtain the corresponding channel similarity ;Finally calculate the channel learning rate: ;in, is the simulated link at time t The channel learning rate of the g-th channel, is the simulated link at time t The random process weights, It is an analog link The channel similarity of the g-th channel.

[0052] It should be understood that the greater the channel similarity, the more the data obtained by the g-th channel during the simulated communication process conforms to the actual communication situation, and the better the simulated data generated. Therefore, the data generated during this communication process should have a higher learning rate.

[0053] Furthermore, the present application improves the transfer rule formula of the Q learning algorithm through the obtained channel learning rate, thereby accelerating the convergence speed of the Q table.

[0054] The present application obtains the node occupancy rate and node bandwidth of the routing network in real time and constructs an R table: at time t, the sth row and the ath column of the R table represent the channel from the sth routing node to the ath routing node, so the node occupancy rates of the sth routing node and the ath routing node are obtained and averaged, and then the average is subtracted from the first preset value to obtain the channel idle rate as the element of the sth row and the ath column of the R table; the larger the value, the more communication information the channel between the sth routing node and the ath routing node can accommodate, and the channel should be used; if the sth routing node and the ath routing node cannot transmit directly, the second preset value is used as the channel idle rate between the sth routing node and the ath routing node, as the element of the sth row and the ath column of the R table; in this embodiment, the first preset value is greater than or equal to 1, and the value is 1; the second preset value is less than 0, and the value is -1, and the implementer can adjust it according to the actual situation.

[0055] It should be understood that there is a corresponding R table at each moment. The larger the value of an element in the R table, the higher the idle rate between the two routing nodes. Selecting the corresponding node for communication will balance the load of the entire routing network and improve network reliability.

[0056] At time t, the node occupancy rate and node bandwidth of N routing nodes can be obtained, and the communication delay of the entire routing network can be obtained at the same time; when using Q learning for simulation calculation, the communication task of the simulated link BG→ED is analyzed in a simulation process, and the channel is selected After the transfer, the Q table needs to be updated In this embodiment, the channel learning rate Replace the fixed learning rate in the traditional transfer rule formulation.

[0057] It should be noted that the channel learning rate Compared with the traditional fixed learning rate, the channel The higher the channel learning rate, the more it conforms to the channel control result of the actual communication link scheme, the more credible the data generated by the simulation operation, and the greater its impact on the Q table. It can make the learning results of Q learning converge faster to the results similar to the actual communication link scheme, and improve the response speed of the Q learning algorithm to the changes in the routing network state when performing routing selection.

[0058] Step 6: Select the route at each moment according to the Q table updated at each moment and the channel similarity.

[0059] The R table is used as input and the Q learning algorithm is used for calculation to obtain the corresponding Q table; wherein the transfer rule formula in the Q learning algorithm is replaced according to the method described in this embodiment, and the specific calculation process of the Q learning algorithm is a common technology in the field of reinforcement learning and will not be described in detail; Finally, the corresponding Q table is obtained in real time. When routing is selected, the current information is in routing node s. At this time, the Q value of the sth row of the Q table is obtained. There are H non-zero Q values ​​in total, corresponding to H channels. If the hth channel is in the ath column, then its corresponding channel is ; Calculate the channel similarity of the hth channel, then use the Q value of the hth channel to compare the sum of the channel similarity and z to obtain the routing control judgment value. In this embodiment, z is a constant to prevent the denominator from being zero, and its value is 0.01. Select the channel composed of the routing node with the smallest routing judgment value with routing node s as the channel for information transmission.

[0060] It should be understood that the smaller the Q value of the hth channel is, the more likely it is that the routing network load will be balanced when the hth channel is selected; the greater the channel similarity is, the more likely the hth channel is to be transmitted toward the target routing node, and the more likely it is that the hth channel should be selected.

[0061] Based on the same concept as the method embodiment of the present application, a routing selection system suitable for power communication services is proposed, including: A router status acquisition module is used to obtain the node occupancy rate and node bandwidth of each routing node used for data transmission in the power communication business at each moment, as well as the communication delay between routing nodes; A graph search algorithm is used to calculate the communication delay between routing nodes to obtain the shortest communication delay between any two routing nodes and obtain a distance matrix. The distance matrix is ​​reduced in dimension to obtain a set of routing node coordinates. A Q table generation module is used to use the vector formed by the starting points of each simulated link as the communication direction vector in the routing node coordinate set, and to obtain the channel similarity of each routing node corresponding to the channel based on each simulated link by combining the Q learning algorithm with the similarity between the vector formed by each routing node and the routing node directly transmitted to it and the communication direction vector; All routing nodes except for each simulated link are recorded as bandwidth distribution nodes of each simulated link, and the similarity between the vector from the starting point of each simulated link to each bandwidth distribution node and the communication direction vector is analyzed to obtain each communication distribution weight, and the random process weight of each simulated link at each moment is obtained by combining the node bandwidth of all bandwidth distribution nodes at each moment; Based on the Q learning algorithm, according to the channel similarity of the corresponding channels of each routing node selected by each simulated link, combined with the random process weight, the channel learning rate of each channel of each simulated link at each moment is obtained; whether direct transmission between routing nodes can be used as a judgment standard, combined with the distribution of node occupancy between routing nodes, an R table is constructed, the channel learning rate is used to replace the original learning rate in the Q learning algorithm, and the Q table is updated; The route selection module is used to select the route at each moment according to the Q table updated at each moment and the channel similarity.

[0062] Among them, the routing system block diagram applicable to power communication business is as follows Figure 3 shown.

[0063] Based on the same concept as the method embodiment of the present application, a routing selection device suitable for power communication services is proposed, including a memory, a processor, and a computer program stored in the memory and running on the processor, and when the processor executes the computer program, the steps of any one of the above methods are implemented.

[0064] Based on the same concept as the method embodiment of the present application, a routing selection device suitable for power communication services is proposed, in which a computer program is stored, and when the computer program is executed by a processor, any one of the above methods is implemented.

[0065] To summarize, the present application first obtains the node occupancy rate, node bandwidth, and communication delay between routing nodes of each routing node used for data transmission in the power communication business at each moment; since two routing nodes cannot communicate directly, a graph search algorithm is used for the communication delay between routing nodes to obtain the shortest communication delay between any two routing nodes, and obtain a distance matrix, which has the beneficial effect of characterizing the distance between routing nodes through the shortest communication delay; the distance matrix is ​​reduced in dimension to obtain a set of routing node coordinates, which is convenient for subsequent judgment of the directional differences between different channels and the final transmission target through the coordinates of the routing nodes, guiding the selection of the simulation operation scheme of the Q learning algorithm, and obtaining the channel similarity of each routing node corresponding to the channel based on each simulated link, which has the beneficial effect of improving the generation quality of simulation data, thereby improving the response speed of the Q learning algorithm to changes in the routing network state.

[0066] Furthermore, the random process weight is calculated through the node bandwidth and the routing node coordinate set to determine the similarity between the simulated data and the actual communication situation, and to screen out the simulated data that is close to the actual transmission process; the channel learning rate is calculated by combining the channel similarity and the random process weight. The higher the channel learning rate, the more it conforms to the channel control result of the actual communication link plan. When updating the Q table, the learning rate of the corresponding channel is increased, so that the Q learning algorithm can learn more quickly to conform to the actual communication link plan; finally, the route selection is performed through the Q table, and the simulated data is given a differentiated learning rate according to the actual router information transmission habits, which speeds up the Q learning algorithm's learning rate and response speed to changes in the routing network state, making the selected router communication link more reasonable and improving the communication quality.

[0067] It should be noted that the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the system, method and computer program product according to the embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, which can depend on the functions involved. In the description corresponding to the flowchart and block diagram in the accompanying drawings, the operations or steps corresponding to different boxes can also occur in an order different from that disclosed in the description, and sometimes there is no specific order between different operations or steps. For example, two consecutive operations or steps can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, which can depend on the functions involved. Each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that performs the specified functions or actions, or may be implemented by a combination of dedicated hardware and computer instructions.

[0068] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A routing selection method applicable to power communication services, characterized in that: The method comprises the following steps: Obtain the node occupancy rate and node bandwidth of each routing node used for data transmission in the power communication service at each time, as well as the communication delay between routing nodes; A graph search algorithm is used to calculate the communication delay between routing nodes to obtain the shortest communication delay between any two routing nodes and obtain a distance matrix. The distance matrix is ​​reduced in dimension to obtain a set of routing node coordinates. In the routing node coordinate set, the vector formed by the starting point of each simulated link is used as the communication direction vector, and the similarity between the vector formed by each routing node and the routing node directly transmitted by it and the communication direction vector is combined with the Q learning algorithm to obtain the channel similarity of each routing node corresponding to the channel based on each simulated link; All routing nodes except for each simulated link are recorded as bandwidth distribution nodes of each simulated link, and the similarity between the vector from the starting point of each simulated link to each bandwidth distribution node and the communication direction vector is analyzed to obtain each communication distribution weight, and the random process weight of each simulated link at each moment is obtained by combining the node bandwidth of all bandwidth distribution nodes at each moment; Based on the Q learning algorithm, according to the channel similarity of the corresponding channels of each routing node selected by each simulated link, combined with the random process weight, the channel learning rate of each channel of each simulated link at each moment is obtained; whether direct transmission between routing nodes can be used as a judgment standard, combined with the distribution of node occupancy between routing nodes, an R table is constructed, the channel learning rate is used to replace the original learning rate in the Q learning algorithm, and the Q table is updated; The route at each moment is selected according to the Q table updated at each moment and the channel similarity.

2. The routing selection method applicable to power communication services according to claim 1, characterized in that: The channel corresponding to each routing node is obtained based on the channel similarity of each simulated link, specifically: Get all routing nodes that directly transmit to each routing node, and record them as the communication nodes of each routing node; The vector from the routing node to each of its communication nodes is recorded as each communication vector; the normalized value of the cosine similarity between each communication vector and the communication direction vector corresponding to each simulated link is used as the rationality of the communication direction of each communication vector based on each simulated link; The communication direction rationality is used to replace the node selection probability in the Q learning algorithm; For each simulated link, the rationality of the communication direction corresponding to the channel selected by each routing node based on the Q learning algorithm is used as the channel similarity of each routing node based on each simulated link.

3. The routing selection method applicable to power communication services according to claim 1, characterized in that: The communication distribution weights are obtained as follows: The vectors from the starting point of each simulated link to each bandwidth distribution node are recorded as each communication distribution vector; The normalized value of the cosine similarity between each communication distribution vector and the communication direction vector of each simulated link is used as each communication distribution weight.

4. The routing selection method applicable to power communication services according to claim 1, characterized in that: The random process weight of each simulated link at each time is obtained as follows: For each simulated link, the node bandwidth proportion of each bandwidth distribution node at each time is obtained; based on each communication distribution weight, the corresponding node bandwidth proportion is weighted and summed to obtain the random process weight of each simulated link at each time.

5. The routing selection method applicable to power communication services according to claim 1, characterized in that: The specific formula for obtaining the channel learning rate of each channel of each simulated link at each moment is: ;in, is the simulated link at time t The channel learning rate of the g-th channel, is the simulated link at time t The random process weights, It is an analog link The channel similarity of the g-th channel.

6. The routing selection method applicable to power communication services according to claim 1, characterized in that: The construction of the R table is specifically as follows: If direct transmission is possible from routing node s to routing node a, the average of the node occupancy rates from routing node s to routing node a is calculated, and the difference between the first preset value and the average is used as the element of the sth row and the ath column of the R table; If it cannot be directly transmitted, the second preset value is used as the element of the sth row and the ath column of the R table; Among them, the first preset value is greater than or equal to 1; the second preset value is less than 0.

7. The routing selection method applicable to power communication services according to claim 1, characterized in that: The selecting of the route at each moment according to the Q table updated at each moment and the channel similarity comprises: For any routing node at each moment, obtain the Q value with other routing nodes on the Q table, merge the negative correlation mapping results with the channel similarity of other routing nodes, and calculate the routing control judgment value; The routing node with the smallest routing judgment value with any of the routing nodes is selected as the next routing node for information transmission.

8. A routing system for electric power communication services, characterized in that: The system comprises: A router status acquisition module is used to obtain the node occupancy rate and node bandwidth of each routing node used for data transmission in the power communication business at each moment, as well as the communication delay between routing nodes; A graph search algorithm is used to calculate the communication delay between routing nodes to obtain the shortest communication delay between any two routing nodes and obtain a distance matrix. The distance matrix is ​​reduced in dimension to obtain a set of routing node coordinates. A Q table generation module is used to use the vector formed by the starting points of each simulated link as the communication direction vector in the routing node coordinate set, and to obtain the channel similarity of each routing node corresponding to the channel based on each simulated link by combining the Q learning algorithm with the similarity between the vector formed by each routing node and the routing node directly transmitted to it and the communication direction vector; All routing nodes except for each simulated link are recorded as bandwidth distribution nodes of each simulated link, and the similarity between the vector from the starting point of each simulated link to each bandwidth distribution node and the communication direction vector is analyzed to obtain each communication distribution weight, and the random process weight of each simulated link at each moment is obtained by combining the node bandwidth of all bandwidth distribution nodes at each moment; Based on the Q learning algorithm, according to the channel similarity of the corresponding channels of each routing node selected by each simulated link, combined with the random process weight, the channel learning rate of each channel of each simulated link at each moment is obtained; according to the transmission situation between the routing nodes, combined with the distribution of the node occupancy rate between the routing nodes, an R table is constructed, the channel learning rate is used to replace the original learning rate in the Q learning algorithm, and the Q table is updated; The route selection module is used to select the route at each moment according to the Q table updated at each moment and the channel similarity.

9. A routing device for electric power communication services, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A routing device suitable for electric power communication services, characterized in that: The device stores a computer program, wherein the computer program, when executed by a processor, implements the method according to any one of claims 1 to 7.

Images