Routing decision optimization method, routing decision method and related equipment

Abstract

The invention relates to the technical field of wireless communication, and discloses a routing decision optimization method, a routing decision method and related equipment. The routing decision optimization method comprises the following steps: initiating a routing decision in an unmanned aerial vehicle cluster; acquiring link state information of a current unmanned aerial vehicle node in a routing decision process; inputting the link state information into a policy network of a near-end policy optimization model to obtain node performance data, and performing routing decision according to the node performance data; and when the quantity of the node performance data reaches a first quantity threshold value, training a near-end strategy optimization model by using the node performance data, and when the near-end strategy optimization model meets a training ending condition, obtaining a routing decision model. According to the embodiment of the invention, the link stability and network survival time of the flight ad hoc network can be improved.

Description

Technical Field

[0001] The present application relates to the field of wireless communication technology, and in particular to a routing decision optimization method, a routing decision method and related equipment. Background Art

[0002] The coordinated communication between each current drone node in the drone cluster relies on the flying ad hoc network.

[0003] However, due to the high mobility of current drone nodes in the flying ad hoc network, the network topology changes frequently, the quality and stability of the transmission link often change intermittently, and it is easy to have problems such as packet loss and long data transmission time. In addition, due to the limited battery capacity of current drone nodes, the uneven energy consumption of current drone nodes will cause the current drone nodes to exit the network early, thus affecting the survival time of the flying ad hoc network. Summary of the invention

[0004] The purpose of this application is to provide a routing decision optimization method, a routing decision method and related equipment, which can improve the link stability and network survival time of a flight ad hoc network.

[0005] The present application provides a routing decision optimization method, including: Initiate routing decisions in the drone swarm; Obtain link status information of the current drone node during the routing decision process; The link state information is input into the policy network of the proximal policy optimization model to obtain node performance data, and a routing decision is made based on the node performance data; the node performance data includes the link state information, jump action information, action probability information and link reward information of the current drone node, and the jump action information, the action probability information and the link reward information are obtained by making a decision based on the link state information; Determining whether the amount of the node performance data reaches a first amount threshold; If not, return to the step of obtaining the link status information of the current drone node when making a routing decision, or when a complete route is obtained through the routing decision, return to the step of initiating a routing decision in the drone cluster; If so, the node performance data is used to train the proximal policy optimization model, and when the proximal policy optimization model meets the training end condition, a routing decision model is obtained.

[0006] In some embodiments, initiating a routing decision in the drone cluster includes: Randomly selecting drone nodes in the drone cluster as source nodes and destination nodes; A routing decision is initiated to find a route from the source node to the destination node.

[0007] In some embodiments, the link state information includes link stability performance information, link delay performance information, remaining energy information of a current neighbor node, and distance information between the current neighbor node and a destination node, and the current neighbor node is a neighbor node of the current drone node; The expression of the link stability performance information is: , in, is the link stability performance information, is the transmission rate information of the link formed by the current drone node and the current neighbor node, is the duration information of the link formed by the current UAV node and the current neighbor node, i=1,2,...,n, j=1,2,...,m, n is the number of current UAV nodes that are not destination nodes on the route, and m is the number of current neighbor nodes; The expression of the link delay performance information is: , in, is the link delay performance information, is the MAC delay information, is the queuing delay information, To transmit delay information, To propagate delay information; The expression of the remaining energy information of the current neighbor node is: , in, is the remaining energy information of the current neighbor node, is the initial energy information of the current neighbor node, is the flight energy consumption information of the current neighbor node, It is the communication energy consumption information of the current neighbor node.

[0008] In some embodiments, inputting the link state information into a proximal policy optimization model to obtain node performance data, and making routing decisions based on the node performance data includes: generating the jump action information according to the link state information and the type of the current neighbor node; the type of the current neighbor node is a destination node or a non-destination node; generating the action probability information according to the link state information and the jump action information; When there is a current neighbor node of the type of the destination node, generating maximum link reward information, and when there is no current neighbor node of the type of the destination node, generating corresponding link reward information according to the link state information; Based on the node performance data, the current drone node makes a routing decision.

[0009] In some embodiments, when there is no current neighbor node of the type of the destination node, the link reward information is equal to the sum of the first reward information, the second reward information, the third reward information and the fourth reward information, the first reward information is negatively correlated with the distance information between the current neighbor node and the destination node, the second reward information is positively correlated with the link stability performance information of the link formed by the current drone node and the current neighbor node, the third reward information is negatively correlated with the link delay performance information formed by the current drone node and the current neighbor node, and the fourth reward information is positively correlated with the remaining energy information of the current neighbor node.

[0010] In some embodiments, the using the node performance data to train the proximal policy optimization model, and obtaining a routing decision model when the proximal policy optimization model meets the training end condition, includes: Inputting the node performance data into the evaluation network of the proximal strategy optimization model to perform value evaluation on the node performance data to obtain an evaluation result of the node performance data; Calculate the output result of the policy network objective function according to the node performance data; Calculate the output result of the evaluation network objective function according to the evaluation result of the node performance data; Calculating the output result of the model objective function according to the output result of the strategy network objective function and the output result of the evaluation network objective function; According to the output result of the model objective function, the weight parameters of the strategy network and the evaluation network are adjusted so that the output result of the model objective function reaches a maximum value, and the node performance data is discarded; Determining whether the link status information reaches a second quantity threshold; If not, return to the step of obtaining the link status information of the current drone node when making a routing decision, or when a complete route is obtained through the routing decision, return to the step of initiating a routing decision in the drone cluster; If it is reached, the training is terminated and the routing decision model is obtained.

[0011] In some embodiments, the objective function is expressed as: , in, is the output result of the model objective function, The current weight parameters of the model for proximal strategy optimization, is the output result of the objective function of the policy network, To evaluate the output of the network objective function, is the entropy of the strategy, and is the weight coefficient, To find the expected value operation, The network parameters are strategy, is the link state information of the t-th node performance data, t=1,2,...,T, T is the number of node performance data; The calculation formula of the output result of the objective function of the strategy network is: , , , , in, is the strategy update amplitude, is the advantage estimate, min() is the minimum operation, and clip() is the truncation function. Restricted to between, is the hyperparameter used to control clip, To evaluate the state value obtained by the network, For discount rewards, is the strategy update amplitude, For the old strategy, The historical weight parameters of the model for proximal strategy optimization, is the action chosen at time t, is the reward at time g, g∈[t,T], is the discount factor; The calculation formula for the output result of the evaluation network objective function is: , in, To evaluate the state value obtained by the network.

[0012] The present application also provides a routing decision method, including: Get the routing decision initiation instruction; Initiating instructions according to the routing decision, determining drone nodes as source nodes and drone nodes as destination nodes; Initiate a routing decision in the drone cluster based on the source node and the destination node; Obtain link status information of the current drone node during the routing decision process; The link status information is input into a routing decision model to obtain a routing strategy result, and a routing decision is made according to the node performance data; the routing decision model is obtained by using the routing decision optimization method described in any one of claims 1 to 7.

[0013] An embodiment of the present application further provides an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the above method when executing the computer program.

[0014] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program implements the above method when executed by a processor.

[0015] The beneficial effects of the present application are as follows: in the process of routing decision-making of the drone cluster, the link status information of the current drone node is obtained, the link status information is decided by using the proximal policy optimization model, and the node performance data including the link status information, jump action information, action probability information and link reward information of the current drone node is generated, and the routing decision is made according to the node performance data, and when the node performance data reaches the first quantity threshold, the proximal policy optimization model is trained by using the node performance data until the training end condition is met, and a routing decision model with optimized routing decision logic is obtained. Since the link status information generated by multiple routing decisions is used to generate the node performance data including the link status information, jump action information, action probability information and link reward information of the current drone node, and then the generated multiple node performance data are used to train the proximal policy optimization model, the trained routing decision model has the optimized routing decision logic, and the routing decision logic comprehensively considers multiple performance indicators of the link, which can improve the link stability and network survival time of the flying ad hoc network. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a flow chart of the routing decision optimization method provided in an embodiment of the present application.

[0017] Figure 2 It is a flowchart of the specific method of step S103 provided in an embodiment of the present application.

[0018] Figure 3 It is a flowchart of the specific method of step S105 provided in an embodiment of the present application.

[0019] Figure 4 It is a flow chart of the routing decision method provided in an embodiment of the present application.

[0020] Figure 5 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0022] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown can be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second" and the like in the specification, claims and drawings are used to distinguish similar objects, and are not used to describe a specific order or sequence.

[0023] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0024] See also Figure 1 , Figure 1 is a flow chart of a routing decision optimization method provided in an embodiment of the present application. In some embodiments of the present application, Figure 1 The method may specifically include but is not limited to steps S101 to S105. Figure 1 These five steps are introduced in detail.

[0025] Step S101, initiating a routing decision in the drone cluster.

[0026] Routing decision refers to the process by which a routing node selects the best path and forwards the data packet to the next routing node based on the neighbor node information in the routing table. It can be understood that routing decision for drone clusters refers to the process by which a best path is selected and the data packet is forwarded to the next drone node based on the neighbor node information in the routing table of the drone node.

[0027] In a specific embodiment, step S101 specifically includes: Randomly select drone nodes in the drone cluster as source nodes and destination nodes; Initiate a routing decision to find a route from a source node to a destination node.

[0028] In the specific implementation, firstly, a drone node as a source node and a drone node as a destination node are randomly selected in the drone cluster, and then a routing decision is initiated to find a route from the source node to the destination node, and a corresponding routing decision initiation instruction is generated. The routing decision initiation instruction is transmitted to the drone node as the source node, so that the drone node as the source node generates a corresponding data packet.

[0029] Step S102, obtaining the link status information of the current drone node during the routing decision process.

[0030] Before obtaining the link status information of the current drone node in the process of routing decision, the drone node performs route discovery. In the specific implementation, during the route discovery process, the drone node regularly broadcasts hello packets, which contain the node information of the drone node. The node information may include the location information, movement information, remaining energy information, and queuing delay information of the drone node. Each drone node will update its neighbor table based on the received hello packets, perceive the surrounding network conditions, and determine its current neighbor nodes.

[0031] When the routing decision is made to the current drone node, the current drone node determines its current neighbor node based on the received hello data packet, and determines the link that it can form with each neighbor node respectively, and determines the link state information of each link based on the node information of the neighbor node contained in the hello data packet, that is, the link state information of the current drone node. By interacting with the current drone node, the link state information of the current drone node can be obtained.

[0032] The link status information includes link stability performance information, link delay performance information, remaining energy information of the current neighbor node, and distance information between the current neighbor node and the destination node.

[0033] The current neighbor node is the neighbor node of the current drone node.

[0034] The expression of link stability performance information is: , , , in, is the link stability performance information, is the transmission rate information of the link formed by the current drone node and the current neighbor node, is the duration information of the link formed by the current drone node and the current neighbor node, i=1,2,...,n, j=1,2,...,m, n is the number of current drone nodes that are not destination nodes on the route, m is the number of current neighbor nodes, is the forward transmission success rate of the link, is the backward transmission success rate of the link, R is the maximum communication distance of the UAV node, is the distance between the current neighbor node and the current neighbor node; The expression of link delay performance information is: , , , , , in, is the link delay performance information, is the MAC delay information, is the queuing delay information, To transmit delay information, To propagate delay information, For festival rewards, is the time when the current drone node sends the data packet, F is the service rate, A is the arrival rate, is the packet length, is the data transmission rate, is the electromagnetic wave transmission rate; The expression of the remaining energy information of the current neighbor node is: , in, is the remaining energy information of the current neighbor node, is the initial energy information of the current neighbor node, is the flight energy consumption information of the current neighbor node, It is the communication energy consumption information of the current neighbor node.

[0035] Step S103: input the link status information into the policy network of the proximal policy optimization model to obtain node performance data, and make routing decisions based on the node performance data.

[0036] Among them, the node performance data includes the link state information, jump action information, action probability information and link reward information of the current drone node. The jump action information, action probability information and link reward information are obtained by making decisions based on the link state information. The jump action information represents the action of the current drone node forwarding the data packet to the corresponding neighbor node, the action probability information represents the probability of executing the action of the jump action information, and the link reward information represents the reward for executing the action of the jump action information.

[0037] The proximal policy optimization model includes a policy network and an evaluation network, and is a network structure model based on the Proximal Policy Optimization (PPO) algorithm.

[0038] The node performance data is obtained by the policy network based on the link state information. In the specific implementation, the link state information is input into the policy network of the proximal policy optimization model. The policy network determines the action probability of the current drone node forwarding the data packet to the corresponding neighbor node based on the link state information, and generates the corresponding jump action information based on the action probability, and then combines the link reward information and the jump action information to generate the corresponding action probability information, and generates the corresponding link reward information.

[0039] After obtaining the node performance data, a routing decision is made based on the node performance data. In the specific implementation, by interacting with the current drone node, an instruction is sent to the current drone node to forward the data packet to the corresponding neighbor node, so that the current drone node sends a command to forward the data packet to the corresponding neighbor node, and the routing decision of the current drone node is ended. If the neighbor node receiving the data packet is not the destination node, the neighbor node receiving the data packet is used as the current drone node in the next routing decision, and a routing decision is made until a route from the source node to the destination node is obtained.

[0040] See also Figure 2 , Figure 2 is a flowchart of a specific method of step S103 provided in an embodiment of the present application. In some embodiments of the present application, step S103 includes but is not limited to steps S201 to S204. Figure 2 These four steps are introduced in detail.

[0041] Step S201: Generate jump action information according to link status information and the type of current neighbor node.

[0042] The type of the current neighbor node is a destination node or a non-destination node.

[0043] Step S202: Generate action probability information according to the link status information and the jump action information.

[0044] Step S203, when there is a current neighbor node of the type of the destination node, the maximum link reward information is generated; when there is no current neighbor node of the type of the destination node, the corresponding link reward information is generated according to the link state information.

[0045] Step S204, based on the node performance data, the current drone node makes a routing decision.

[0046] First, node performance data is generated. In a specific implementation, when there is a current neighbor node of the type of the destination node, jump action information for forwarding a data packet to the destination node is generated, and the maximum link reward information is generated. When there is no current neighbor node of the type of the destination node, the link with the best link performance is determined according to the link state information, and jump action information for forwarding a data packet to the neighbor node corresponding to the link is generated. According to the link performance of the link represented by the link state information, corresponding link reward information is generated, so that the link reward information is positively correlated with the link performance of the link represented by the link state information, and action probability information is generated according to the link state information and the jump action information to evaluate the probability of forwarding the data packet to the neighbor node corresponding to the action probability information, and the link state information, jump action information, action probability information and link reward information are integrated to obtain node performance data.

[0047] Then, the routing decision is made according to the node performance data. In the specific implementation, the neighbor node to be jumped is determined according to the jump action information in the node performance data, and by interacting with the current drone node, an instruction to forward the data packet to the corresponding neighbor node is issued to the current drone node, so that the current drone node issues an instruction to forward the data packet to the corresponding neighbor node.

[0048] In a specific embodiment, when there is no current neighbor node of the destination node type, the link reward information is equal to the sum of the first reward information, the second reward information, the third reward information and the fourth reward information, the first reward information is negatively correlated with the distance information between the current neighbor node and the destination node, the second reward information is positively correlated with the link stability performance information of the link formed by the current drone node and the current neighbor node, the third reward information is negatively correlated with the link delay performance information formed by the current drone node and the current neighbor node, and the fourth reward information is positively correlated with the remaining energy information of the current neighbor node.

[0049] In a specific embodiment, the calculation formula of the link reward information is: , , , , , in, is the output link reward information, is the maximum link reward information, For the first reward information, For the second reward information, For the third reward information, For the fourth reward information, , , and are all reward weight coefficients, is the distance between the current neighbor node and the destination node.

[0050] Step S104, determining whether the amount of node performance data reaches a first quantity threshold.

[0051] If not, return to step S102, or return to step S101 when a complete route is obtained through routing decision; if reached, execute step S105.

[0052] In a specific implementation, each time a set of node performance data is obtained, the set of node performance data is cached, and then it is determined whether the number of cached node performance data reaches the first quantity threshold. If it reaches, step S105 is executed to train the proximal strategy optimization model using the currently cached node performance data. If it does not reach, it is determined that the amount of node performance data is insufficient. When a complete route is not obtained through routing decision, that is, the next drone node to which the current drone node makes a routing decision and jumps is not the destination node, return to step S102, take the drone node that receives the data packet as the current drone node, obtain the link state information of the current drone node, and then generate corresponding node performance data. When a complete route is obtained through routing decision, that is, the next drone node to which the current drone node makes a routing decision and jumps is the destination node, obtain the route from the source node to the destination node, return to step S101, initiate a new routing decision in the drone cluster, randomly select the source node and the destination node in the drone cluster, obtain the link state information of the source node, and then generate corresponding node performance data.

[0053] Step S105, training the proximal policy optimization model using the node performance data, and obtaining a routing decision model when the proximal policy optimization model meets the training end condition.

[0054] In the specific implementation, the node performance data is respectively input into the policy network and the evaluation network of the proximal policy optimization model to fit the policy network objective function in the policy network and the evaluation network objective function in the evaluation network, and then the model objective function is fitted using the output results of the policy network objective function and the output results of the evaluation network objective function so that the output results of the model objective function reach the maximum value. The output results of the model objective function represent the performance value of the target link, which is obtained by the proximal policy optimization model based on the node performance data generated by the link status information.

[0055] See also Figure 3 , Figure 3 is a flowchart of a specific method of step S105 provided in an embodiment of the present application. In some embodiments of the present application, step S105 includes but is not limited to steps S301 to S307. Figure 3 These seven steps are introduced in detail.

[0056] Step S301, input the node performance data into the evaluation network of the proximal strategy optimization model to perform value evaluation on the node performance data to obtain an evaluation result of the node performance data.

[0057] Step S302, calculating the output result of the policy network objective function according to the node performance data.

[0058] Step S303, calculating the output result of the evaluation network objective function according to the evaluation result of the node performance data.

[0059] Step S304, calculating the output result of the model objective function according to the output result of the policy network objective function and the output result of the evaluation network objective function.

[0060] Step S305, according to the output result of the model objective function, adjust the weight parameters of both the policy network and the evaluation network so that the output result of the model objective function reaches the maximum value, and discard the node performance data.

[0061] Step S306: determine whether the link status information reaches a second quantity threshold.

[0062] If not, return to step S102, or return to step S101 when a complete route is obtained through routing decision; if reached, execute step S307.

[0063] Step S307, end the training and obtain the routing decision model.

[0064] In a specific implementation, the node performance data is input into the evaluation network of the proximal policy optimization model, the evaluation network performs value evaluation on the node performance data and obtains the evaluation result of the node performance data, then the policy network objective function in the policy network is fitted using the node performance data to obtain the output result of the policy network objective function, the evaluation result of the node performance data is used to fit the evaluation network objective function in the evaluation network to obtain the output result of the evaluation network objective function, and then the output result of the policy network objective function and the output result of the evaluation network objective function are used to fit the model objective function, and the output result of the model objective function is made to reach the maximum value by updating the weight parameters of both the policy network and the evaluation network, and the currently cached node performance data is discarded after fitting the model objective function, and it is determined whether the link state information generated in the process of training the proximal policy optimization model reaches the second quantity threshold, if not, it is determined that the training end condition is not met, and the process returns to step S101 or step S102 to generate new link state information, and when the number of node performance data reaches the first quantity threshold again, the proximal policy optimization model is continued to be trained, if it is reached, it is determined that the training end condition is met, and the training is terminated, and the latest policy network objective function is used as the routing decision model, and the routing decision model has an optimized routing decision logic.

[0065] In a specific embodiment, the expression of the objective function is: , in, is the output result of the model objective function, The current weight parameters of the model for proximal strategy optimization, is the output result of the objective function of the policy network, To evaluate the output of the network objective function, is the entropy of the strategy, and is the weight coefficient, To find the expected value operation, The network parameters are strategy, is the link state information of the t-th node performance data, t=1,2,...,T, T is the number of node performance data; The calculation formula for the output of the policy network objective function is: , , , , in, is the strategy update amplitude, is the advantage estimate, min() is the minimum operation, and clip() is the truncation function. Restricted to between, is the hyperparameter used to control clip, To evaluate the state value obtained by the network, For discount rewards, is the strategy update amplitude, For the old strategy, The historical weight parameters of the model for proximal strategy optimization, is the action chosen at time t, is the reward at time g, g∈[t,T], is the discount factor; The calculation formula for evaluating the output of the network objective function is: , in, To evaluate the state value obtained by the network.

[0066] See also Figure 4 , Figure 4 is a flow chart of a routing decision optimization method provided in an embodiment of the present application. In some embodiments of the present application, Figure 4 The method may specifically include but is not limited to steps S401 to S405. Figure 4 These five steps are introduced in detail.

[0067] Step S401, obtaining a routing decision initiation instruction.

[0068] Step S402, initiating instructions according to the routing decision, determining the drone nodes as the source node and the drone nodes as the destination node.

[0069] Step S403, initiating a routing decision in the drone cluster based on the source node and the destination node.

[0070] Step S404, obtaining the link status information of the current drone node during the routing decision process.

[0071] Step S405: input the link status information into the routing decision model, obtain the routing policy result, and make routing decisions according to the node performance data.

[0072] The routing decision model is obtained by using the above-mentioned routing decision optimization method.

[0073] If there are changes in the queues of two drone nodes or drone clusters that have not established routes, it is necessary to establish the corresponding routes through route discovery and route decision. The route discovery is completed in real time by the drone nodes, and the route decision is completed by the terminal that stores the route decision model.

[0074] In the specific implementation, the terminal storing the routing decision model obtains an external routing decision initiation instruction, and according to the routing decision initiation instruction, it is necessary to establish two UAV nodes for routing, that is, determine the UAV node as the source node and the UAV node as the destination node, and then initiate the routing decision in the UAV cluster, and interact with the UAV nodes, starting from the source node, to obtain the link state information of the current UAV node in the routing decision process, and input the link state information into the trained routing decision model, and generate the corresponding routing policy result based on the policy network of the routing decision model. The terminal storing the routing decision model sends the corresponding routing decision instruction to the current UAV node according to the routing policy result, so that the current UAV node makes a routing decision according to the node performance data until a route from the source node to the destination node is obtained.

[0075] Figure 5 It is a block diagram of an electronic device according to an exemplary embodiment.

[0076] Refer to the following Figure 5 The electronic device 500 according to this embodiment of the present disclosure is described. Figure 5 The electronic device 500 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0077] like Figure 5 As shown, the electronic device 500 is in the form of a general computing device. The components of the electronic device 500 may include, but are not limited to: at least one processing unit 510, at least one storage unit 520, a bus 530 connecting different system components (including the storage unit 520 and the processing unit 510), a display unit 540, etc.

[0078] The storage unit stores program codes, and the program codes can be executed by the processing unit 510, so that the processing unit 510 executes the steps according to various exemplary embodiments of the present disclosure described in the above method part of this specification.

[0079] The storage unit 520 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 5201 and / or a cache storage unit 5202 , and may further include a read-only storage unit (ROM) 5203 .

[0080] The storage unit 520 may also include a program / utility 5204 having a set (at least one) of program modules 5205, such program modules 5205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0081] Bus 530 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0082] The electronic device 500 may also communicate with one or more external devices 500' (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 500, and / or communicate with any device that enables the electronic device 500 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 550. Furthermore, the electronic device 500 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 560. The network adapter 560 may communicate with other modules of the electronic device 500 via a bus 530. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 500, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0083] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program, and the computer program implements the above method when executed by a processor.

[0084] The routing decision optimization method, routing decision method and related equipment provided by the embodiment of the present application obtain the link state information of the current drone node in the routing decision process of the drone cluster, use the proximal policy optimization model to make a decision on the link state information, generate node performance data including the link state information, jump action information, action probability information and link reward information of the current drone node, make routing decisions according to the node performance data, and use the node performance data to train the proximal policy optimization model until the training end condition is met when the node performance data reaches the first quantity threshold, so as to obtain a routing decision model with optimized routing decision logic. Since the link state information generated by multiple routing decisions is used to generate the node performance data including the link state information, jump action information, action probability information and link reward information of the current drone node, and then the generated multiple node performance data are used to train the proximal policy optimization model, the trained routing decision model has the optimized routing decision logic, and the routing decision logic comprehensively considers multiple performance indicators of the link, which can improve the link stability and network survival time of the flying ad hoc network.

[0085] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, or a network device, etc.) to execute the above method according to the implementation of the present disclosure.

[0086] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0087] Computer readable storage media may include data signals propagated in baseband or as part of a carrier wave, wherein readable program codes are carried. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or device. The program codes contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination thereof.

[0088] Program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0089] Those skilled in the art will appreciate that the above modules can be distributed in the device according to the description of the embodiment, or can be changed accordingly and only used in one or more devices different from the embodiment. The modules of the above embodiments can be combined into one module, or further divided into multiple sub-modules.

[0090] The exemplary embodiments of the present disclosure are specifically shown and described above. It should be understood that the present disclosure is not limited to the detailed structures, configurations or implementations described herein; on the contrary, the present disclosure is intended to cover various modifications and equivalent configurations included in the spirit and scope of the appended claims.

Claims

1. A routing decision optimization method, characterized in that: include: Initiate routing decisions in the drone swarm; Obtain link status information of the current drone node during the routing decision process; The link state information is input into the policy network of the proximal policy optimization model to obtain node performance data, and a routing decision is made based on the node performance data; the node performance data includes the link state information, jump action information, action probability information and link reward information of the current drone node, and the jump action information, the action probability information and the link reward information are obtained by making a decision based on the link state information; Determining whether the amount of the node performance data reaches a first amount threshold; If not, return to the step of obtaining the link status information of the current drone node when making a routing decision, or when a complete route is obtained through the routing decision, return to the step of initiating a routing decision in the drone cluster; If so, the node performance data is used to train the proximal policy optimization model, and when the proximal policy optimization model meets the training end condition, a routing decision model is obtained.

2. The routing decision optimization method according to claim 1, characterized in that: The initiating of routing decision in the drone cluster includes: Randomly selecting drone nodes in the drone cluster as source nodes and destination nodes; A routing decision is initiated to find a route from the source node to the destination node.

3. The routing decision optimization method according to claim 1, characterized in that: The link state information includes link stability performance information, link delay performance information, remaining energy information of the current neighbor node, and distance information between the current neighbor node and the destination node, and the current neighbor node is a neighbor node of the current drone node; The expression of the link stability performance information is: ， in, is the link stability performance information, is the transmission rate information of the link formed by the current drone node and the current neighbor node, is the duration information of the link formed by the current UAV node and the current neighbor node, i=1,2,...,n, j=1,2,...,m, n is the number of current UAV nodes that are not destination nodes on the route, and m is the number of current neighbor nodes; The expression of the link delay performance information is: ， in, is the link delay performance information, is the MAC delay information, is the queuing delay information, To transmit delay information, To propagate delay information; The expression of the remaining energy information of the current neighbor node is: ， in, is the remaining energy information of the current neighbor node, is the initial energy information of the current neighbor node, is the flight energy consumption information of the current neighbor node, It is the communication energy consumption information of the current neighbor node.

4. The routing decision optimization method according to claim 1, characterized in that: The step of inputting the link status information into a proximal policy optimization model to obtain node performance data, and making a routing decision based on the node performance data includes: generating the jump action information according to the link state information and the type of the current neighbor node; the type of the current neighbor node is a destination node or a non-destination node; generating the action probability information according to the link state information and the jump action information; When there is a current neighbor node of the type of the destination node, generating maximum link reward information, and when there is no current neighbor node of the type of the destination node, generating corresponding link reward information according to the link state information; Based on the node performance data, the current drone node makes a routing decision.

5. The routing decision optimization method according to claim 4, characterized in that: When there is no current neighbor node of the type of the destination node, the link reward information is equal to the sum of the first reward information, the second reward information, the third reward information and the fourth reward information, the first reward information is negatively correlated with the distance information between the current neighbor node and the destination node, the second reward information is positively correlated with the link stability performance information of the link formed by the current drone node and the current neighbor node, the third reward information is negatively correlated with the link delay performance information formed by the current drone node and the current neighbor node, and the fourth reward information is positively correlated with the remaining energy information of the current neighbor node.

6. The routing decision optimization method according to claim 1, characterized in that: The using the node performance data to train the proximal policy optimization model, and obtaining a routing decision model when the proximal policy optimization model meets the training end condition, includes: Inputting the node performance data into the evaluation network of the proximal strategy optimization model to perform value evaluation on the node performance data to obtain an evaluation result of the node performance data; Calculate the output result of the policy network objective function according to the node performance data; Calculate the output result of the evaluation network objective function according to the evaluation result of the node performance data; Calculating the output result of the model objective function according to the output result of the strategy network objective function and the output result of the evaluation network objective function; According to the output result of the model objective function, the weight parameters of the strategy network and the evaluation network are adjusted so that the output result of the model objective function reaches a maximum value, and the node performance data is discarded; Determining whether the link status information reaches a second quantity threshold; If not, return to the step of obtaining the link status information of the current drone node when making a routing decision, or when a complete route is obtained through the routing decision, return to the step of initiating a routing decision in the drone cluster; If it is reached, the training is terminated and the routing decision model is obtained.

7. The routing decision optimization method according to claim 6, characterized in that: The expression of the objective function is: ， in, is the output result of the model objective function, The current weight parameters of the model for proximal strategy optimization, is the output result of the objective function of the policy network, To evaluate the output of the network objective function, is the entropy of the strategy, and is the weight coefficient, To find the expected value operation, The network parameters are strategy, is the link state information of the t-th node performance data, t=1,2,...,T, T is the number of node performance data; The calculation formula of the output result of the objective function of the strategy network is: ， ， ， ， in, is the strategy update amplitude, is the advantage estimate, min() is the minimum operation, and clip() is the truncation function. Restricted to between, is the hyperparameter used to control clip, To evaluate the state value obtained by the network, For discount rewards, is the strategy update amplitude, For the old strategy, The historical weight parameters of the model for proximal strategy optimization, is the action chosen at time t, is the reward at time g, g∈[t,T], is the discount factor; The calculation formula for the output result of the evaluation network objective function is: ， in, To evaluate the state value obtained by the network.

8. A routing decision method, characterized in that: include: Get the routing decision initiation instruction; Initiating instructions according to the routing decision, determining drone nodes as source nodes and drone nodes as destination nodes; Initiate a routing decision in the drone cluster based on the source node and the destination node; Obtain link status information of the current drone node during the routing decision process; Inputting the link status information into a routing decision model to obtain a routing strategy result, and making a routing decision based on the node performance data; The routing decision model is obtained by using the routing decision optimization method described in any one of claims 1 to 7.

9. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 8 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images