A reinforcement learning-based intelligent opportunistic routing method, system, device, and storage medium for self-organizing networks of power line inspection drones.

By introducing a reinforcement learning-based intelligent opportunistic routing method into the self-organizing network of power inspection drones, appropriate forwarding nodes are selected and Q-values ​​are updated. This solves the problems of easy interruption of routing links, imbalance between energy consumption and latency, and data packet loss in network partitions in the self-organizing network of drones, improves the packet delivery rate and end-to-end latency, and supports efficient inspection tasks.

CN123124048APending Publication Date: 2026-11-17INFORMATION & COMM BRANCH OF STATE GRID JIANGSU ELECTRIC POWER +2
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611311525.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-08-27
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

In the self-organizing network of power inspection drones, existing technologies suffer from problems such as easy interruption of routing links, imbalance between energy consumption and latency, and data packet loss in network partitions. Traditional MANET routing protocols cannot adapt to the high-speed movement and energy constraints of drones, resulting in low data packet delivery rate and increased end-to-end latency.

Method used

A reinforcement learning-based intelligent opportunistic routing method for power line inspection UAV ad hoc networks is adopted. Each UAV node updates its neighbor information table, selects candidate forwarding nodes based on latency models, energy consumption models, and residual value filtering strategies, and selects target forwarding nodes by combining a Q-learning decision framework or DTN carrying strategy. The Q value is updated by acknowledgment response (ACK) messages to achieve dynamic routing decisions.

Benefits of technology

It effectively improves the data packet delivery rate, achieves reasonable end-to-end latency, solves the problems of easy interruption of routing links, imbalance between energy consumption and latency, and data packet loss in network partitions in power inspection scenarios, and improves the efficiency of inspection tasks of drone swarms.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123124048A_ABST
    Figure CN123124048A_ABST
Patent Text Reader

Abstract

This invention discloses a reinforcement learning-based intelligent opportunistic routing method, system, device, and storage medium for self-organizing networks of power line inspection UAVs. It includes: each current UAV node updating its neighbor information table based on probe messages from neighboring UAV nodes; when the current UAV node sends a data packet, it selects one or more neighboring UAV nodes as candidate forwarding nodes from the neighbor information table based on a latency model, energy consumption model, and a residual value filtering strategy; if candidate forwarding nodes exist, it selects the node with the highest Q value from all candidate forwarding nodes as the target forwarding node using a Q-learning decision framework; otherwise, it selects the node with the highest DTN value from the current UAV node and neighboring UAV nodes as the target forwarding node based on a DTN carrying strategy; after the current UAV node forwards the data packet to the target forwarding node, it calculates a reward value based on the received ACK message after forwarding and updates the Q value accordingly, thus improving routing reliability.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of communication technology for power line inspection by unmanned aerial vehicle (UAV) swarms, and particularly to a reinforcement learning intelligent opportunistic routing method, system, device, and storage medium for self-organizing networks of UAVs used for power line inspection. Background Technology

[0002] In recent years, with the rapid development of drone technology, Flying Ad-Hoc Networks (FANETs) have been widely used in emergency disaster relief, environmental monitoring, and refined inspection of power transmission and transformation facilities due to their flexibility and efficiency. Although FANETs originated from traditional Mobile Ad-Hoc Networks (MANETs), their core characteristics differ significantly. Traditional MANETs are mainly designed for low-speed ground movement scenarios, with topology change cycles mostly on the order of seconds, and some nodes can rely on fixed power sources or large-capacity batteries to alleviate energy pressure. In contrast, drone nodes in FANETs move at higher speeds, increasing the topology change frequency to the order of milliseconds. At the same time, drones are limited by payload, and with energy reserves far lower than those of ground MANET nodes, the increased energy consumption from high-speed aerial movement and hovering operations must be considered. In addition, aerial communication links are easily blocked by power transmission towers, dense power lines, and mountainous terrain. When drones fly around towers or traverse complex terrain, intermittent network partitions can easily form, affecting link stability.

[0003] The highly dynamic topology, energy constraints, and network partitioning issues caused by the FANET characteristics mentioned above result in three major drawbacks when using traditional MANET routing protocols for direct migration: First, topology tracking lags, failing to match the millisecond-level position changes of UAVs in real time, easily leading to routing decisions being out of sync with the actual topology; second, link maintenance overhead surges, with frequent link disconnections and reconnections severely consuming UAV energy; and third, network partitioning and packet loss occur, failing to address the lack of immediate paths faced by UAVs when multiple UAVs are collaboratively inspecting the same tension section or encountering geographical obstacles, resulting in routing holes and causing significant packet loss. Summary of the Invention

[0004] This invention provides a reinforcement learning-based intelligent opportunistic routing method, system, device, and storage medium for self-organizing networks of power line inspection drones, in order to solve the problems of easy interruption of routing links, imbalance between energy consumption and latency, and data packet loss due to network partitioning in existing power line inspection scenarios.

[0005] According to one aspect of the present invention, a reinforcement learning intelligent opportunistic routing method for power line inspection unmanned aerial vehicle (UAV) ad hoc networks is provided, applied to a UAV ad hoc network (FANET), wherein the FANET comprises multiple randomly distributed UAV nodes and a fixed base station, the method comprising: Each current UAV node updates its neighbor information table based on the probe messages from its neighboring UAV nodes; When the current UAV node sends a data packet, it selects one or more neighboring UAV nodes as candidate forwarding nodes from the neighbor information table based on the latency model, energy consumption model, and remaining degree value filtering strategy. When candidate forwarding nodes exist, the node with the highest action value Q is selected from all candidate forwarding nodes using the Q-learning decision framework as the target forwarding node; otherwise, the node with the highest DTN value is selected from the current UAV node and the neighboring UAV nodes as the target forwarding node based on the DTN carrying strategy. After the current UAV node forwards the data packet to the target forwarding node, the reward value is calculated based on the acknowledgment (ACK) message received after forwarding, and the Q value is updated based on the reward value. The Q-learning decision framework includes a state space, an action space, and a value evaluation mechanism. The state space is defined as the remaining energy of the current UAV node, the state of neighboring UAV nodes, and the link state. The action space is defined as the selection of the next-hop node, including the data packets carried by the node itself. The value evaluation mechanism is implemented through the Q value.

[0006] According to another aspect of the present invention, a reinforcement learning intelligent opportunistic routing system for power line inspection unmanned aerial vehicle (UAV) ad hoc networks is provided, applied to a UAV ad hoc network (FANET), wherein the FANET comprises multiple randomly distributed UAV nodes and fixed base stations, and the system includes: The neighbor interaction module is used by each current UAV node to update the neighbor information table based on the probe messages of neighboring UAV nodes; The network modeling module is used to select one or more neighboring UAV nodes as candidate forwarding nodes from the neighbor information table based on the latency model, energy consumption model and remaining degree value filtering strategy when the current UAV node sends a data packet. The routing decision module is used to select the node with the highest action value Q from all candidate forwarding nodes as the target forwarding node when candidate forwarding nodes exist, using the Q-learning decision framework; otherwise, it selects the node with the highest DTN value from the current UAV node and the neighboring UAV nodes as the target forwarding node based on the DTN carrying strategy. The performance optimization module is used to calculate the reward value based on the ACK message received after forwarding the data packet from the current UAV node to the target forwarding node, and update the Q value based on the reward value. The Q-learning decision framework includes a state space, an action space, and a value evaluation mechanism. The state space is defined as the remaining energy of the current UAV node, the state of neighboring UAV nodes, and the link state. The action space is defined as the selection of the next-hop node, including the data packets carried by the node itself. The value evaluation mechanism is implemented through the Q value.

[0007] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising: at least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, which enables the at least one processor to execute the reinforcement learning intelligent opportunistic routing method for power line inspection drone ad hoc networks as described in any embodiment of the present invention.

[0008] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions, the computer instructions being configured to cause a processor to execute and implement the reinforcement learning intelligent opportunistic routing method for power line inspection UAV ad hoc networks as described in any embodiment of the present invention.

[0009] This invention discloses a reinforcement learning-based intelligent opportunistic routing method, system, device, and storage medium for self-organizing networks of power line inspection UAVs. The method includes: each current UAV node updating its neighbor information table based on probe messages from neighboring UAV nodes; when the current UAV node sends a data packet, selecting one or more neighboring UAV nodes as candidate forwarding nodes from the neighbor information table based on a latency model, energy consumption model, and a remaining value filtering strategy; when candidate forwarding nodes exist, selecting the node with the highest action value Q from all candidate forwarding nodes using a Q-learning decision framework as the target forwarding node; otherwise, selecting the node with the highest DTN value from the current UAV node and the neighboring UAV nodes as the target forwarding node based on a DTN carrying strategy; after the current UAV node forwards the data packet to the target forwarding node, calculating a reward value based on the received ACK message after forwarding, and updating the Q value based on the reward value; wherein the Q-learning decision framework includes a state space, an action space, and a value evaluation mechanism; the state space is defined as the remaining energy of the current UAV node, the state of neighboring UAV nodes, and the link state; the action space is defined as the next-hop node selection, including the data packet carried by the node itself; the value evaluation mechanism is implemented through the Q value. This method, by introducing a three-layer collaborative mechanism of residual value filtering, DTN carrying strategy and Q-learning decision, can effectively improve the packet delivery rate during routing and achieve reasonable end-to-end latency. It solves the key problems of easy interruption of routing links, imbalance between energy consumption and latency and data packet loss in network partitions in power inspection scenarios.

[0010] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a flowchart illustrating a reinforcement learning intelligent opportunistic routing method for self-organizing networks of power line inspection drones, as provided in Embodiment 1 of the present invention. Figure 2 This is a flowchart illustrating a reinforcement learning intelligent opportunistic routing method for self-organizing networks of power line inspection drones, as disclosed in an embodiment of the present invention. Figure 3A comparison chart of data packet delivery rates between the present invention and existing technologies is provided for embodiments of the present invention. Figure 4 A comparison diagram of end-to-end latency between the present invention and existing technologies is provided for embodiments of the present invention. Figure 5 A comparison chart of maximum speed and data packet delivery rate of UAVs of the present invention and prior art is provided for embodiments of the present invention. Figure 6 A comparison chart of maximum speed and average end-to-end delay of UAVs of the present invention and prior art is provided for embodiments of the present invention. Figure 7 This is a schematic diagram of the structure of a reinforcement learning intelligent opportunistic routing system for power inspection drone self-organizing networks provided in Embodiment 2 of the present invention; Figure 8 This is a schematic diagram of the electronic device for a reinforcement learning intelligent opportunistic routing method for power line inspection drone self-organizing networks, according to an embodiment of the present invention. Detailed Implementation

[0013] To enable those skilled in the art to better understand the present invention, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention. It should be understood that the various steps described in the method embodiments of the present invention can be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present invention is not limited in this respect.

[0014] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0015] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, any variations of the terms "comprising" and "having," etc., are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0016] It should be noted that the terms "a" and "a plurality of" used in this invention are illustrative rather than restrictive. Those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0017] The names of the messages or information exchanged between the multiple devices in the embodiments of the present invention are for illustrative purposes only and are not intended to limit the scope of these messages or information.

[0018] Traditional ad hoc network routing algorithms still rely on the instantaneous location dependence of MANETs, ​​failing to consider the drastic dynamic changes in FANET topology. This makes them prone to routing holes or loops, resulting in low packet delivery rates and increased end-to-end latency. While some reinforcement learning-based routing protocols can adapt to environmental changes by dynamically updating link values, they still suffer from redundant routing overhead and lack effective strategies for handling network partitioning scenarios. Existing research has considered the high-speed movement of UAVs and the limited energy of nodes. Ignoring node movement and remaining energy in routing decisions can lead to premature battery depletion for some UAVs, affecting the endurance integrity and data transmission reliability of a single inspection mission.

[0019] To address the existing problems, this invention provides an intelligent opportunistic routing method based on reinforcement learning policy forwarding for ad hoc networks of power line inspection drones, as detailed below: Example 1 Figure 1 This is a flowchart illustrating a reinforcement learning intelligent opportunistic routing method for power inspection drone ad hoc networks provided in Embodiment 1 of the present invention. This method is applicable to the situation of determining forwarding strategies for drone nodes in drone ad hoc networks. This method can be executed by a reinforcement learning intelligent opportunistic routing system for power inspection drone ad hoc networks, wherein the system can be implemented by software and / or hardware, and is generally integrated on electronic devices. In this embodiment, the electronic devices include, but are not limited to, devices such as computers.

[0020] like Figure 1 As shown in Embodiment 1 of the present invention, a reinforcement learning intelligent opportunistic routing method for power line inspection UAV ad hoc networks is provided. This method is applied to a UAV ad hoc network (FANET), which includes multiple randomly distributed UAV nodes and a fixed base station. The method comprises the following steps: S110. Each current UAV node updates its neighbor information table based on the probe messages from its neighboring UAV nodes.

[0021] The current UAV node can be the drone node that needs to forward data packets, and the neighboring UAV nodes can be the drone nodes detected by the current UAV node through probe packets. The neighbor information table is a table used to store information about neighboring UAV nodes, and each UAV node has its own corresponding neighbor information table.

[0022] In this embodiment, the FANET includes multiple randomly distributed UAV nodes and fixed base stations. The number of UAV nodes and fixed base stations can be set according to actual conditions. Each UAV node can send data packets to other UAV nodes or fixed base stations. The destination node of the data packets sent by the UAV can be other UAV nodes or fixed base stations. When the destination node is not a neighboring UAV node, the data packet can be forwarded by relay nodes and eventually reach the destination node.

[0023] Before forwarding data packets, each UAV node can update its neighbor information table based on the probe messages from neighboring UAV nodes. If the local neighbor information table does not exist, it can be constructed based on the probe messages from neighboring UAV nodes.

[0024] S120. When the current UAV node sends a data packet, based on the latency model, energy consumption model and remaining degree value filtering strategy, select one or more neighboring UAV nodes from the neighbor information table as candidate forwarding nodes.

[0025] Among them, the latency model can be a model to determine the latency of forwarding data packets, the energy consumption model can be a model to determine the energy consumption of forwarding data packets, and the remaining value filtering strategy can be a strategy to determine whether there are other UAV nodes that can forward data packets besides the current UAV node.

[0026] In this embodiment, when the current UAV node sends a data packet, it can select one or more neighboring UAV nodes that meet the requirements of latency, energy consumption and remaining value as candidate forwarding nodes from the neighbor information table based on the common constraints of the latency model, energy consumption model and remaining value screening strategy.

[0027] S130. When there are candidate forwarding nodes, the node with the highest action value Q is selected from all candidate forwarding nodes using the Q-learning decision framework as the target forwarding node; otherwise, the node with the highest DTN value is selected from the current UAV node and the neighboring UAV nodes as the target forwarding node based on the DTN carrying strategy.

[0028] Q-learning decision framework can be a model-free reinforcement learning algorithm. It decomposes the value (Q-value) of taking a specific action in a given state into the sum of the immediate reward obtainable by the current action and the total potential future value. Through continuous trial and error and experience accumulation, Q-learning iteratively updates the estimate of the Q-value, gradually approaching the true optimal value, thus enabling the agent to learn to find the optimal strategy in unknown environments. The action value Q refers to the value obtainable by taking the current action. The target forwarding node can be the next node to arrive at the data packet. Delay-Tolerant Networking (DTN) can be a type of heterogeneous network architecture oriented towards intermittent connectivity, dynamically changing topologies, and the difficulty in maintaining end-to-end paths, enabling data transmission under unreliable, high-latency, or disconnected conditions. DTN carrying strategy can be a strategy used to quantify the value of each node carrying data packets while waiting for the link to recover and subsequently forwarding them in scenarios where no available candidate forwarding nodes exist (such as network partitioning). DTN value can be an indicator used to evaluate the quality of data packets carried by a node.

[0029] In this embodiment, when candidate forwarding nodes exist, the Q-value for the current UAV node to forward data packets to each candidate forwarding node can be determined using the Q-learning decision framework. The node with the highest action value Q among all candidate forwarding nodes is then selected as the target forwarding node. When no candidate forwarding nodes exist, the DTN value of the data packets carried by the current UAV node and its neighboring UAV nodes can be determined based on the DTN carrying strategy. The node with the highest DTN value among the current UAV node and its neighboring UAV nodes is then selected as the target forwarding node. This ensures the stability of data packet forwarding and enables scenario-based adaptation to the joint optimization objective.

[0030] S140. After the current UAV node forwards the data packet to the target forwarding node, calculate the reward value based on the ACK message received after forwarding, and update the Q value based on the reward value.

[0031] The Q-learning decision framework can include a state space, an action space, and a value evaluation mechanism. The state space can be defined as the remaining energy of the current UAV node, the state of neighboring UAV nodes, and the link state. The action space can be defined as the selection of the next-hop node, including the data packets carried by the node itself. The value evaluation mechanism can be implemented using the Q value. An acknowledgment (ACK) message can be a message returned by the target forwarding node to the current UAV node after receiving the data packet.

[0032] In this embodiment, after the current UAV node forwards the data packet to the target forwarding node, a reward value can be calculated based on the received ACK message after forwarding, and the Q value can be updated based on the reward value. If the data packet is delivered or expires, performance indicators including delivery rate and latency are recorded and the data packet is cleared.

[0033] This embodiment is based on the Q-learning based Intelligent Opportunistic Routing (QIOR) protocol. It addresses core challenges such as millisecond-level topology changes in unmanned aerial vehicle (FANET) networks, limited node energy during inspection mission cycles, and low data transmission reliability. It introduces a three-layer collaborative mechanism of residual degree value screening, DTN carrying strategy, and Q-learning decision-making to solve key problems in power grid inspection scenarios, such as easy interruption of routing links, imbalance between energy consumption and latency, and data packet loss in network partitions.

[0034] The optimization problem of QIOR can be expressed as minimizing end-to-end latency and average energy consumption of UAV nodes. End-to-end latency is calculated as the sum of single-hop latencies generated by all next-hop routing choices for the current data packet. Single-hop latency consists of MAC latency and queuing latency. Average energy consumption of UAV nodes is calculated as the average energy consumed by all nodes in forwarding data packets. This embodiment, considering constraints such as data packet expiration time and node energy thresholds, can achieve multi-objective joint optimization that maximizes data packet delivery rate and minimum remaining node energy, and minimizes end-to-end latency and average node energy consumption. Based on Q-learning, the distributed intelligent decision-making algorithm, through dynamic neighbor perception, closed node exclusion, partition adaptive processing, and reward function iterative optimization, can simultaneously achieve real-time performance, stability, and energy efficiency in routing selection, supporting UAV swarms to efficiently complete inspection data transmission tasks in scenarios such as refined inspection of power transmission and transformation lines and collaborative surveys around substations.

[0035] This invention provides a reinforcement learning-based intelligent opportunistic routing method for self-organizing networks of power line inspection UAVs, comprising: each current UAV node updating its neighbor information table based on probe messages from neighboring UAV nodes; when the current UAV node sends a data packet, selecting one or more neighboring UAV nodes as candidate forwarding nodes from the neighbor information table based on a latency model, energy consumption model, and a remaining value filtering strategy; when a candidate forwarding node exists, selecting the node with the highest action value Q from all candidate forwarding nodes using a Q-learning decision framework as the target forwarding node; otherwise, selecting the node with the highest DTN value from the current UAV node and the neighboring UAV nodes as the target forwarding node based on a DTN carrying strategy; after the current UAV node forwards the data packet to the target forwarding node, calculating a reward value based on the received ACK message after forwarding, and updating the Q value based on the reward value; wherein, the Q-learning decision framework includes a state space, an action space, and a value evaluation mechanism, the state space being defined as the remaining energy of the current UAV node, the state of neighboring UAV nodes, and the link state; the action space being defined as the next-hop node selection, including the data packet carried by itself; and the value evaluation mechanism being implemented through the Q value. This method, by introducing a three-layer collaborative mechanism of residual value filtering, DTN carrying strategy and Q-learning decision, can effectively improve the packet delivery rate during routing and achieve reasonable end-to-end latency. It solves key problems such as easy interruption of routing links, imbalance between energy consumption and latency, and data packet loss in network partitions in power inspection scenarios.

[0036] Based on the above embodiments, modified embodiments of the above embodiments are proposed. It should be noted that, in order to keep the description brief, only the differences from the above embodiments are described in the modified embodiments.

[0037] This invention provides an architecture based on the QIOR protocol, which may include a network topology, node model, coordinate system definition, and a Q-learning decision framework. The network topology can be a reinforcement learning-driven, highly dynamic FANET routing system architecture, comprising randomly distributed unmanned aerial vehicle (UAV) nodes and fixed wireless communication base stations (BS). Each data packet randomly selects one UAV as the source node, with other UAVs acting as relay nodes, communicating with other UAVs or base stations via wireless links. The base station can act as the destination node, receiving and storing data. The QIOR protocol, based on the Q-learning reinforcement learning algorithm, models the routing problem as a Markov decision process. During training, data packets can be randomly generated. UAV nodes determine their actions based on their neighbor Q-tables and receive feedback to update their own Q-value tables; this training is repeated until all data packets have been sent.

[0038] For example, in the QIOR protocol, UAVs can adopt a random waypoint movement model. The target area for UAV flight can be set as a two-dimensional simulation area of ​​1500m × 1500m, containing 30 UAVs conforming to the random waypoint movement model and 1 base station. A certain speed lower limit is set for the UAVs during network formation to adapt to their highly dynamic movement characteristics.

[0039] Establish a node model: Assume that UAV nodes are randomly distributed, defined as follows: Each UAV is equipped with a wireless communication module and a positioning module, supporting detection message (such as Hello message) interaction and location awareness. It moves using random waypoints at a speed range of 10-60m / s. The base station is located at the edge of the area, with a fixed position of (750m, 0m) and a communication radius of 450m, serving as the final delivery node for data packets.

[0040] Coordinate system definition: In this embodiment of the invention, a two-dimensional Cartesian coordinate system can be used to describe the positions of multiple unmanned aerial vehicles (UAVs) and the base station. Each UAV is located in a time slot. t Location Represented as , n This is the serial number of the drone. The first n Driving a UAV in a time slot t hour, x and y Location in the direction. Base station location. for . No. n Driving a UAV in a time slot t The moving velocity vector for , The first n Driving a UAV in a time slot t hour, x and y The velocity vector in the direction of movement. The movement trajectory of neighboring nodes can be predicted through Hello message interaction.

[0041] Q-learning decision-making framework: can include state space, action space, and value evaluation mechanism: State space SThis can be defined as the remaining energy of the current drone node, the status of neighboring drone nodes, and the link status. The neighboring drone node status can include remaining energy, location, and speed, while the link status can include Media Access Control (MAC) latency, queuing latency, link lifetime, and buffer queue length. Action Space A It can be defined as the selection of the next hop node, including the data packet it carries; the value assessment is achieved through the Q value, which represents the cumulative reward for choosing a certain action in the current state, and is updated iteratively through node feedback to adapt to the dynamic changes of the network.

[0042] In one embodiment, each current UAV node periodically sends probe messages to achieve neighbor discovery. The probe message format includes the current UAV node's node identifier, velocity, degree value, remaining energy, current location, and the location of the next target point. Correspondingly, each current UAV node updates its neighbor information table based on the probe messages from neighboring UAV nodes, including: When the current UAV node receives a probe message from a neighboring UAV node, the current UAV node determines the velocity component of the neighboring UAV node based on the neighboring UAV node's current position, velocity, and the position of the next target point: ; in, i For the current UAV node, j For the neighboring UAV nodes of the UAV node, Indicates neighboring UAV nodes j The velocity vector of movement, , These represent neighboring UAV nodes. j exist x and y The velocity vector in the direction of movement; , Neighboring UAV nodes j exist x and y Displacement components in the direction, , , ( x j , y j ) is the neighboring UAV node j The current position coordinates, ( x j ’ , y j ’ ) is the neighboring UAV node j The coordinates of the next target point; Based on the velocity components of the current UAV node itself and the velocity components of neighboring UAV nodes, determine the relative velocity components of the neighboring UAV nodes relative to the current UAV node: ; in, , These represent the current UAV nodes. i exist x and y The velocity vector in the direction of movement. , These represent the relative velocity components at... x and y The velocity vector in the direction of movement; Based on the relative velocity components, the current position of the current UAV node, and the current positions of neighboring UAV nodes, determine the link lifetime between the current UAV node and its neighboring UAV nodes. : ; in, For link l Survival time, ( x i , y i ) represents the current UAV node i The current position coordinates, R c For drone nodes i and j Communication radius between them t Indicates time; Based on the link lifetime and the arrival time of the probe packets from the neighboring UAV node, the failure time of the link of the neighboring UAV node is determined; The neighbor information table is updated based on the failure time, the velocity components of the neighbor UAV nodes, and the probe messages.

[0043] Here, "link" can refer to the current UAV node. i With one of its neighboring UAV nodes j A direct wireless communication link between them.

[0044] In this embodiment, based on the above architecture, a neighbor information interaction model, an energy consumption model, and a latency model can be established.

[0045] Establishing a neighbor information interaction model: In the flight ad hoc network scenario proposed in this embodiment, the UAV discovers neighbors by periodically sending Hello messages. The message sending period can be... The format of the Hello message is defined to include the node identifier (ID), velocity, degree value, remaining energy, current position, and the position of the next target point. Assume the current UAV node... i Received from neighbor UAV node j After sending the Hello message, based on the neighboring UAV nodes j The velocity components of neighboring nodes are calculated based on their location information, velocity, and the location information of the next target point. , .

[0046] Similarly, the current UAV node i By calculating its own velocity components, the neighboring UAV nodes can be obtained. j The relative velocity component relative to the current node is , Through the following formula The formula for calculating link lifetime is as follows: This formula represents the time between two nodes (nodes) during the link's lifespan. i and j The distance between nodes should not exceed the communication radius. This should be combined with information from neighboring UAV nodes. j The arrival time of the Hello message can be used to obtain the failure time of the neighbor's UAV node link. Based on this, a neighbor information table is constructed, which includes information such as the neighbor node ID, remaining degree value, remaining energy, speed component, link failure time, MAC delay, and Q value.

[0047] In one embodiment, the energy consumption model is: ; in, The average energy consumption generated by the drone swarm. For the first i The initial energy of the drone The total number of drones in the network. Energy loss for each data packet sent and received by the drone; The cost of a drone node is represented as follows: ; in, For the first i Energy consumption of a drone This refers to the energy loss during the propulsion of drones.

[0048] In this embodiment, the energy loss of the drone for each 127-byte data packet sent and received can be defined as follows: The propulsion energy loss of the drone is This invention focuses on the energy loss of drones during the routing process. The average energy consumption of the drone swarm during network operation is calculated as follows: .

[0049] For each drone, when designing the reward function for routing, the energy loss of a single drone at each step is considered, using the remaining energy of the drone node and its initial energy. The ratio of the cost of a node to its cost can be expressed as: .

[0050] In one embodiment, the delay model is: Data packets from the current UAV node i To neighboring UAV nodes j The total latency is: ; in, To start from the current UAV node i To neighboring UAV nodes j Media access control MAC latency, To start from the current UAV node i To neighboring UAV nodes j Queuing delay; ; in, The weighting coefficient indicates the difference between the current latency and the past latency. l The proportion of time delay, n For time length windows, t mac The MAC latency at the current moment can be defined as the time required for the MAC protocol to successfully send or discard a data packet. k ∈[ t - n , t -1]; ; in, t que Queuing delay for the current moment.

[0051] In this embodiment, UAVs need to consider not only the waiting time required for their own data packet forwarding, but also the MAC latency recorded in the neighbor information table. Data packets from nodes i To the node j The total latency is , The MAC delay can be estimated by the difference between the ACK message reception time and the data packet transmission time, and updated using the Window Mean Exponentially Weighted Moving Average (WMEWMA) method. Queuing delay can also be updated using the WMEWMA method.

[0052] The expiration time of the data packet can be defined as follows: Then the expiration time can be updated to Neighboring UAV nodes j When a data packet is received, if the data packet's expiration time is detected... If the value is less than 0, the data packet is discarded.

[0053] In one embodiment, the residual value is the number of edges connecting the neighboring UAV node to nodes outside the local topology range of the current UAV node, and the residual value filtering strategy is to filter out closed nodes with a residual value of 0.

[0054] In this embodiment, the residual value The meaning is neighboring UAV nodes j With the current UAV node i The number of edges connecting nodes outside the local topology range. Since neighboring nodes with a residual degree of 0 within a one-hop range have no routing forwarding value, the residual degree filtering strategy during routing decisions is to filter out closed nodes with a degree of 0 to avoid additional routing overhead and routing holes.

[0055] In one embodiment, the DTN carrying strategy is to select the node with the highest DTN value to forward or carry data packets, and the DTN value is determined as follows: ; in, For neighboring UAV nodes j speed, This represents the maximum speed (e.g., 60 m / s). The maximum distance in the scene, such as , For neighboring UAV nodes j The difference between the distance from the destination node at the current moment and the distance at the data packet expiration time; ; in,( x dst , y dst ) is the target location. t deadlineThis refers to the data packet expiration time. The destination location can refer to the location of the final destination node to which the data packet is to be delivered.

[0056] In this embodiment, the DTN carrying strategy can be to select the node with the highest DTN value to forward or carry data packets. When the remaining value of all neighboring UAV nodes... When a network partition is identified, the Data Transmission Network (DTN) value carried by each neighboring node is calculated.

[0057] The DTN calculation logic is as follows: it comprehensively considers key factors such as the current speed of neighboring UAV nodes, the current distance between neighboring UAV nodes and the destination node, the change in distance between neighboring UAV nodes and the destination node before the data packet expires, the maximum distance in the scenario, and the maximum speed. For example, DTN is calculated through node motion trajectory prediction and is divided into three cases: for neighboring UAV nodes that can reach the communication range of the destination node within the data packet expiration time, the DTN value is positively correlated with the speed of the neighboring UAV node; for neighboring UAV nodes moving towards the destination node, the DTN value is positively correlated with the proximity to the destination node; in other cases (such as nodes not moving towards the destination node), the DTN value of the neighboring UAV node is negative, indicating no forwarding value.

[0058] In one embodiment, the initial Q-value of the nodes within the communication range of the Q-learning decision framework is: ; in, Representative node i As the current UAV node This indicates the choice to send to neighboring UAV nodes. j Forward data packets, Indicates the current UAV node i In the state, send to neighboring UAV nodes j The Q value that can be obtained when forwarding data packets. d i,dst and d j,dst These represent the distance differences from the current UAV node and its neighboring UAV nodes to the target location, respectively; the Q value for nodes outside the communication range is -1. The reward function of the Q-learning decision framework for: ; in, Indicates and The corresponding neighboring UAV node, β The weights representing the balance between latency and energy consumption; r max This represents the maximum reward given to the link; r min This represents the maximum penalty imposed on the link.e It is a natural constant.

[0059] In this embodiment, a Q-learning routing and forwarding mechanism can be established. The initial Q value of nodes within the communication range is set to... The Q-value for nodes outside the communication range is -1. The reward function is designed as follows: ,in r max This indicates that the data packet is in a node. i Forwarded to node j The maximum reward given for a data packet to successfully reach its destination node is set as a positive incentive to encourage successful delivery of data packets to the destination node. r min If a node determines there is no valid path, the maximum penalty for this link is given, and a negative penalty is set to suppress temporary routing holes or high-speed link breaks caused by pole obstruction. The reward scheme for other cases can be composed of a weighted combination of inter-node transmission delay and remaining energy to balance delay and energy consumption optimization objectives.

[0060] When a data packet successfully reaches its destination node after being forwarded by the current UAV node to the selected next-hop neighbor UAV node, the forwarding link receives the set maximum reward value. When the current UAV node determines that there is no valid forwarding path and the data packet cannot be forwarded, the link receives the set maximum penalty value. Except for the above two cases, the reward value is determined by weight. β The latency and the remaining energy of the next-hop node are integrated for calculation, thereby providing feedback on the merits of different forwarding decisions.

[0061] When making routing decisions, the optimal forwarding node is selected based on the weighted Q value, and when partitioning the network, the node is selected based on the DTN.

[0062] The method of this invention can construct a joint optimization objective model including packet delivery rate, end-to-end latency, and node energy consumption, under constraints of communication transmission loss, flight energy consumption, and data packet expiration time, with efficient information sharing between nodes and next-hop action decision of data packets as optimization variables. The optimization objective model can be solved by weighted summation and multi-objective constraint algorithm. For each data packet sent by the node, an agent is constructed. Based on the reinforcement learning algorithm framework, the forwarding strategy of the node for data packets is optimized according to the neighbor information and the reward function. When a network partition is detected, the data packet carrying forwarding is adopted according to the value carried by the neighbor node's data packets, and offline routing decision is made comprehensively.

[0063] For example, Figure 2 This is a flowchart illustrating a reinforcement learning intelligent opportunistic routing method for ad hoc networks of power line inspection drones, as disclosed in an embodiment of the present invention. Figure 2As shown, based on the above architecture, the specific process of the UAV Intelligent Opportunity Routing Protocol (QIOR) based on reinforcement learning strategy forwarding is obtained as follows: First, initialize a scenario containing 30 UAVs and 1 base station, setting parameters such as a communication radius of 450m, a total data packet count of 1000, a Hello message period of 100ms, and a data packet expiration time of 300ms; then, in each time slot loop, each UAV sends and receives Hello messages, parses and calculates the neighbor node velocity component, relative velocity, link lifetime, and remaining degree value to update the neighbor information table; if there are data packets to be forwarded in the UAV buffer and the destination node is not the next hop, first execute the degree value filtering strategy to exclude closed neighbor nodes; if there are candidate neighbors, select the best forwarding node to forward the data packet according to the weighted Q value; if there are no candidate neighbors, determine the network partition and calculate the DTN value to select the forwarding node; if the destination node is the next hop, forward directly; finally, receive ACK messages, calculate latency, energy consumption, and reward, and update the Q value.

[0064] The QIOR protocol of this invention firstly uses a collaborative architecture of UAV swarm combined with inspection ground control base stations. Taking into account high dynamic mobility characteristics, node energy constraints, and channel transmission loss characteristics, the next-hop node selection is used as the core optimization variable to construct a four-dimensional joint optimization model that integrates latency, energy consumption, delivery rate, and remaining energy. Taking a single data packet as an independent intelligent agent, a state-action-reward mapping mechanism is designed based on a reinforcement learning framework. Combined with the weighted criterion of remaining degree value and DTN value evaluation, the routing policy is updated online adaptively.

[0065] During implementation, the neighbor link status of the UAV is perceived in real time through periodic Hello messages. The range of candidate forwarding nodes is dynamically narrowed by filtering with residual values. In network partitioning scenarios, the routing space is expanded by carrying policies through the DTN. Finally, the optimal next-hop decision is output by iteratively updating the Q-value. Simulation results show that compared with the traditional Greedy Perimeter Stateless Routing (GPSR) protocol, the QIOR protocol improves the packet delivery rate by 20%. Compared with the existing Q-learning based Multi-objectiveoptimization Routing (QMR) protocol, the end-to-end average latency is reduced by 10%-25%. Even in a high-dynamic scenario with a maximum node speed of 60m / s, it can still maintain a 65% delivery rate, verifying its excellent performance in high-dynamic inspection environments.

[0066] Based on the technical solutions of the above embodiments, this invention provides several specific implementation methods.

[0067] As a specific implementation of this invention, a simulation experiment was conducted using a Python-based DroNet simulator. The invention was also compared with traditional algorithms to verify its beneficial effects.

[0068] Figure 3 A comparison chart of data packet delivery rates between the present invention and existing technologies is provided for embodiments of the present invention, such as... Figure 3 As shown, the performance of the embodiments of the present invention, the existing mainstream solutions QMR and GPSR, and the proposed algorithm are compared under different packet transmission intervals. As the packet transmission interval increases, the packet delivery rate of all three protocols shows an upward trend, but the QIOR protocol consistently maintains the highest rate. This is because the QIOR protocol employs a path selection optimization strategy designed for dynamic topologies. It considers both packet forwarding in network partition scenarios and predicts link lifetime based on node movement trajectories to avoid failed routes, thereby effectively improving the success rate. Its packet delivery rate is consistently higher than that of the QMR algorithm.

[0069] Figure 4 An end-to-end latency comparison diagram between the present invention and the prior art is provided for an embodiment of the present invention, such as... Figure 4 As shown, in terms of average end-to-end latency, the QIOR protocol has a significantly lower end-to-end latency than the QMR protocol. This is due to its neighbor node filtering strategy based on residual degree values, which can eliminate redundant forwarding nodes to reduce routing overhead. While the GPSR protocol has the lowest end-to-end latency among successfully delivered packets, its overall transmission efficiency is insufficient due to its low delivery rate. In a dynamic topology scenario with a maximum speed of 15 m / s for fixed drone nodes, the QIOR protocol outperforms the QMR protocol by 10% in both packet delivery success rate and end-to-end latency.

[0070] Figure 5 A comparison chart of the maximum speed and data packet delivery rate of UAVs of the present invention and prior art is provided for embodiments of the present invention, as shown in the figure. Figure 5 As shown, the performance of the proposed algorithm compared with existing mainstream solutions QMR and GPSR under different maximum drone speeds is as follows: With the increase in the maximum drone speed, the high dynamism of the ad hoc network becomes more apparent, and the packet delivery rate of all three protocols decreases. However, the QIOR protocol shows the smallest decrease, at only 32.1%. This is because the QIOR protocol addresses the problem of frequent network partitioning in highly dynamic environments by designing a packet-carrying forwarding strategy, effectively mitigating the impact of route path failures on the delivery rate. In contrast, the QMR protocol's packet delivery rate decreases by 52.6%, while the GPSR protocol, due to its susceptibility to routing holes, exhibits the lowest delivery rate and the most significant downward trend. Figure 6 A comparison chart of the maximum speed-average end-to-end delay of the UAV of the present invention and the prior art is provided for embodiments of the present invention, as shown in the figure. Figure 6 As shown, the end-to-end latency of all three protocols shows an increasing trend. The QIOR protocol shows the slowest increase, at only 50.7%, while the QMR protocol increases by 60.2%. The GPSR protocol, due to its susceptibility to routing holes, experiences a continuous increase in end-to-end latency as node mobility increases. This further validates the advantages of the QIOR protocol in highly dynamic environments. With its packet-carrying forwarding strategy for partitioned scenarios, it effectively controls the growth rate of end-to-end latency while maintaining a high packet delivery rate, demonstrating a significant advantage in overall routing performance.

[0071] This invention relies on degree value filtering and Q-learning mechanisms to dynamically update routing strategies without requiring global topology information. In typical high-dynamic scenarios such as simulated UAVs inspecting power transmission line channels and flying around towers, its performance degradation is better than that of QMR and GPSR algorithms. It is highly adaptable and suitable for scenarios such as high-dynamic UAV swarm networks and UAV self-organizing network power inspection. It can effectively solve problems such as routing failure and high latency under dynamic topology, and improve the overall system stability and scalability.

[0072] Example 2 Figure 7 This is a schematic diagram of a reinforcement learning intelligent opportunistic routing system for power inspection drone ad hoc networks provided in Embodiment 2 of the present invention. The system is applicable to the situation of determining forwarding strategies for drone nodes in drone ad hoc networks. It is applied to drone ad hoc networks FANET, which includes multiple randomly distributed drone UAV nodes and fixed base stations.

[0073] like Figure 7 As shown, the system includes: The neighbor interaction module 210 is used for each current UAV node to update the neighbor information table based on the probe messages of neighboring UAV nodes; Network modeling module 220 is used to select one or more neighboring UAV nodes as candidate forwarding nodes from the neighbor information table based on the latency model, energy consumption model and remaining degree value filtering strategy when the current UAV node sends a data packet; The routing decision module 230 is used to select the node with the highest action value Q from all candidate forwarding nodes as the target forwarding node when there are candidate forwarding nodes, in combination with the Q learning decision framework; otherwise, it selects the node with the highest DTN value from the current UAV node and the neighboring UAV nodes as the target forwarding node based on the DTN carrying strategy. The performance optimization module 240 is used to calculate a reward value based on the acknowledgment response ACK message received after forwarding the data packet from the current UAV node to the target forwarding node, and update the Q value based on the reward value. The Q-learning decision framework includes a state space, an action space, and a value evaluation mechanism. The state space is defined as the remaining energy of the current UAV node, the state of neighboring UAV nodes, and the link state. The action space is defined as the selection of the next-hop node, including the data packets carried by the node itself. The value evaluation mechanism is implemented through the Q value.

[0074] This embodiment provides a reinforcement learning intelligent opportunistic routing system for self-organizing networks of power line inspection UAVs, including: a neighbor interaction module, used by each current UAV node to update its neighbor information table based on probe messages from neighboring UAV nodes; a network modeling module, used to select one or more neighboring UAV nodes as candidate forwarding nodes from the neighbor information table when the current UAV node sends a data packet, based on a latency model, energy consumption model, and a residual value filtering strategy; and a routing decision module, used to select the node with the highest action value Q from all candidate forwarding nodes as the target forwarding node when candidate forwarding nodes exist, using a Q-learning decision framework; otherwise, based on... The DTN carrying strategy selects the node with the highest DTN value from the current UAV node and the neighboring UAV nodes as the target forwarding node; the performance optimization module is used to calculate the reward value based on the acknowledgment response (ACK) message received after forwarding the data packet from the current UAV node to the target forwarding node, and update the Q value based on the reward value; wherein, the Q learning decision framework includes a state space, an action space, and a value evaluation mechanism, the state space is defined as the remaining energy of the current UAV node, the state of the neighboring UAV nodes, and the link state; the action space is defined as the next-hop node selection, including the data packet carried by itself; the value evaluation mechanism is implemented through the Q value.

[0075] Furthermore, each current UAV node periodically sends probe messages to achieve neighbor discovery. The probe message format includes the current UAV node's node identifier, speed, degree value, remaining energy, current location, and the location of the next target point. Correspondingly, the neighbor interaction module 210 is specifically used for: When the current UAV node receives a probe message from a neighboring UAV node, the current UAV node determines the velocity component of the neighboring UAV node based on the neighboring UAV node's current position, velocity, and the position of the next target point: ; in, i For the current UAV node, j For the neighboring UAV nodes of the UAV node, Indicates neighboring UAV nodes j The velocity vector of movement, , These represent neighboring UAV nodes. j exist x andy The velocity vector in the direction of movement; , Neighboring UAV nodes j exist x and y Displacement components in the direction, , , ( x j , y j ) is the neighboring UAV node j The current position coordinates, ( x j ’ , y j ’ ) is the neighboring UAV node j The coordinates of the next target point; Based on the velocity components of the current UAV node itself and the velocity components of neighboring UAV nodes, determine the relative velocity components of the neighboring UAV nodes relative to the current UAV node: ; in, , These represent the current UAV nodes. i exist x and y The velocity vector in the direction of movement. , These represent the relative velocity components at... x and y The velocity vector in the direction of movement; Based on the relative velocity components, the current position of the current UAV node, and the current positions of neighboring UAV nodes, determine the link lifetime between the current UAV node and its neighboring UAV nodes. : ; in, For link l Survival time, ( x i , y i ) represents the current UAV node i The current position coordinates, R c For drone nodes i and j Communication radius between them t Indicates time; Based on the link lifetime and the arrival time of the probe packets from the neighboring UAV node, the failure time of the link of the neighboring UAV node is determined; The neighbor information table is updated based on the failure time, the velocity components of the neighbor UAV nodes, and the probe messages.

[0076] Furthermore, the energy consumption model is as follows: ; in, The average energy consumption generated by the drone swarm. For the first i The initial energy of the drone The total number of drones in the network. Energy loss for each data packet sent and received by the drone; The cost of a drone node is represented as follows: ; in, For the first i Energy consumption of a drone This refers to the energy loss during the propulsion of drones.

[0077] Furthermore, the delay model is as follows: Data packets from the current UAV node i To neighboring UAV nodes j The total latency is: ; in, To start from the current UAV node i To neighboring UAV nodes j Media access control MAC latency, To start from the current UAV node i To neighboring UAV nodes j Queuing delay; ; in, These are the weighting coefficients. n For time length windows, t mac The MAC latency at the current moment. k ∈[ t - n , t -1]; ; in, t que Queuing delay for the current moment.

[0078] Furthermore, the residual value is the number of edges connecting the neighboring UAV node to nodes outside the local topology range of the current UAV node, and the residual value filtering strategy is to filter out closed nodes with a residual value of 0.

[0079] Furthermore, the DTN carrying strategy involves selecting the node with the highest DTN value to forward or carry data packets. The DTN value is determined as follows: ; in, For neighboring UAV nodes j speed, This represents the maximum speed. The maximum distance in the scene. For neighboring UAV nodes j The difference between the distance from the destination node at the current moment and the distance at the data packet expiration time; ; in,( x dst , y dst ) is the target location. t deadline This is the data packet expiration time.

[0080] Furthermore, the initial Q-values ​​of nodes within the communication range of the Q-learning decision framework are: ; in, Representative node i As the current UAV node This indicates the choice to send to neighboring UAV nodes. j Forward data packets, Indicates the current UAV node i In the state, send to neighboring UAV nodes j The Q value that can be obtained when forwarding data packets. d i,dst and d j,dst These represent the distance differences from the current UAV node and its neighboring UAV nodes to the target location, respectively; the Q value for nodes outside the communication range is -1. The reward function of the Q-learning decision framework for: ; in, Indicates and The corresponding neighboring UAV node, β The weights representing the balance between latency and energy consumption; r max This represents the maximum reward given to the link;r min This represents the maximum penalty imposed on the link. e It is a natural constant.

[0081] The reinforcement learning intelligent opportunistic routing system for power inspection drone ad hoc networks described above can execute the reinforcement learning intelligent opportunistic routing method for power inspection drone ad hoc networks provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the method execution.

[0082] Example 3 Figure 8 A schematic diagram of an electronic device that can be used to implement embodiments of the present invention is shown. The electronic device 10 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0083] like Figure 8 As shown, the electronic device 10 includes at least one processor 11 and a memory communicatively connected to the at least one processor 11. The memory can be read-only memory (ROM) and / or random access memory (RAM), such as ROM 12 and RAM 13. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in ROM 12 or loaded into RAM 13 from storage unit 18. RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An I / O interface 15 with input / output (I / O) functionality is also connected to the bus 14.

[0084] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0085] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as the reinforcement learning intelligent opportunistic routing method for power line inspection drone ad hoc networks.

[0086] In some embodiments, the reinforcement learning intelligent opportunistic routing method for power line inspection drone ad hoc networks can be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the reinforcement learning intelligent opportunistic routing method for power line inspection drone ad hoc networks described above can be performed. Alternatively, in other embodiments, processor 11 can be configured to execute the reinforcement learning intelligent opportunistic routing method for power line inspection drone ad hoc networks by any other suitable means (e.g., by means of firmware).

[0087] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0088] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0089] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory, read-only memory, erasable programmable read-only memory (EPROM) or flash memory, optical fibers, compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0090] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device for displaying information to the user, such as a cathode ray tube (CRT) or liquid crystal display (LCD) monitor; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0091] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0092] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system. It addresses the shortcomings of traditional physical hosts and Virtual Private Server (VPS) services, such as high management difficulty and weak business scalability.

[0093] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0094] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A reinforcement learning intelligent opportunistic routing method for power inspection unmanned aerial vehicle ad hoc network, characterized in that, Applied to a self-organizing unmanned aerial vehicle (UAV) network (FANET), the FANET comprising multiple randomly distributed UAV nodes and fixed base stations, the method includes: Each current UAV node updates its neighbor information table based on the probe messages from its neighboring UAV nodes; When the current UAV node sends a data packet, it selects one or more neighboring UAV nodes as candidate forwarding nodes from the neighbor information table based on the latency model, energy consumption model, and remaining degree value filtering strategy. When candidate forwarding nodes exist, the node with the highest action value Q is selected from all candidate forwarding nodes using the Q-learning decision framework as the target forwarding node; otherwise, the node with the highest DTN value is selected from the current UAV node and the neighboring UAV nodes as the target forwarding node based on the DTN carrying strategy. After the current UAV node forwards the data packet to the target forwarding node, the reward value is calculated based on the acknowledgment (ACK) message received after forwarding, and the Q value is updated based on the reward value. The Q-learning decision framework includes a state space, an action space, and a value evaluation mechanism. The state space is defined as the remaining energy of the current UAV node, the state of neighboring UAV nodes, and the link state. The action space is defined as the selection of the next-hop node, including the data packets carried by the node itself. The value evaluation mechanism is implemented through the Q value.

2. The method of claim 1, wherein, Each current UAV node periodically sends probe messages to achieve neighbor discovery. The probe message format includes the current UAV node's node identifier, velocity, degree value, remaining energy, current location, and the location of the next target point. Correspondingly, each current UAV node updates its neighbor information table based on the probe messages from neighboring UAV nodes, including: When the current UAV node receives a probe message from a neighboring UAV node, the current UAV node determines the velocity component of the neighboring UAV node based on the neighboring UAV node's current position, velocity, and the position of the next target point: ; wherein, i is a current UAV node, j is a neighbor UAV node of the UAV node, denotes a moving speed vector of the neighbor UAV node j , , denote moving speed vectors of the neighbor UAV node j in x and y directions, respectively; , are displacement components of the neighbor UAV node j in x and y directions, , , ( x j , y j ) are current position coordinates of the neighbor UAV node j , x j ’ , y j ’ are next target point position coordinates of the neighbor UAV node j . Based on the velocity components of the current UAV node itself and the velocity components of neighboring UAV nodes, determine the relative velocity components of the neighboring UAV nodes relative to the current UAV node: ; in, , These represent the current UAV nodes. i exist x and y The velocity vector in the direction of movement. , These represent the relative velocity components at... x and y The velocity vector in the direction of movement; Based on the relative velocity components, the current position of the current UAV node, and the current positions of neighboring UAV nodes, determine the link lifetime between the current UAV node and its neighboring UAV nodes. : ; in, For link l Survival time, ( x i , y i ) represents the current UAV node i The current position coordinates, R c For drone nodes i and j Communication radius between them t Indicates time; Based on the link lifetime and the arrival time of the probe packets from the neighboring UAV node, the failure time of the link of the neighboring UAV node is determined; The neighbor information table is updated based on the failure time, the velocity components of the neighbor UAV nodes, and the probe messages.

3. The method according to claim 1, characterized in that, The energy consumption model is as follows: ; in, The average energy consumption generated by the drone swarm. For the first i The initial energy of the drone The total number of drones in the network. Energy loss for each data packet sent and received by the drone; The cost of a drone node is represented as follows: ; in, For the first i Energy consumption of a drone This refers to the energy loss during the propulsion of drones.

4. The method according to claim 1, characterized in that, The time delay model is as follows: Data packets from the current UAV node i To neighboring UAV nodes j The total latency is: ; in, To start from the current UAV node i To neighboring UAV nodes j Media access control MAC latency, To start from the current UAV node i To neighboring UAV nodes j Queuing delay; ; in, These are the weighting coefficients. n For time length windows, t mac The MAC latency at the current moment. k ∈[ t - n , t -1]; ; in, t que Queuing delay for the current moment.

5. The method according to any one of claims 1-4, wherein the residual value is the number of edges connecting the neighboring UAV node to nodes outside the local topology range of the current UAV node, and the residual value filtering strategy is to filter out closed nodes with a residual value of 0.

6. The method according to claim 1, wherein the DTN carrying strategy is to select the node with the highest DTN value to forward or carry data packets, and the DTN value is determined as follows: ; in, For neighboring UAV nodes j speed, This represents the maximum speed. The maximum distance in the scene. For neighboring UAV nodes j The difference between the distance from the destination node at the current moment and the distance at the data packet expiration time; ; in,( x dst , y dst ) is the target location. t deadline This is the data packet expiration time.

7. The method according to claim 1, wherein the initial Q-value of the nodes within the communication range of the Q-learning decision framework is: ; in, Representative node i As the current UAV node This indicates the choice to send to neighboring UAV nodes. j Forward data packets, Indicates the current UAV node i In the state, send to neighboring UAV nodes j The Q value that can be obtained when forwarding data packets. d i,dst and d j,dst These represent the distance differences from the current UAV node and its neighboring UAV nodes to the target location, respectively; the Q value for nodes outside the communication range is -1. The reward function of the Q-learning decision framework for: ; in, Indicates and The corresponding neighboring UAV node, β The weights representing the balance between latency and energy consumption; r max This represents the maximum reward given to the link; r min This represents the maximum penalty imposed on the link. e It is a natural constant.

8. A reinforcement learning intelligent opportunistic routing system for self-organizing networks of power line inspection drones, characterized in that, The system is applied to a self-organizing network for unmanned aerial vehicles (UAVs) called a FANET, which includes multiple randomly distributed UAV nodes and fixed base stations. The system comprises: The neighbor interaction module is used by each current UAV node to update the neighbor information table based on the probe messages of neighboring UAV nodes; The network modeling module is used to select one or more neighboring UAV nodes as candidate forwarding nodes from the neighbor information table based on the latency model, energy consumption model and remaining degree value filtering strategy when the current UAV node sends a data packet. The routing decision module is used to select the node with the highest action value Q from all candidate forwarding nodes as the target forwarding node when candidate forwarding nodes exist, using the Q-learning decision framework; otherwise, it selects the node with the highest DTN value from the current UAV node and the neighboring UAV nodes as the target forwarding node based on the DTN carrying strategy. The performance optimization module is used to calculate the reward value based on the ACK message received after forwarding the data packet from the current UAV node to the target forwarding node, and update the Q value based on the reward value. The Q-learning decision framework includes a state space, an action space, and a value evaluation mechanism. The state space is defined as the remaining energy of the current UAV node, the state of neighboring UAV nodes, and the link state. The action space is defined as the selection of the next-hop node, including the data packets carried by the node itself. The value evaluation mechanism is implemented through the Q value.

9. An electronic device, characterized in that, The device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to execute the reinforcement learning intelligent opportunistic routing method for power line inspection UAV ad hoc networks as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed by a processor, implement the reinforcement learning intelligent opportunistic routing method for self-organizing networks of power line inspection drones as described in any one of claims 1-7.