Unmanned aerial vehicle ad hoc network link maintenance method based on GNN and DQN

Abstract

The invention relates to the technical field of unmanned aerial vehicle communication, in particular to an unmanned aerial vehicle ad hoc network link maintenance method based on GNN and DQN, and the method comprises the steps: collecting the sending / receiving time interval, link quality, link life prediction value and the like of the unmanned aerial vehicle ad hoc network and a neighbor in real time; constructing a space-time dynamic graph as the input of a graph neural network GNN model; outputting values of a deep Q network corresponding to a group of specific Hello data packet sending period actions; selecting an optimal action and immediately executing the action in the environment; updating parameters through multiple times of training; and copying a small batch of samples randomly extracted from the buffer area into the target network at every iteration to calculate a target value, sending updated model parameters to the main network, and updating the cycle of each node to adjust the model parameters. The problems of Hello data packet redundancy, link failure discovery lagging, network energy consumption sharp increase and the like caused by severe topology change in a high-maneuverability environment are solved.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of unmanned aerial vehicle communication, and particularly relates to a method for maintaining a link of an unmanned aerial vehicle ad hoc network based on GNN and DQN. BACKGROUND

[0002] In recent years, small unmanned aerial vehicles (UAVs) have shown exponential growth in onboard computing, communication payloads, and energy density. They have evolved from early single-aircraft aerial photography tools to cooperative "air-edge nodes" and have been widely deployed in tasks such as battlefield reconnaissance, disaster relief, logistics delivery, and wide-area Internet of Things. To break away from the dependence on ground infrastructure, the academic and industrial communities have proposed a new networking paradigm called "flying ad hoc network (FANET)". Through self-organization, multi-hop forwarding, and dynamic topology, FANET provides a wide coverage, low latency, and high reliability air communication platform for UAV clusters.

[0003] In the application environment of high-speed movement and dynamic topology change, the connectivity of the flying node is the focus of FANET research. Traditional table-driven routing protocols have limitations, and the pre-established routes may become invalid soon. Therefore, UAV nodes need to maintain connectivity in a constantly changing environment. UAVs in the FANET network usually maintain links by broadcasting Hello packets. However, FANET has exposed the following key bottlenecks in application scenarios with high-speed maneuvering, dramatic changes in node density, and frequent task switching:

[0004] High dynamicity of topology: In typical tasks, the relative speed of UAVs can reach 200-300 m / s, and the lifetime of a single-hop link is often shorter than 1 s. The traditional Hello discovery mechanism based on fixed cycles cannot complete neighbor updates before the link breaks, leading to invalid routing tables, packet retransmission, and a decrease in overall throughput.

[0005] Energy consumption imbalance between control and data: If the Hello broadcast period is set too short, the neighbor discovery probability can be improved, but the control signaling ratio will soar, directly compressing the bandwidth of task data; if it is set too long, a large number of retransmissions will be caused due to disconnection, and energy waste will be aggravated.

[0006] Insufficient degree of neighbor state judgment: Existing protocols are too rough in the characterization of neighbor states, only counting the number of neighbors or judging whether the distance is less than a certain fixed threshold, without fully utilizing the relative motion speed of neighbors, the decay rate of link quality over time, and the topology redundancy information within two hops. This simplified state description method makes the routing decision lag behind the topology change, leading to mismatched resource allocation.

[0007] Poor scalability: When the number of UAV nodes expands to more than 50, centralized or semi-distributed algorithms face an explosion of state space, and the convergence time increases exponentially, making it difficult to meet real-time task requirements.

[0008] In summary, achieving "low-energy, high-real-time, and topology-predictable" UAV link maintenance in complex three-dimensional spatial domains has become a core challenge restricting the large-scale commercial application of FANET. While existing research has attempted adaptive Hello, power control, or reinforcement learning optimization, it has failed to simultaneously balance topology prediction accuracy, algorithm convergence speed, and energy consumption constraints. Therefore, a new method is urgently needed that can explicitly model neighbor topology relationships, perceive link evolution trends in real time, and jointly adapt the Hello period and transmit power. Summary of the Invention

[0009] To address the contradiction between latency and energy consumption in existing FANET link discovery technologies, such as Figure 1 As shown, the purpose of this invention is to provide a method for maintaining the link of unmanned aerial vehicles (UAVs) based on graph neural networks (GNN) and deep Q networks (DQN) to solve problems such as Hello packet redundancy, delayed link failure detection, and surge in network energy consumption caused by drastic topology changes in high-mobility environments.

[0010] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0011] A UAV link maintenance method based on GNN and DQN is considered, which involves a... The FANET, composed of high-speed UAVs, operates at maximum speed in three-dimensional space. In the relative flight phase, neighbor discovery is achieved by broadcasting Hello packets via wireless links with adjustable transmit power. Channel fading follows a Rayleigh distribution, and the link size is determined by distance and a signal-to-interference-plus-noise ratio (SINR) threshold. Each UAV records its own position, velocity, remaining energy, and neighbor information in real time, and adjusts the Hello packet transmission period jointly by GNN-DQN to maintain dynamic topology connectivity with low energy consumption. The DQN model introduces a master network, an experience replay buffer, and a target network. The master network generates training experience through continuous interaction with the environment and accepts the current network state s as input, outputting a prediction of each possible action. The experience replay buffer is used to update and store training experience in real time, while randomly sampling a portion of samples for training; the target network is used to periodically copy parameters from the main network and update them, and to calculate the target. The value and the convergence speed of the algorithm are improved through gradient training.

[0012] A method for maintaining unmanned aerial vehicle (UAV) links based on graph neural networks (GNNs) and deep Q-networks (DQNs) includes:

[0013] S100, in the environment, each UAV node periodically broadcasts Hello packets and listens to neighbor Hello packets, and collects the transmission time interval, the reception time interval, the number of neighbors, the residual energy, the speed, the link quality and the link lifetime prediction value of itself and neighbors in real time;

[0014] S200, in the deep Q network, each UAV encapsulates the collected information as node features and edge features, and constructs a spatio-temporal dynamic graph as the network state and as the input of the graph neural network (GNN) model;

[0015] S300, in the deep Q network, the GNN model integrates the node features and the edge features and presents them in the form of a graph, aggregates the global hidden state through multiple rounds of iteration by running an iterative message passing algorithm between the hidden states of the links, and outputs the value corresponding to a specific Hello packet transmission period action to be evaluated;

[0016] S400, the UAV selects the optimal action from the value according to the greedy strategy and executes the action immediately in the environment, if the action is successfully executed, that is, the selected Hello broadcast period adjustment amount is sufficient to meet the maintenance needs of the link, a normalized reward value is returned, and the network state is updated ; if the action cannot be completed, the training is ended;

[0017] S500, after multiple training, the model parameters are constantly updated according to the network state , the action , and the reward value , and the training experience generated by the main network is stored in the experience replay buffer of the deep Q network;

[0018] S600, every iteration, a small batch of samples randomly extracted from the buffer are copied to the target network for calculating the target value, the network parameters in the target network are constantly updated, and the updated model parameters are sent to each node in the main network, when the main network receives the updated parameters, each node updates its period adjustment model parameters; repeat this step until convergence.

[0019] Preferably, S100 comprises:

[0020] S101, the information collection step is: all UAV nodes periodically generate and broadcast Hello packets;

[0021] ​The data packet fields include the drone node's ID, 3D coordinates, remaining energy, current Hello cycle, link quality, neighbor list, and predicted link lifetime.

[0022] S102. The methods for assessing the remaining lifetime of the link include:

[0023] Signal strength trend acquisition: UAV In continuous Record neighboring drones within each Hello cycle. The received signal strength indicator or SINR is used to form a time series. ;

[0024] Threshold extrapolation: Setting a link failure threshold Linear extrapolation or exponential regression is used to... Perform fitting to estimate the signal strength when it first falls below... Expected time ;

[0025] Lifespan calculation: With current time By subtracting the values, we can obtain the remaining lifetime of the link. If in If a fit cannot be achieved, then the default is used. Minimum value (preset) .

[0026] Preferably, the GNN model in S200 includes three core layers: a feature aggregation layer, a state update layer, and an output decision layer.

[0027] S201. In the feature aggregation layer, each node receives state information from its neighboring nodes and generates a preliminary fusion message through an aggregation function.

[0028] In the feature aggregation layer, for any node in the network, the feature information of its neighboring nodes is aggregated to generate a message vector, and the specific formula is as follows: ;

[0029] in, Represents a node In the The messages aggregated by the layer Represents the set of its neighboring nodes. This represents the characteristic state of the neighboring node in the previous layer. This represents a predefined aggregate function;

[0030] S202. In the state update layer, the aggregated message is combined with the node's current state, and the node state is updated through a neural network. The specific formula is as follows: ;

[0031] in, and These represent the trainable parameters of the layer, It is a non-linear activation function. This represents a vector concatenation operation;

[0032] go through In the next iteration, the hidden states of each link are aggregated again to obtain the final hidden state, which is then entered into the output decision layer and read using the function. Calculate the node state based on the global hidden state graph: ;

[0033] in, This indicates the network topology.

[0034] Preferably, S300 includes:

[0035] According to the formula: In state Follow the strategy Execute action The return estimate value;

[0036] Among them, the policy function It is in state Select action The conditional probability distribution at time, the policy in The goal at any given moment is to maximize the rewards accumulated from historical experience. , It is a discount factor that weighs the reward data at the current time step. For a certain time interval, In order to take Expected value when using a strategy.

[0037] Preferably, S600 includes:

[0038] S601, Target The value is the reward value at the current time step. Maximum prediction in the next state The sum of the decayed values ​​is calculated using the following formula: ;

[0039] in, Represents the parameters of the neural network. This represents the discount factor that weighs historical and current reward data. Indicates the next state. Indicates the action to be chosen in the next state;

[0040] The loss value is calculated using the mean squared error between the predicted value and the target value, and is used to calibrate the neural network parameters. The specific formula is as follows: ;

[0041] Where E represents the expected value;

[0042] S602. The model is continuously trained using gradient descent, causing the model loss value to decrease continuously. The gradient calculation formula is: ;

[0043] in, To determine the strategy to adopt Expected value This represents the natural logarithm of the probability when using this strategy, where k is a certain time interval and T is the training period. and This represents the status and reward within that time interval;

[0044] At the same time, continuously update network parameters: ;

[0045] in, The learning rate determines the speed at which parameters are updated. It is the gradient update value weighting factor, which is dynamically calculated based on the discount return.

[0046] After the parameters of the target network are updated at each stage, the updated model parameters are sent to each node in the main network. When the main network receives the updated parameters, each node updates its periodic adjustment model parameters. This process is repeated until convergence, that is, the predefined standard is reached.

[0047] Compared with the prior art, the beneficial effects achieved by the present invention are:

[0048] This invention achieves real-time perception of the UAV network by mapping a UAV cluster to a spatiotemporal dynamic graph. Through the dynamic update capability of the GNN, the algorithm can capture the continuous changes in network topology characteristics, providing accurate state input for link maintenance. By using DQN technology to learn the mapping relationship between network state and actions, intelligent adjustment of link maintenance strategies is achieved. This invention uses the scalar L obtained from "link remaining lifetime assessment" directly as the "link lifetime" dimension in the edge features. Its magnitude reflects the urgency of link failure within the next 1-2 Hello cycles, realizing the construction of a "predictive" graph. Attached Figure Description

[0049] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0050] Figure 1 This is the framework of the link maintenance method based on GNN and DQN of this invention;

[0051] Figure 2 This is a UAV cluster scenario diagram of the present invention;

[0052] ;

[0053] ;

[0054] picture;

[0055] picture. Detailed Implementation

[0056] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0057] Please see Figures 1-6 The present invention provides the following technical solution:

[0058] Example 1: System Model: Consider a system consisting of... A FANET consisting of UAVs is constructed. For the communication channel between UAVs, this invention adopts the Rayleigh channel model because the Rayleigh channel is a worst-case extreme special case of the Ricean channel, which better reflects the UAV swarm communication scenario. It is assumed that the network's transmission bandwidth is divided into... There are 3 orthogonal sub-channels, and the set of orthogonal channels is denoted as . Assume that UAVi is in sub-channel m ( Upward UAV Sending a signal, then in UAV at all times From UAV The received power can be expressed as ;

[0059] in, This indicates the transmit power of each UAV. It is the constant power gain factor introduced into amplifiers and antennas. This represents path loss. UAV and The distance between them This represents the path loss index. Let be the magnitude of the small-scale Rayleigh fading, a random variable whose envelope follows a Rayleigh distribution: ;

[0060] in, The power component coefficients representing non-direct paths.

[0061] Assume there is UAVs in sub-channel drones with communication capabilities The set of UAVs that generate interference is denoted as . Then the drone ( ) for drones In sub-channel The interference on can be represented as: ;

[0062] in, Sub-channel Up UAV With UAV The pairing relationship between them, if and Its corresponding value is ;like ,but .

[0063] Furthermore, drones In sub-channel SINR is represented as:

[0064] ;

[0065] in, This represents the variance of Gaussian noise. One of the purposes of link maintenance is to select the link with the highest SINR for data transmission to the UAV.

[0066] This invention discloses a UAV ad hoc network link maintenance method based on GNN and DQN (hereinafter referred to as the GNN-DQN model), the model architecture of which is as follows: Figure 1 As shown. This model uses DQN as the main framework for deep reinforcement learning, and vectorizes the attributes of nodes and edges in the UAV topology through GNN. For example... Figure 2 The image shown is a... The FANET, consisting of elevated mobile UAVs, uses circles to represent the communication range of UAV C. (In time slots) Inside, each UAV The position and velocity are respectively:

[0067] ;

[0068] ;

[0069] in, These represent the coordinates of UAVi in three directions. These represent the velocities in these three coordinate directions, respectively.

[0070] UAV With UAV The conditions for establishing a communication link between them are:

[0071] ;

[0072] in, In order to be in UAV at all times The set of adjacent edges, The minimum communication distance between two UAVs. This indicates the SINR threshold that needs to be met;

[0073] Action space This is a discrete set of actions, representing the UAV's instructions for adjusting the frequency of sending Hello data packets to its neighbors. Specifically, the action space... Actions in Defined as:

[0074] ;

[0075] Where 0 indicates that the current cycle remains unchanged. This indicates a reduction in the transmission cycle. A fixed periodic unit, Indicates an increase The action output by DQN in a fixed periodic unit. The transmission period applied to the UAV node : ;

[0076] This invention uses a GNN model to capture the relationships and local structure between nodes in a topological graph. First, to describe link quality, this invention defines an edge feature vector: ;

[0077] in, UAV With UAV The instantaneous distance between them This indicates the remaining lifetime of the link between them;

[0078] In UAV Maintain a 5-dimensional node feature vector: ; represent the sending interval, receiving interval, number of neighbors, remaining energy percentage, and speed modulus of the Hello data packet, respectively;

[0079] This invention constructs a spatiotemporal dynamic graph based on the above features: ;

[0080] in, and These represent the node feature matrix and the edge feature matrix, respectively.

[0081] Node feature matrix The column vectors are as follows:

[0082] ; ; ; ; ;

[0083] Among them, the 5 column vectors are UAV in order. The Hello data packet sending interval, receiving interval, number of neighbors, remaining energy, and speed modulus;

[0084] Edge feature matrix The column vectors are as follows:

[0085] ; ; ;

[0086] Construct an undirected adjacency matrix With communication radius With minimum SINR threshold For constraint, if and only if and hour ,otherwise .

[0087] state space It is a multidimensional feature vector that contains UAV The Hello data packet sending and receiving time intervals, number of neighbors, remaining energy, and velocity magnitude are all considered. These are then constructed into a state vector at each decision-making moment. : ;

[0088] The reward function represents the different immediate rewards resulting from adjustments at different periods. The model aims to consider both channel-level performance metrics and network load capacity: channel quality is characterized by throughput and packet delivery rate, and link stability is measured by the number of link handovers. Therefore, QoS metrics are incorporated into the value function.

[0089] (1) Throughput is to The total payload size of the Hello data packets received by the destination node at any given time:

[0090] ;

[0091] in, Represents average throughput. Iteration cycle time Inside, the Hello data packet received by the destination node, This represents the payload size of the Hello data packets. Directly applying throughput to the value function yields poor results and requires normalization. Here, throughput in the value function is defined as:

[0092] .

[0093] (2) Group submission rate:

[0094] The number of Hello packets received by the destination node divided by the number of data packets sent by the originating node is directly defined as the packet delivery rate in the reward function. P refers to the data packets received by the destination node within the iteration period T, while P is the data packet sent by the source node. .

[0095] (3) Number of link handovers:

[0096] For UAV With UAV The communication link between them defines two adjacent time slots. and The link state changes are as follows: ;

[0097] The total number of network link switching times is: ;

[0098] in, This is an indicator function; it is 1 if the link exists, and 0 otherwise. For UAV The set of adjacent edges at time t; N is the total number of UAVs.

[0099] Therefore, the reinforcement learning reward function is designed as follows: ;

[0100] in , , These are adjustable hyperparameters used to balance throughput, packet loss rate, and link stability, and satisfy the following conditions: .

[0101] The DQN module employs a dual-network collaborative mechanism, leveraging the training experience generated by the GNN. First, it is stored in the experience replay buffer, where s represents the network input state of DQN, a represents the action, and r represents the reward. This represents the updated network state. At fixed iteration intervals, the system extracts a small batch of samples without replacement from this memory unit and synchronizes only the gradient information of these samples to the independent target network to update the target value estimate. This design, through offline replay and asynchronous updates, reduces the temporal correlation between samples and avoids oscillations caused by real-time environment interactions, thereby significantly improving the stability of value estimation and the convergence efficiency of policy learning.

[0102] Example 2: Simulation Verification:

[0103] This invention employs the NS3-Gym platform and PyTorch for multi-scenario comparative experiments. The NS3 network simulator is widely used and powerful, capable of fully simulating the network experimental environment of this patent. The experimental simulation platform is as follows: Figure 1 As shown in Table 1:

[0104] Table 1 Simulation Parameters

[0105]

[0106] Figure 5 and Figure 6 This patent demonstrates the impact of UAV communication range on protocol performance, showing how adjusting the UAV's transmit power to 5-16km alters its communication range. Figure 5 The results demonstrate how throughput, packet loss rate, and the number of Hello packets vary with node speed for the Optimized Link State Routing Protocol (OLSR).

[0107] Among them, the circular line represents the scheme based on the traditional fixed link maintenance cycle, the triangular line represents the comparative scheme (OLSR-Mahmud), and the square line represents the performance of the scheme proposed in this invention (OLSR-GNN).

[0108] from Figure 6 This allows us to see the performance changes of the underlying protocol, Distance Vector Routing Protocol (AODV).

[0109] in Figure 6 The circular line represents the scheme based on the traditional fixed link maintenance cycle, the triangular line is the comparison scheme (AODV-Mahmud), and the square line represents the performance of the scheme proposed in this invention (AODV-GNN).

[0110] As shown in the two figures, the throughput of both OLSR and AODV protocols increases with the increase of communication range. When the communication range reaches a certain value, the throughput growth rate slows down, indicating that the communication range is no longer the sole factor determining protocol performance. The packet loss rate of both protocols decreases with increasing transmission distance. Although a certain packet loss rate still exists as the transmission distance continues to increase, the method proposed in this invention exhibits superior performance in both throughput and packet loss rate. By adjusting the Hello packet broadcast period using the method proposed in this invention, performance indicators such as network throughput, packet loss rate, and link maintenance overhead were tested, verifying the superiority of this invention. To study the application characteristics of this scheme in FANET, the protocol performance was verified by changing the node movement speed and data traffic volume in the tests.

[0111] This invention achieves different communication ranges by adjusting the signal transmission power of the UAV in the NS3; the higher the signal transmission power, the larger the communication range. For example... Figure 5 , 6 As shown, the horizontal axis represents the transmission range, which can actually be considered as the node's transmission power, because the higher the transmission power, the larger the node's communication range. Clearly, as the transmitted signal strengthens, the probability of data packet loss decreases; that is, the longer the transmission range, the lower the packet loss rate. The algorithm in the comparison scheme considers the UAV's speed and communication range. Experimental results show that when using the GNN-DQN algorithm, the number of Hello data packets decreases as the communication range expands, thereby improving energy efficiency. The method described in this invention also establishes a UAV channel model and considers SINR when defining the value function. As the communication range expands, the number of Hello data packets decreases, effectively reducing protocol overhead in highly dynamic environments.

[0112] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1.A GNN and DQN based unmanned aerial vehicle ad hoc network link maintenance method, characterized in that: The method comprises: S100, in the environment, each unmanned aerial vehicle node periodically broadcasts Hello data packets and listens to neighbor Hello data packets, and real-time collects sending time intervals, receiving time intervals, neighbor numbers, residual energies, speeds, link qualities and link lifetime prediction values of itself and neighbors; S200, in the deep Q network, each unmanned aerial vehicle encapsulates the collected information as node features and edge features, constructs a space-time dynamic graph as a network state and as an input of a graph neural network (GNN) model; S300, in the deep Q network, the GNN model integrates node features and edge features and presents in the form of a graph, through running an iterative message passing algorithm between the hidden states of the links, and after multiple rounds of iteration aggregation, outputs the global hidden state corresponding to a specific set of Hello data packet sending period action to be evaluated value ; S400、the unmanned aerial vehicle depends on The greedy strategy selects the optimal action from the value and immediately executes the action in the environment, if the action is successfully executed, that is, the selected Hello broadcast period adjustment amount is sufficient to meet the maintenance requirements of this link, a normalized reward value is returned , update the network state ; if the action cannot be completed, end this training; S500, after multiple training sessions, based on network status ,action Reward Value Continuously update model parameters and incorporate training experience generated by the main network. Stored in the experience replay buffer of the deep Q-network; S600, every In the next iteration, a small batch of randomly drawn samples from the buffer is copied into the target network for computation of the target. The system continuously updates the network parameters in the target network and sends the updated model parameters to each node in the main network. When the main network receives the updated parameters, each node updates its periodically adjusted model parameters. This process is repeated until convergence. 2.The method of claim 1, wherein, The S100 comprises: S101, the information collection step is that all unmanned aerial vehicle nodes periodically generate and broadcast Hello data packets; The data packet field comprises an ID, a three-dimensional coordinate, a residual energy, a current Hello period, a link quality, a neighbor list and a link lifetime prediction value of the unmanned aerial vehicle node; S102, the evaluation method of the link residual lifetime comprises: Signal strength trend collection: drone In successive Hello periods, record the received signal strength indication or SINR of neighbor drones , forming a time series ; Threshold extrapolation: Setting a link failure threshold Linear extrapolation or exponential regression is used to... Perform fitting to estimate the signal strength when it first falls below... Expected time ; Lifespan calculation: With current time By subtracting the values, we can obtain the remaining lifetime of the link. If in If a fit cannot be achieved, then the default is used. Minimum value (preset) . 3.The method of claim 1, wherein, The GNN model in the S200 comprises three core layers, namely a feature aggregation layer, a state updating layer and an output decision layer; S201, in the feature aggregation layer, each node receives state information from adjacent nodes, and generates a preliminary fusion message through an aggregation function; In the feature aggregation layer, for any node in the network, the feature information of its neighbor nodes is aggregated to generate a message vector, and the specific formula is: ; wherein, representing a node In a first messages aggregated at a layer, representing a set of its neighbor nodes, representing a feature state of a neighbor node at a previous layer, representing a predetermined aggregation function; In S202, the combined message is combined with the current state of the node itself in the state updating layer, and the state of the node is updated through a graph neural network, and the specific formula is as follows: ; wherein, and respectively represent trainable parameters of the layer, is a non-linear activation function, represents a vector concatenation operation; After each iteration, the final hidden state of each link is obtained by aggregating again, and the output decision layer is entered, using the read function Calculate the node state based on the global hidden state graph: ; wherein represents a network topology. 4.The method of claim 1, wherein, The S300 comprises: According to the formula: , in the state follows the policy performs the action the estimated value of the deep Q-network returned after ; Among them, the policy function It is in state Select action The conditional probability distribution at time, the policy in The goal at any given moment is to maximize the rewards accumulated from historical experience. , It is a discount factor that weighs the reward data at the current time step. For a certain time interval, In order to take Expected value when using a strategy. 5.The method of claim 1, wherein, The S600 comprises: S601、target value, i.e., the reward value at the current time step the maximum estimated value at the next state the sum of the values after decay, and the specific formula is: ; wherein, denotes a neural network parameter, denotes a discount factor that weighs historical and current reward data, denotes a next state, denotes an action selected at the next state; The loss value is calculated using the mean square error between the estimated value and the target value for calibrating the neural network parameters, and the specific formula is: ; Wherein, E represents an expected value; S602, continuously train the model using gradient descent, so that the model loss value continuously decreases, wherein the gradient calculation formula is: ; wherein, To obtain the expected value of the strategy , represents the natural logarithm of the probability of taking this action using the strategy, k is a certain time interval, T is the training period, and represent the state and reward in this time interval; Continuously updating network parameters: ; where, denotes the learning rate, which determines the speed of parameter update; is the gradient update value weighting factor, which is dynamically calculated according to the discounted return; After the parameter update of each stage target network, the updated model parameters are sent to each node in the main network; when the main network receives the updated parameters, each node updates the periodic adjustment model parameters; the process is repeated until convergence, that is, a predefined standard is reached.

Images