D3QN-Based Multi-Domain Joint Anti-Jamming Method and Device for UAV Ad Hoc Networks

Abstract

The present application discloses a multi-domain joint anti-interference method and device for a UAV self-organizing network based on D3QN, wherein the method comprises: inputting a preset number of historical channel selection data and historical channel perception data per unit time into an anti-interference model based on D3QN, then processing the historical channel selection data and historical channel perception data through the anti-interference model based on D3QN, outputting a Q value sequence, and finally determining the action corresponding to the maximum Q value in the Q value sequence as the target action. The present application can determine the optimal anti-interference action based on the D3QN anti-interference model, which can be used for data transmission in a UAV self-organizing network, and realize joint interference resistance of channel selection and transmission rate adaptation, thereby improving network throughput and enhancing the anti-interference performance of the UAV self-organizing network.

Description

Technical Field

[0001] The present application relates to the field of unmanned aerial vehicle (UAV) self-organizing networks, and in particular to a multi-domain joint anti-interference method and device for unmanned aerial vehicle (UAV) self-organizing networks based on D3QN. Background Art

[0002] As a highly decentralized network that operates independently of infrastructure, drone ad hoc networks can solve many dangerous and complex tasks, such as fire rescue and forest border patrols. As a key area of ad hoc networking technology, the development of drone ad hoc networks has brought significant convenience to modern life and technology.

[0003] With the application and development of swarm collaborative unmanned systems, the amount of service information in drone ad hoc networks will become increasingly rich, and the demand for secure and reliable swarm collaborative networks will also increase. Drone ad hoc networks are susceptible to malicious interference during application, which can lead to network errors, blockages, and even crashes.

[0004] Entering the 21st century, drone self-organizing networks have gradually expanded from military applications to civilian ones, often used in environments where manual intervention is difficult, such as border patrols, pesticide spraying, fire rescue, and other dangerous and complex tasks, greatly facilitating people's lives. With the development of artificial intelligence and machine learning, drone self-organizing networks, combined with AI, can effectively cope with the dynamic changes in drone scenarios and intelligently adopt different strategies to improve performance. Therefore, drone self-organizing networks combined with machine learning have become a hot research topic.

[0005] Deep reinforcement learning is a combination of deep learning (DL) and reinforcement learning (RL). The concept was first proposed by Deep Mind in 2013, along with the DQN (Deep Q-Network) algorithm. The DQN algorithm can be used to autonomously play Atari games using images as input and algorithmic learning. Because DQN can generate low-dimensional action outputs from high-dimensional state inputs, it has garnered widespread attention and has become a cutting-edge research direction in deep learning. It continues to develop and undergo numerous improvements.

[0006] The characteristics of drone ad hoc networks make them susceptible to interference. In a drone ad hoc network system based on TDMA communication, it is assumed that the interference source varies the interference power released on each channel at the beginning of each time slot. When a drone node transmits at a certain transmission rate on the channel, the SINR (Signal to Interference plus Noise Ratio) value at the receiving end must be greater than a specific threshold for the received data to be successfully decoded.

[0007] To sum up, how to solve the anti-interference problem of UAV self-organizing networks based on deep reinforcement learning methods is a technical problem that needs to be solved urgently. Summary of the Invention

[0008] The present application aims to solve one of the technical problems in the related art at least to a certain extent.

[0009] To this end, the purpose of this application is to solve the anti-interference problem of UAV self-organizing networks and propose a multi-domain joint anti-interference method for UAV self-organizing networks based on D3QN.

[0010] Another purpose of this application is to propose a multi-domain joint anti-interference device for drone self-organizing networks based on D3QN.

[0011] To achieve the above objectives, this application proposes a multi-domain joint anti-interference method for UAV self-organizing networks based on D3QN, including:

[0012] Inputting a preset number of historical channel selection data and historical channel perception data per unit time into the D3QN-based anti-interference model;

[0013] Processing the historical channel selection data and the historical channel perception data through the D3QN-based anti-interference model to output a Q value sequence;

[0014] The action corresponding to the maximum Q value in the Q value sequence is determined as the target action.

[0015] In one possible implementation, the D3QN-based anti-interference model includes the following loss function:

[0016] L(θ)=E[(yQ(st,at;θi)2]

[0017] Among them, Q(s t ,a t θ i ) represents the current state s t 、Current moment action a t and the model parameters θ of the i-th iteration iWhen is input, the Q value of the Q-Network under the model parameters of the i-th iteration is evaluated, and y represents the target value calculated by the target Q-Network under the model parameters of the i-th iteration.

[0018] In a possible implementation, the calculation formula of y is as follows:

[0019]

[0020] Among them, r t represents the feedback at the current moment, γ represents the attenuation coefficient, s t+1 represents the state at the next moment, θ - represents the model parameters of the i-1th iteration, Indicates the action with the largest Q value evaluated by the Q-Network.

[0021] In a possible embodiment, the r t The calculation formula is as follows:

[0022]

[0023] Among them, λ and μ represent the adjustment parameters, It represents the ratio of transmission power to the maximum supported rate, and J represents the loss value when transmission fails due to interference.

[0024] In a possible implementation, the Q value is calculated as follows:

[0025]

[0026] Among them, V(s; θ V ) represents the state value function, which is used to evaluate each state, A(s,a;θ A ) represents the advantage function, which is used to evaluate different actions in a specific state.

[0027] To achieve the above objectives, the present application proposes a multi-domain joint anti-interference device for drone self-organizing networks based on D3QN, comprising:

[0028] An input module, configured to input a preset number of historical channel selection data and historical channel perception data per unit time into a D3QN-based anti-interference model;

[0029] A processing module, configured to process the historical channel selection data and the historical channel perception data through the D3QN-based anti-interference model and output a Q value sequence;

[0030] The determination module is configured to determine the action corresponding to the maximum Q value in the Q value sequence as the target action.

[0031] Beneficial effects of this application:

[0032] In an embodiment of the present application, a preset number of historical channel selection data and historical channel perception data per unit time are input into an anti-interference model based on D3QN, and then the historical channel selection data and historical channel perception data are processed by the anti-interference model based on D3QN to output a Q value sequence, and finally the action corresponding to the maximum Q value in the Q value sequence is determined as the target action. The present application can determine the optimal anti-interference action based on the D3QN anti-interference model for data transmission in a UAV self-organizing network, and realize joint interference resistance of channel selection and transmission rate adaptation, thereby improving network throughput and enhancing the anti-interference performance of the UAV self-organizing network.

[0033] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0035] Figure 1 A topological diagram of a UAV ad hoc network system being attacked by interference in the prior art;

[0036] Figure 2 A schematic diagram of an interference pattern of an interference source in the prior art;

[0037] Figure 3 Flowchart of a multi-domain joint anti-interference method for a UAV self-organizing network based on D3QN according to an embodiment of the present application;

[0038] Figure 4 Schematic diagram of the structure of the D3QN-based anti-interference model according to an embodiment of the present application;

[0039] Figure 5 Schematic diagram showing a comparison of average returns between an anti-interference model based on DQN and an anti-interference model based on D3QN according to an embodiment of the present application;

[0040] Figure 6 Schematic diagram showing a comparison of average throughputs of CSRAJ-D3QN, CS-D3QN, and random channel selection methods according to an embodiment of the present application;

[0041] Figure 7 Schematic diagram of the structure of a multi-domain joint anti-interference device for a drone self-organizing network based on D3QN according to an embodiment of the present application. DETAILED DESCRIPTION

[0042] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in this application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0043] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0044] Figure 1 The topology diagram of the UAV self-organizing network system under interference attack in the prior art is shown as follows: Figure 1 As shown, drone nodes are networked in a hierarchical structure. Cluster heads within the network act as local control centers, possessing high levels of hierarchy and data processing capabilities. Nodes within a cluster communicate by forwarding data through the cluster head of their respective clusters. Interference opportunities can interfere with drone nodes within the interference zone by distributing different interference power patterns across the communication channels of the drone ad hoc network. This can hinder data transmission and reduce network throughput. Drone nodes within the same cluster experience consistent interference.

[0045] The figure is a schematic diagram of the interference pattern of the interference source in the prior art, such as Figure 2 As shown in the figure, the drone self-organizing network communication scenario is based on TDMA (Time Division Multiple Access), establishing communication in time slots. Each time slot is short enough, and assuming that the interference situation within each time slot remains unchanged, the channels are modeled in time slots according to the different interference powers allocated by the jammer. Channels are numbered in sequence according to different channel indices. The darker the color on the channel, the higher the power allocated by the jammer. The jammer can distribute power across all channels or concentrate the power on several channels. The interference source will repeat the interference pattern at a certain period.

[0046] The following describes the multi-domain joint anti-interference method and device for the drone self-organizing network based on D3QN proposed in accordance with the embodiments of the present application with reference to the accompanying drawings. First, the multi-domain joint anti-interference method for the drone self-organizing network based on D3QN proposed in accordance with the embodiments of the present application will be described with reference to the accompanying drawings.

[0047] Figure 3 Flowchart of the multi-domain joint anti-interference method of the UAV self-organizing network based on D3QN according to the embodiment of the present application. Figure 3 As shown in FIG, the multi-domain joint anti-interference method of the UAV self-organizing network based on D3QN includes:

[0048] Step S310 : Inputting a preset number of historical channel selection data and historical channel perception data per unit time into an anti-interference model based on D3QN.

[0049] In an embodiment of the present application, historical channel selection data and historical channel perception data of a preset number of unit time can be obtained. Taking the preset number 16 as an example, historical channel selection data and historical channel perception data of 16 unit time can be obtained. The historical channel selection data and historical channel perception data can be the channel selection data and channel perception data of the transmitting end before the current moment. After obtaining the historical channel selection data and historical channel perception data of the preset number of unit time, the historical channel selection data and historical channel perception data of the preset number of unit time can be input into the anti-interference model based on D3QN. For example, still taking the preset number 16 as an example, the historical channel selection data and historical channel perception data of the 16 unit time before the current moment of the transmitting end can be combined into a sequence s t , Among them, action Indicates that the sender sends a packet at time ti, and the action value range is Select data for the channel, Select the transmission rate. Channel sensing data o t-i Indicates the interference status on the transmission channel at time ti, and its value range is P J ={0,1,2,3,4}, 0 means there is no interference on the channel, the larger the number, the greater the interference, and different gears are set according to the perceived interference intensity. Finally, the sequence can be input into the D3QN-based anti-interference model.

[0050] Step S320 : Process the historical channel selection data and the historical channel perception data through the D3QN-based anti-interference model to output a Q value sequence.

[0051] In the embodiments of this application, Figure 4 FIG is a structural diagram of an anti-interference model based on D3QN according to an embodiment of the present application, as shown in FIG. Figure 4 As shown, the D3QN-based anti-interference model can include a D3QN error function, an environment, an evaluation network, a Q-value network, and an experience replay pool. The D3QN-based anti-interference model can process historical channel selection data and historical channel perception data to output a Q-value sequence.

[0052] Step S330 : Determine the action corresponding to the maximum Q value in the Q value sequence as the target action.

[0053] In an embodiment of the present application, after obtaining a Q value sequence, the action corresponding to the maximum Q value in the Q value sequence can be determined as the target action. The target action can be the action to be executed by the transmitter at the next moment. For example, the action corresponding to the maximum Q value in the Q value sequence can be determined as the target action by the following formula:

[0054]

[0055] in, represents the target action, Q(s t ,a t ; θ) represents the Q value sequence.

[0056] In an embodiment of the present application, a preset number of historical channel selection data and historical channel perception data per unit time are input into an anti-interference model based on D3QN, and then the historical channel selection data and historical channel perception data are processed by the anti-interference model based on D3QN to output a Q value sequence, and finally the action corresponding to the maximum Q value in the Q value sequence is determined as the target action. The present application can determine the optimal anti-interference action based on the D3QN anti-interference model for data transmission in a UAV self-organizing network, and realize joint interference resistance of channel selection and transmission rate adaptation, thereby improving network throughput and enhancing the anti-interference performance of the UAV self-organizing network.

[0057] In one possible implementation, the D3QN-based anti-interference model includes the following loss function:

[0058] L(θ)=E[(yQ(s t ,a t θ i ) 2 ]

[0059] Among them, Q(s t ,a t θ i ) represents the current state s t 、Current moment action a t and the model parameters θ of the i-th iteration i When is input, the Q value of the Q-Network under the model parameters of the i-th iteration is evaluated, and y represents the target value calculated by the target Q-Network under the model parameters of the i-th iteration.

[0060] In an embodiment of the present application, the D3QN-based anti-interference model may include a loss function of the following formula:

[0061] L(θ)=E[(yQ(s t ,a t θ i ) 2 ]

[0062] Among them, Q(s t ,a t θ i ) represents the current state s t 、Current moment action a t and the model parameters θ of the i-th iteration i When is input, the Q value of the Q-Network under the model parameters of the i-th iteration is evaluated, and y represents the target value calculated by the target Q-Network under the model parameters of the i-th iteration.

[0063] It's important to note that the D3QN algorithm uses a double network to minimize the loss function using an evaluation network (evaluation Q-Network) and a target network (target Q-Network). The evaluation network and the target network have identical structures, and the evaluation network's parameters can be updated in real time. At fixed intervals, the target network's parameters are replaced by the evaluation network's parameters, with each replacement considered an iteration.

[0064] In one possible implementation, the calculation formula for y is as follows:

[0065]

[0066] Among them, r t represents the feedback at the current moment, γ represents the attenuation coefficient, s t+1 represents the state at the next moment, θ - represents the model parameters of the i-1th iteration, Indicates the action with the largest Q value evaluated by the Q-Network.

[0067] In the embodiment of the present application, under the model parameters of the i-th iteration, the target value y calculated by the target Q-Network can be calculated as follows:

[0068]

[0069] Among them, r t represents the feedback at the current moment, γ represents the attenuation coefficient, s t+1 represents the state at the next moment, θ - represents the model parameters of the i-1th iteration, Indicates the action with the largest Q value evaluated by the Q-Network.

[0070] It should be noted that after the target Q-Network is processed with the model parameters of the i-1th iteration, the action with the highest value in the evaluation Q-Network output by the model corresponds to the Q value. The network parameters θ of the target Q-Network are- Keeping it fixed for a period of time can ensure that the evaluation Q-Network can relatively fix the target value y, which is conducive to the convergence of the algorithm.

[0071] In one possible implementation, r t The calculation formula is as follows:

[0072]

[0073] Among them, λ and μ represent the adjustment parameters, It represents the ratio of transmission power to the maximum supported rate, and J represents the loss value when transmission fails due to interference.

[0074] In the embodiment of the present application, after the transmitting end executes the target action and transmits the data, the receiving end can return an ACK (Acknowledge character) based on whether the transmission is successful. The D3QN-based anti-interference model can obtain feedback at the current moment based on the ACK, and the network state is transferred to the next moment. The calculation formula for the feedback at the current moment can be as follows:

[0075]

[0076] Among them, λ and μ represent the adjustment parameters, It represents the ratio of transmission power to the maximum supported rate, and J represents the loss value when transmission fails due to interference.

[0077] It should be noted that the adjustment parameters are used to adjust the D3QN-based anti-interference model's reward weighting for successful transmission and maximized throughput. The ratio of transmission power to the maximum supported rate is used to maximize network throughput. When transmission is successful, the feedback at the current moment is related to the transmission rate selected by the drone ad hoc network node. The higher the selected transmission rate, the higher the reward value. When transmission fails, the feedback at the current moment is negative, indicating that the D3QN-based anti-interference model has failed to make a decision. The network throughput during this time slot is 0, which is a penalty.

[0078] In one possible implementation, the Q value is calculated as follows:

[0079]

[0080] Among them, V(s; θ V ) represents the state value function, which is used to evaluate each state, A(s,a;θ A ) represents the advantage function, which is used to evaluate different actions in a specific state.

[0081] In the embodiment of the present application, the D3QN algorithm adopts Dueling Network, which divides the output layer into two parts for output. One part is output through the state value function V(s; θ V ), evaluate the quality of each state, and the other part is evaluated by the advantage function A(s,a;θ A ), evaluates the quality of different actions under specific states. Therefore, in the D3QN-based anti-interference model, the Q-worth calculation formula can be as follows:

[0082]

[0083] Among them, V(s; θ V ) represents the state value function, which is used to evaluate each state, A(s,a;θ A ) represents the advantage function, which is used to evaluate different actions in a specific state.

[0084] It should be noted that Figure 5 This is a convergence comparison chart of the DQN-based anti-interference model and the D3QN-based anti-interference model under the same interference mode, and the horizontal axis is the training time. Figure 5 As can be seen, both can converge, but the D3QN-based anti-interference model converges faster than DQN, can learn good strategies more quickly during training, and improve network throughput. This is because the D3QN algorithm decouples the evaluation Q network and the target Q network, alleviating the overestimation problem of DQN. It also introduces Dueling Network to separate the environment and action to a certain extent. Therefore, using the D3QN algorithm for anti-interference model training can effectively guide the transmitter to perform joint anti-interference on channel selection and transmission rate.

[0085] Figure 6 This is a performance comparison chart of the CSRAJ-D3QN method, CS-D3QN, and random channel selection algorithm. The latter two methods only use 54Mbps communication. Figure 6 It can be seen that the random frequency hopping algorithm randomly selects a channel for communication each time. Therefore, when the number of channels interfered by the interference source is large, the performance is poor, and the throughput is close to the minimum rate of 9Mbps. CS-D3QN only considers channel selection, so the convergence speed is fast. However, when the interference source interferes on every channel, the anti-interference model cannot find a channel for communication. Therefore, although the CS-D3QN algorithm converges quickly, its throughput at convergence is about half of the maximum throughput. The CSRAJ-D3QN method adopts a joint anti-interference method of channel selection and rate adaptation. When interference exists on all channels, it can complete signal transmission under interference conditions by reducing the transmission rate. Therefore, the throughput at convergence is close to the maximum throughput of 54Mbps, indicating that the proposed method can effectively guide the transmitter to perform joint anti-interference selection of channel selection and transmission rate.

[0086] In order to implement the above embodiment, Figure 7 As shown, this embodiment also provides a multi-domain joint anti-interference device 700 for a UAV self-organizing network based on D3QN. The device 700 includes: an input module 710, a processing module 720, and a determination module 730.

[0087] An input module 710 is configured to input a preset number of unit time historical channel selection data and historical channel perception data into a D3QN-based anti-interference model;

[0088] A processing module 720 is configured to process historical channel selection data and historical channel perception data using a D3QN-based anti-interference model and output a Q value sequence;

[0089] The determination module 730 is configured to determine the action corresponding to the maximum Q value in the Q value sequence as the target action.

[0090] According to the D3QN-based multi-domain joint anti-interference device for drone self-organizing networks in an embodiment of the present application, an input module is used to input a preset number of unit time historical channel selection data and historical channel perception data into an anti-interference model based on D3QN; a processing module is used to process the historical channel selection data and historical channel perception data through an anti-interference model based on D3QN and output a Q value sequence; a determination module is used to determine the action corresponding to the maximum Q value in the Q value sequence as the target action. The present application can determine the optimal anti-interference action based on the D3QN anti-interference model, which can be used for data transmission in drone self-organizing networks, and realize joint interference resistance of channel selection and transmission rate adaptation, thereby improving network throughput and enhancing the anti-interference performance of drone self-organizing networks.

[0091] It should be noted that the above explanation of the embodiment of the multi-domain joint anti-interference method of the drone self-organizing network based on D3QN is also applicable to the multi-domain joint anti-interference device of the drone self-organizing network based on D3QN in this embodiment, and will not be repeated here.

[0092] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referred to. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of such features. Throughout the description of this application, "plurality" means at least two, for example, two, three, etc., unless otherwise specifically defined.

[0093] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0094] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in this field can change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A multi-domain joint anti-interference method for UAV ad hoc networks based on D3QN, characterized by: include: Inputting a preset number of historical channel selection data and historical channel perception data per unit time into the D3QN-based anti-interference model; Processing the historical channel selection data and the historical channel perception data through the D3QN-based anti-interference model to output a Q value sequence; Determine the action corresponding to the maximum Q value in the Q value sequence as the target action; The D3QN-based anti-interference model includes the following loss function: in, Indicates the current state , current moment action Hedi i Model parameters for iterations When the input is i The Q value of the model parameters under the iteration, y represents the i The target value calculated by the target Q-Network under the model parameters of the iteration; The calculation formula of y is as follows: in, Indicates the feedback at the current moment, represents the attenuation coefficient, Indicates the state at the next moment. represents the model parameters of the i-1th iteration, Indicates the action with the largest Q value evaluated by the Q-Network; described The calculation formula is as follows: in, and Represents the adjustment parameter used to adjust the reward weight of the D3QN-based anti-interference model for successful transmission and maximized throughput; It represents the ratio of transmission power to the maximum supported rate, which is used to maximize network throughput; J represents the loss value when transmission fails due to interference.

2. The multi-domain joint anti-interference method for UAV ad hoc networks based on D3QN according to claim 1 is characterized in that: The calculation formula of the Q value is as follows: in, Represents the state value function, which is used to evaluate each state, represents the advantage function, which is used to evaluate different actions in a specific state.

3. A multi-domain joint anti-interference device for UAV self-organizing network based on D3QN, characterized by: include: An input module, configured to input a preset number of historical channel selection data and historical channel perception data per unit time into a D3QN-based anti-interference model; A processing module, configured to process the historical channel selection data and the historical channel perception data through the D3QN-based anti-interference model and output a Q value sequence; A determination module, configured to determine the action corresponding to the maximum Q value in the Q value sequence as the target action; The D3QN-based anti-interference model includes the following loss function: in, Indicates the current state , current moment action Hedi i Model parameters for iterations When the input is i The Q value of the model parameters under the iteration, y represents the i The target value calculated by the target Q-Network under the model parameters of the iteration; The calculation formula of y is as follows: in, Indicates the feedback at the current moment, represents the attenuation coefficient, Indicates the state at the next moment. represents the model parameters of the i-1th iteration, Indicates the action with the largest Q value evaluated by the Q-Network; described The calculation formula is as follows: in, and Represents the adjustment parameter used to adjust the reward weight of the D3QN-based anti-interference model for successful transmission and maximized throughput; It represents the ratio of transmission power to the maximum supported rate, which is used to maximize network throughput. J represents the loss value when transmission fails due to interference.

Images