Adaptive frequency hopping anti-interference method and device based on federated learning and electronic equipment

Abstract

This application provides an adaptive frequency hopping anti-jamming method, apparatus, and electronic device based on federated learning. The method is applied to various unmanned aerial vehicles (UAVs) in a flight ad hoc network. It utilizes a pre-set frequency hopping anti-jamming model to determine a target frequency hopping strategy based on the communication environment of the UAV and executes that strategy. Because the pre-set frequency hopping anti-jamming model in this embodiment employs a dual deep Q-network, it effectively reduces the overestimation problem of existing intelligent anti-jamming algorithms. Furthermore, by using federated learning between the UAV and base station equipment, this embodiment can obtain the carrier frequencies selected by different UAVs in different environments and the instantaneous rewards received. The resulting dual deep Q-network not only reduces overestimation but is also applicable to different anti-jamming environments, exhibiting higher versatility.

Description

Technical Field

[0001] This application relates to the field of communication technology, and in particular to an adaptive frequency hopping anti-interference method, apparatus and electronic device based on federated learning. Background Technology

[0002] Communication anti-interference technology is a crucial technology for ensuring communication quality. Traditional communication anti-interference technologies fall into two main categories: spread spectrum technology and non-spread spectrum technology. Spread spectrum technology enhances communication anti-interference capabilities by expanding the information bandwidth for data transmission, while non-spread spectrum technology enhances the anti-interference capabilities of the communication system without expanding the signal spectrum through techniques such as adaptive filtering, interference notch filtering, and adaptive frequency selection. Practice has shown that combining spread spectrum and non-spread frequency technologies can achieve superior anti-interference effects. However, with the increasing complexity of the electromagnetic environment and the advancement of interference technologies, the shortcomings of traditional anti-interference technologies are becoming increasingly apparent. Therefore, intelligent anti-interference technology has become a research hotspot in the field of anti-interference technology.

[0003] Common intelligent anti-interference technologies currently include reinforcement learning-based anti-interference algorithms. However, existing intelligent anti-interference algorithms are prone to problems such as inability to adapt to different interference environments and poor versatility. Summary of the Invention

[0004] In view of this, embodiments of this application provide an adaptive frequency hopping anti-interference method, apparatus, and electronic device based on federated learning, so as to provide an adaptive anti-interference strategy for diverse interference environments.

[0005] In a first aspect, embodiments of this application provide an adaptive frequency hopping anti-jamming method based on federated learning, wherein the method is applied to various unmanned aerial vehicle (UAV) devices in a flight ad hoc network system, and the method includes:

[0006] Using a pre-set frequency hopping anti-interference model, the target frequency hopping strategy is determined based on the communication environment of the UAV device, and the target frequency hopping strategy is executed.

[0007] The flight ad hoc network system also includes: base station equipment, and the pre-set frequency hopping anti-interference model is trained through the following steps:

[0008] The UAV device trains a pre-set initial dual deep Q network model based on initial network parameters and interaction information with its environment. The interaction information with the environment includes: its own communication state information, actions taken, immediate rewards obtained, and new state information. The actions taken are carrier frequencies selected under the existing state information.

[0009] The UAV device then uploads the model parameters of the trained dual deep Q-network model to the base station device.

[0010] The base station equipment generates an aggregated network model based on the model parameters uploaded by each of the drone devices, using a federated averaging algorithm, and then sends the aggregated network model to each of the drone devices.

[0011] The UAV device receives the aggregated network model sent by the base station device, performs secondary learning training on the dual deep Q network model based on the aggregated network model, and determines the dual deep Q network model obtained from the secondary learning training as the preset frequency hopping anti-interference model.

[0012] Optionally, the dual-depth Q-network model includes: a behavior network and a target Q-value generation network, wherein the behavior network is used to select the action to be taken, and the target Q-value generation network is used to generate a Q-value based on the action selected by the behavior network; the step of training the preset initial dual-depth Q-network model by the UAV device based on the initialized network parameters and the interaction information with the environment includes:

[0013] Training sample data is obtained from a preset experience replay pool. The training sample data is the interaction information between the UAV equipment and its environment collected in the past.

[0014] The training sample data is input into the preset initial dual-depth Q-network model. The behavior network selects different carrier frequencies under different state information, and the target Q-value generation network outputs the predicted Q-value based on the carrier frequency selected by the behavior network.

[0015] The weights of the behavior network are updated based on the error between the predicted Q-value and the Q-value label until the error between the predicted Q-value and the Q-value label converges, wherein the Q-value label is the training label of the dual deep network model.

[0016] Optionally, updating the weights of the behavioral network based on the error between the predicted Q-value and the Q-value label includes:

[0017] Based on the error between the predicted Q-value and the Q-value label, the weights of the behavioral network are updated using the following weight update equation:

[0018] Q(s,a)←Q(s,a)+α[r+γmax a′ Q(s ′ ,a ′ )-Q(s,a)]

[0019] Where Q(s) ′,a ′ Let Q(s,a) be the predicted Q-value, Q(s,a) be the Q-value label, s be the current channel state of the UAV, a be the frequency used by the UAV in its current state s, α be the model's learning rate, and γ be a discount factor used to measure the importance of the reward. ′ For the UAV equipment to be in the channel state of the next time slot, a ′ The frequency selected for the next time slot, r is the frequency at which the UAV device is in s ′ The instantaneous reward obtained.

[0020] Optionally, the step of training a pre-set initial dual-depth Q-network model by the UAV device based on initial network parameters and interaction information with the environment further includes:

[0021] According to the preset training rounds, the parameters of the target Q-value generation network are updated based on the updated parameters of the behavior network.

[0022] Optionally, the step of generating an aggregated network model by the base station device based on the model parameters uploaded by each of the UAV devices and using a federated averaging algorithm includes:

[0023] The model parameters uploaded by each of the aforementioned UAV devices are received, and the received model parameters are input into the following aggregation equation to generate the aggregation network model:

[0024]

[0025] Where ω is the weight parameter of the model, t is the time slot, N is the total number of UAV devices, k1 is each UAV device in the first interference environment, k2 is each UAV device in the second interference environment, and n k1 Let n be the number of drones in the first interference environment. k2 Let be the number of drones in the second interference environment, 1k be the model parameters of the drones in the first interference environment, and 2k be the model parameters of the drones in the second interference environment. The weight parameters of the model in the first disturbance environment at time slot t+1 are... These are the weight parameters of the model in the second disturbance environment at time slot t+1.

[0026] Optionally, the step of receiving the aggregated network model sent by the base station device from the UAV device, and performing secondary learning training on the dual deep Q-network model based on the aggregated network model, includes:

[0027] The global model parameters of the aggregated network model are used as the initial model parameters for the dual deep Q-network model trained in the second learning stage.

[0028] Based on the initial model parameters and the interaction information with the environment, the dual deep Q network model is trained until the carrier frequency output by the dual deep Q network model meets the preset anti-interference conditions, wherein the preset anti-interference conditions are determined according to the environment.

[0029] Secondly, this application provides an adaptive frequency hopping anti-jamming device based on federated learning, which is applied to various unmanned aerial vehicle (UAV) devices in a flight ad hoc network system. The device includes:

[0030] The frequency hopping strategy determination module is used to determine the target frequency hopping strategy based on the communication environment of the UAV device using a preset frequency hopping anti-interference model.

[0031] An anti-interference module is used to execute the target frequency hopping strategy;

[0032] The flight ad hoc network system also includes: base station equipment, and the pre-set frequency hopping anti-interference model is trained through the following steps:

[0033] The UAV device trains a pre-set initial dual deep Q network model based on initial network parameters and interaction information with its environment. The interaction information with the environment includes: its own communication state information, actions taken, immediate rewards obtained, and new state information. The actions taken are carrier frequencies selected under the existing state information.

[0034] The UAV device then uploads the model parameters of the trained dual deep Q-network model to the base station device.

[0035] The base station equipment generates an aggregated network model based on the model parameters uploaded by each of the drone devices, using a federated averaging algorithm, and then sends the aggregated network model to each of the drone devices.

[0036] The UAV device receives the aggregated network model sent by the base station device, performs secondary learning training on the dual deep Q network model based on the aggregated network model, and determines the dual deep Q network model obtained from the secondary learning training as the preset frequency hopping anti-interference model.

[0037] Optionally, the dual-depth Q-network model includes: a behavior network and a target Q-value generation network, wherein the behavior network is used to select the action to be taken, and the target Q-value generation network is used to generate a Q-value based on the action selected by the behavior network; the step of training the preset initial dual-depth Q-network model by the UAV device based on the initialized network parameters and the interaction information with the environment includes:

[0038] Training sample data is obtained from a preset experience replay pool. The training sample data is the interaction information between the UAV equipment and its environment collected in the past.

[0039] The training sample data is input into the preset initial dual-depth Q-network model. The behavior network selects different carrier frequencies under different state information, and the target Q-value generation network outputs the predicted Q-value based on the carrier frequency selected by the behavior network.

[0040] The weights of the behavior network are updated based on the error between the predicted Q-value and the Q-value label until the error between the predicted Q-value and the Q-value label converges, wherein the Q-value label is the training label of the dual deep network model.

[0041] Optionally, updating the weights of the behavioral network based on the error between the predicted Q-value and the Q-value label includes:

[0042] Based on the error between the predicted Q-value and the Q-value label, the weights of the behavioral network are updated using the following weight update equation:

[0043] Q(s,a)←Q(s,a)+α[r+γmax a′ Q(s ′ ,a ′ )-Q(s,a)]

[0044] Where Q(s) ′ ,a ′ Let Q(s,a) be the predicted Q-value, Q(s,a) be the Q-value label, s be the current channel state of the UAV, a be the frequency used by the UAV in its current state s, α be the model's learning rate, and γ be a discount factor used to measure the importance of the reward. ′ For the UAV equipment to be in the channel state of the next time slot, a ′ The frequency selected for the next time slot, r is the frequency at which the UAV device is in s ′ The instantaneous reward obtained.

[0045] Optionally, the step of training a pre-set initial dual-depth Q-network model by the UAV device based on initial network parameters and interaction information with the environment further includes:

[0046] According to the preset training rounds, the parameters of the target Q-value generation network are updated based on the updated parameters of the behavior network.

[0047] Optionally, the step of generating an aggregated network model by the base station device based on the model parameters uploaded by each of the UAV devices and using a federated averaging algorithm includes:

[0048] The model parameters uploaded by each of the aforementioned UAV devices are received, and the received model parameters are input into the following aggregation equation to generate the aggregation network model:

[0049]

[0050] Where ω is the weight parameter of the model, t is the time slot, N is the total number of UAV devices, k1 is each UAV device in the first interference environment, k2 is each UAV device in the second interference environment, and n k1 Let n be the number of drones in the first interference environment. k2 Let be the number of drones in the second interference environment, 1k be the model parameters of the drones in the first interference environment, and 2k be the model parameters of the drones in the second interference environment. The weight parameters of the model in the first disturbance environment at time slot t+1 are... These are the weight parameters of the model in the second disturbance environment at time slot t+1.

[0051] Optionally, the step of receiving the aggregated network model sent by the base station device from the UAV device, and performing secondary learning training on the dual deep Q-network model based on the aggregated network model, includes:

[0052] The global model parameters of the aggregated network model are used as the initial model parameters for the dual deep Q-network model trained in the second learning stage.

[0053] Based on the initial model parameters and the interaction information with the environment, the dual deep Q network model is trained until the carrier frequency output by the dual deep Q network model meets the preset anti-interference conditions, wherein the preset anti-interference conditions are determined according to the environment.

[0054] Thirdly, embodiments of this application provide an electronic device, wherein the electronic device includes:

[0055] Processor; and

[0056] Stored program memory,

[0057] The program includes instructions that, when executed by the processor, cause the processor to perform the adaptive frequency hopping anti-interference method based on federated learning as described in the first aspect.

[0058] Fourthly, embodiments of this application provide a non-transitory computer-readable storage medium storing computer instructions, characterized in that the computer instructions are used to cause a computer to execute the adaptive frequency hopping anti-interference method based on federated learning described in the first aspect.

[0059] The beneficial effects of this application are:

[0060] This application provides an adaptive frequency hopping anti-jamming method, device, and electronic device based on federated learning. The method is applied to various UAV devices in a flight ad hoc network. The method uses a pre-set frequency hopping anti-jamming model to determine the target frequency hopping strategy based on the communication environment of the UAV device and executes the target frequency hopping strategy.

[0061] Since the pre-set frequency hopping anti-interference model is pre-trained by the UAV equipment in conjunction with the base station equipment, specifically, the UAV equipment trains the initial dual deep Q network model based on the initial network parameters and the interaction information between the UAV equipment and its environment. After training, the model parameters of the trained dual deep Q network model are uploaded to the base station equipment. The base station equipment generates an aggregated network model based on the model parameters uploaded by each UAV equipment using the federated averaging algorithm. Then, the aggregated network model is sent to each UAV equipment, and the UAV equipment performs secondary learning and training on the dual deep Q network model to obtain the pre-set frequency hopping anti-interference model.

[0062] Since the preset frequency hopping anti-interference model in this application embodiment adopts a dual deep Q network, the dual deep Q network itself can reduce the overestimation of the generated Q value, which can effectively reduce the overestimation problem of existing intelligent anti-interference algorithms. Furthermore, this application embodiment can obtain the carrier frequency selected by different UAV devices in different environments and the real-time reward obtained by them through federated learning between UAV devices and base station devices. The dual deep Q network trained in this way can not only reduce the overestimation problem, but also be applicable to different anti-interference environments, thus having higher versatility. Attached Figure Description

[0063] Further details, features, and advantages of this application are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:

[0064] Figure 1 This paper illustrates a flowchart of an adaptive frequency hopping anti-interference method based on federated learning provided in an embodiment of this application.

[0065] Figure 2 This paper illustrates a training process diagram of the frequency hopping anti-interference model provided in an embodiment of this application.

[0066] Figure 3 This diagram illustrates the hybrid interference in tracking interference using a comb-like interference model provided by the prior art.

[0067] Figure 4This diagram illustrates the hybrid interference in comb-like interference using a prior art tracking interference model.

[0068] Figure 5 This diagram illustrates the hybrid interference in tracking interference using a dynamic probabilistic interference model provided by the prior art.

[0069] Figure 6 This diagram illustrates the hybrid interference in dynamic probabilistic interference using existing tracking interference models.

[0070] Figure 7 This diagram illustrates the accuracy of the combination of comb interference and tracking interference provided in an embodiment of this application.

[0071] Figure 8 A schematic diagram illustrating the effect of the comb-shaped interference model provided in the embodiments of this application is shown;

[0072] Figure 9 This paper illustrates the effect of the tracking interference model provided in an embodiment of this application.

[0073] Figure 10 A schematic diagram illustrating the combined interference model effect provided in an embodiment of this application is shown.

[0074] Figure 11 This illustration shows the accuracy of the combined effect of dynamic probability interference and tracking interference provided in the embodiments of this application;

[0075] Figure 12 This paper illustrates the effect of the dynamic probability interference model provided in an embodiment of this application.

[0076] Figure 13 This paper illustrates the effect of the tracking interference model provided in an embodiment of this application.

[0077] Figure 14 A schematic diagram illustrating the combined interference model effect provided in an embodiment of this application is shown.

[0078] Figure 15 This paper illustrates a schematic diagram of the global model performance accuracy provided in an embodiment of this application.

[0079] Figure 16 This paper illustrates a logical structure diagram of an adaptive frequency hopping anti-interference device based on federated learning provided in an embodiment of this application.

[0080] Figure 17 A schematic diagram of the logical structure of an electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0081] Embodiments of this application will now be described in more detail with reference to the accompanying drawings. While some embodiments of this application are shown in the drawings, it should be understood that this application can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this application. It should be understood that the drawings and embodiments of this application are for illustrative purposes only and are not intended to limit the scope of protection of this application.

[0082] It should be understood that the steps described in the method embodiments of this application may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this application is not limited in this respect.

[0083] The term "comprising" and its variations as used herein are open-ended, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the following description. It should be noted that the concepts of "first", "second", etc., mentioned in this application are used only to distinguish different devices, modules, or units, and are not intended to limit the order of functions performed by these devices, modules, or units or their interdependencies.

[0084] It should be noted that the terms "a" and "a plurality of" used in this application are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0085] As described in the background section, common intelligent anti-interference techniques include reinforcement learning-based anti-interference algorithms. Among these, reinforcement learning-based anti-interference algorithms have been widely used and researched due to their natural suitability for anti-interference scenarios. However, existing reinforcement learning models are prone to the following problems:

[0086] Problem 1: Overestimation. This problem is particularly prevalent when using Q-learning and its variants (such as Deep Q-Networks). Deep Q-Networks (DQN) uses the same neural network to simultaneously estimate the Q-value of the current state and the target Q-value (Q-value refers to the value of the dynamic value function). This can easily lead to an overly optimistic estimate of the target Q-value, resulting in overestimation. Therefore, overestimation means that the Q-learning model tends to choose the largest Q-value when estimating the expected reward. This can lead to a systematic overestimation of the value of certain behaviors, resulting in poor stability of the learning process and affecting the effectiveness of the anti-interference strategy ultimately provided by the model. Specifically, overestimation causes the model to favor overestimated behaviors during the learning process, potentially ignoring other better or more suitable strategies for the current environment, thus reducing the algorithm's generality and adaptability.

[0087] Problem 2: Inability to adapt to environmental changes leads to low communication efficiency. Specifically, most existing reinforcement learning models can only effectively handle one type of interference, such as comb interference or tracking interference. In some practical applications, such as in-flight ad hoc networks (e.g., UAV ad hoc networks), communication devices typically move at high speeds and change positions rapidly, making them susceptible to both dynamic and static interference. The complex interactions of different types of interference can significantly impact communication reliability. When faced with multiple complex interferences, models with a single interference handling strategy cannot effectively adapt to environmental changes, leading to low communication efficiency and, in some cases, even failure to complete the task.

[0088] Problem 3: The issue of neglecting resource sharing among multiple communication devices exists. Current intelligent anti-jamming frequency modulation technologies mostly focus on the communication of a single node or a few nodes, which easily overlooks the issue of resource sharing among multiple communication devices. This is especially true in flight ad hoc networks, which contain multiple UAVs. The interaction between these devices can significantly affect the overall performance of the system, and existing models may not be effectively implemented in such multi-device environments.

[0089] In view of this, this application provides an adaptive frequency hopping anti-jamming method, apparatus, and electronic device based on federated learning, to provide an adaptive anti-jamming strategy for complex interference environments. Specifically, in a first aspect, this application provides an adaptive frequency hopping anti-jamming method based on federated learning, which can be applied to any electronic device with anti-jamming capabilities, including but not limited to personal mobile terminals, computers, or servers. As a preferred embodiment, this method can be applied to unmanned aerial vehicle (UAV) devices in ad hoc flight networks.

[0090] In some embodiments, such as Figure 1As shown, the method includes the following steps:

[0091] S11. Using a pre-set frequency hopping anti-interference model, determine the target frequency hopping strategy based on the communication environment of the UAV device;

[0092] S12. Execute the target frequency hopping strategy.

[0093] In addition to various unmanned aerial vehicle (UAV) devices, the flight ad hoc network system also includes base station equipment, which in some embodiments can be as follows: Figure 2 As shown, the pre-set frequency hopping anti-interference model is trained in advance through the following steps:

[0094] S21. The UAV device trains a pre-set initial dual-depth Q-network model based on the initial network parameters and the interaction information between the UAV and its environment.

[0095] The interaction information with the environment includes: its own communication status information, actions taken, immediate rewards obtained, and new status information. The actions taken are carrier frequencies selected under the existing status information.

[0096] S22. The UAV device uploads the model parameters of the trained dual deep Q network model to the base station device;

[0097] S23. The base station device generates an aggregated network model based on the model parameters uploaded by each of the UAV devices using a federated averaging algorithm, and sends the aggregated network model to each of the UAV devices.

[0098] S24. The UAV device receives the aggregated network model sent by the base station device, performs secondary learning training on the dual deep Q network model based on the aggregated network model, and determines the dual deep Q network model obtained from the secondary learning training as the preset frequency hopping anti-interference model.

[0099] As one implementation method, the Double Deep Q-Network (DDQN) model can be pre-deployed in various UAV devices. The Double Deep Q-Network (DDQN) model is an improved Q-learning algorithm. In some embodiments, the DDQN model includes a behavior network and a target Q-value generation network. The behavior network selects actions to be taken, and the target Q-value generation network generates Q-values ​​based on the actions selected by the behavior network. This DDQN model employs a separate network architecture: one network selects actions, and the other generates target Q-values ​​based on the selected actions. The target Q-value refers to the value of performing the selected action in a specific state, typically calculated using a Q-function (i.e., the Q(s,a) function, where s represents a specific state and a represents the selected action). This Q-function characterizes the "quality" of performing an action in a specific state.

[0100] This structure helps separate action selection from the generation of the target Q-value, thus enabling more accurate estimation of action values ​​and enhancing the algorithm's applicability. Here, action value refers to the frequency hopping strategy selected by the UAV in its current environment, including the carrier frequency chosen by the UAV in that environment. By selecting a frequency hopping strategy and changing the carrier frequency for each communication, it is possible to avoid interference channels and continue communication.

[0101] In this embodiment, the difference between the dual-depth Q-network model and the ordinary Q-network model lies in the following: In this embodiment, the action with the highest Q-value is selected through a behavior network, and then the target Q-value generation network is used to calculate the Q-value of the action based on this action. This helps to reduce the problem of Q-value overestimation commonly found in traditional single-network models. Based on this, the dual-depth Q-network model provided in this embodiment can more accurately determine the value of the selected frequency hopping strategy based on the state of the UAV device and the selected frequency hopping strategy. This allows for a more accurate assessment of the usability of the selected frequency hopping strategy, thus reducing the influence of the simple Q-network model mentioned in Problem 1, which tends to select overestimated actions. The dual-depth Q-network can provide more stable and reliable learning performance, thereby enhancing the ability of the trained frequency hopping anti-interference model to cope with complex environments.

[0102] Based on this, step S21 above can be implemented in the following way:

[0103] S21-1. Obtain training sample data from a preset experience playback pool. The training sample data is the interaction information between the UAV equipment and its environment that was collected historically.

[0104] S21-2. The training sample data is input into the preset initial dual-depth Q-network model. The behavior network selects different carrier frequencies under different state information, and the target Q-value generation network outputs the predicted Q-value based on the carrier frequency selected by the behavior network.

[0105] S21-3. Update the weights of the behavior network according to the error between the predicted Q value and the Q value label until the error between the predicted Q value and the Q value label converges, wherein the Q value label is the training label of the dual deep network model.

[0106] Each UAV device can have a pre-installed client containing a pre-deployed initial dual-depth Q-network model. This client is specifically used to train this pre-installed initial dual-depth Q-network model. As one implementation, before training the pre-installed initial dual-depth Q-network model, the client initializes the network parameters, resulting in initialized network parameters. These initialized network parameters can be initial values, either factory defaults or values ​​obtained based on historical data. As another implementation, during secondary training of the dual-depth Q-network model, the client can also perform secondary network parameter initialization based on the network parameters sent by the base station, ensuring that the UAV device's network parameters are identical to those sent by the base station. As yet another implementation, the initialized network parameters can be randomly set, with any value arbitrarily selected.

[0107] Since unmanned aerial vehicles (UAVs) continuously interact with their environment during flight missions, the various types of information generated by this interaction are collectively referred to as "interaction information with the environment" in this embodiment. This interaction information specifically includes: the state information of the UAV at each interaction, the actions taken at each interaction (i.e., the communication carrier frequency selected at each interaction), the immediate rewards obtained at each interaction, and new state information.

[0108] Specifically, the state information of the drone device during each interaction includes the frequency information of the current time slot, and the new state information refers to the frequency information of the next time slot. The frequency information of a time slot refers to a sub-band with a fixed frequency range divided within a specific bandwidth.

[0109] The immediate reward obtained at each interaction specifically refers to the evaluation value output after selecting a frequency point in the current time slot, based on whether the selected frequency point is correct and whether the frequency point has changed. This evaluation value is the immediate reward obtained. A mapping function can be constructed using the selected frequency point, whether the frequency point is correct, and whether the frequency point has changed as independent variables, and the evaluation value as the dependent variable. When obtaining the immediate reward at each interaction, the corresponding immediate reward can be directly calculated using this mapping function.

[0110] As one implementation, the interaction information between the UAV device and its environment generated during each interaction can be stored in an experience replay pool. Alternatively, various data generated during each interaction can be stored in this experience replay pool to obtain sufficient training sample data during subsequent training, thus improving learning efficiency. In this embodiment, by randomly setting the initial network parameters of the dual deep Q-network model and combining it with sufficient historical data in the experience replay pool, the dual deep Q-network model can learn from historical experience, breaking the temporal correlation of data. Furthermore, using random sampling to obtain training sample data can balance the training of data with various types of interference.

[0111] During the training process of this dual deep Q network model, the drone will randomly draw a batch of data from the experience replay pool as training samples. These sample data include data on the drone in various states and the corresponding rewards. This helps the dual deep Q network model learn to predict the optimal action in different states.

[0112] In some embodiments, updating the weights of the behavioral network based on the error between the predicted Q-value and the Q-value label includes:

[0113] Based on the error between the predicted Q-value and the Q-value label, the weights of the behavioral network are updated using the following weight update equation:

[0114] Q(s,a)←Q(s,a)+α[r+γmax a′ Q(s ′ ,a ′ )-Q(s,a)]

[0115] Where Q(s) ′ ,a ′Let Q(s,a) be the predicted Q-value, where s is the current state of the UAV, which can be the channel state of the UAV in the current communication time slot. a is the action taken by the UAV in the current state s (specifically, the carrier frequency chosen), i.e., the frequency selected by the UAV in the current state s. α is the learning rate, and γ is the discount factor used to measure the importance of the reward. ′ For the UAV device to be in the channel state of the next communication time slot, a ′ For drone equipment in s ′ The selected frequency point under the state, r refers to the state s of the drone equipment. ′ The instantaneous reward obtained.

[0116] Here, Q(s,a) on the left refers to the weights of the updated behavior network, and Q(s,a) on the right refers to the weights of the behavior network before the update; specifically, it means using the Q value on the right to correct the Q value on the left. At each time step (a time step refers to the time interval between one observation point and the next observation point in the sequence data, or it can refer to consecutive observation points input into the model), the model selects an action a (carrier frequency) based on the current state s and the Q value. After executing action a, the model transitions to the next state s. ′ The model then obtains an instantaneous reward *r*. It then updates the Q-values ​​corresponding to the current state and action based on the above equations to bring them closer to the optimal Q-value. This process continues until the Q-value function converges to its optimal value.

[0117] To ensure the stability of the dual deep Q-network model during the learning process, the parameters of the target Q-value generation network model are not updated with each training step. In some embodiments, the step of training a pre-set initial dual deep Q-network model by the UAV device based on the initialized network parameters and interaction information with the environment further includes:

[0118] According to the preset training rounds, the parameters of the target Q-value generation network are updated based on the updated parameters of the behavior network.

[0119] Specifically, after a preset number of training epochs, the network model parameters can be copied from the updated behavior network, and then the parameters of the target Q-value generation network can be updated based on the copied network model parameters. The preset number of training epochs can be set according to the results of multiple training sessions, and this application does not impose strict limitations on it.

[0120] In this embodiment, this delayed update strategy helps stabilize the learning process and reduce training fluctuations caused by constantly changing objectives. Through the above learning process, the UAV device can predict the long-term reward of each possible action based on the current network status, interference type, and other environmental factors. Thus, selecting the action with the highest expected reward as the optimal frequency hopping strategy optimizes the communication efficiency and reliability of the frequency hopping strategy in dynamic and complex wireless communication environments.

[0121] In some embodiments, the generation of the aggregated network model by the base station device based on the model parameters uploaded by each of the UAV devices and using a federated averaging algorithm includes:

[0122] The model parameters uploaded by each of the aforementioned UAV devices are received, and the received model parameters are input into the following aggregation equation to generate the aggregation network model:

[0123]

[0124] Where ω is the model weight parameter, t is the time slot, N is the total number of UAVs, k1 is each UAV in the first interference environment, k2 is each UAV in the second interference environment, and n k1 Let n be the number of drones in the first interference environment. k2 Let be the number of drones in the second interference environment, 1k be the model parameters of each drone in the first interference environment, and 2k be the model parameters of each drone in the second interference environment. The weight parameters of the model in the first disturbance environment at time slot t+1 are... These are the weight parameters of the model in the second disturbance environment at time slot t+1.

[0125] In this system, the base station receives model parameters uploaded by each drone device. The central server model of the base station aggregates the model parameters of each drone based on the proportion of drones in each type of interference environment, obtaining the latest global model. Specifically, in this embodiment, after training its local dual-depth Q-network model, the drone device client uploads the parameters of its trained dual-depth Q-network model to the central server model in the base station. The central server model then generates an aggregated network model based on the aforementioned formula. This process is crucial for the collaborative operation of the entire flight ad hoc network system. The base station collects learning results from the clients of each drone device and aggregates the data and models in the central server to obtain a globally optimized model (i.e., the aggregated network model).

[0126] The core of this process lies in its ability to effectively integrate the local optimization results of multiple clients to achieve global communication strategy optimization. The central server aggregates the model parameters uploaded by each UAV, realizing knowledge transfer and optimization from local to global perspectives, greatly enhancing the adaptability and anti-interference capabilities of the entire network. Furthermore, by using federated learning between UAVs and base station equipment, not only is the selection of anti-interference strategies for UAVs optimized, but technical support is also provided for cooperation and coordination among UAVs in a flight ad hoc network system.

[0127] In some embodiments, the step of receiving the aggregated network model sent by the base station device from the UAV device, and performing secondary learning training on the dual deep Q-network model based on the aggregated network model, includes:

[0128] The global model parameters of the aggregated network model are used as the initial model parameters for the dual deep Q-network model trained in the second learning stage.

[0129] Based on the initial model parameters and the interaction information with the environment, the dual deep Q network model is trained until the carrier frequency output by the dual deep Q network model meets the preset anti-interference conditions, wherein the preset anti-interference conditions are determined according to the environment.

[0130] In this embodiment, after federated learning is performed by the central server of the base station equipment, an aggregated network model, or a global model, is generated. The base station equipment then distributes this aggregated and optimized global network model to the client of each drone device. This enables iterative model updates and ensures that all drone devices maintain the optimal output transition frequency in new environments.

[0131] In this embodiment, the updated global model parameters are used as the initial model parameters for the UAV device's secondary learning and training. This is equivalent to a secondary initialization of the model parameters of the dual-depth Q-network model within the UAV device. The UAV device then re-initializes the dual-depth Q-network model based on these new network model parameters. The dual-depth Q-network model is the network model deployed in the UAV device. The aggregated network model is used to aggregate the model parameters uploaded by each UAV device to generate new parameters for the dual-depth Q-network model, thereby improving the adaptability of the dual-depth Q-network model deployed in each UAV device to different environments.

[0132] Thus, the resulting latest global model (corresponding to the dual-depth Q-network model with updated model parameters generated based on the aggregated network model) possesses the ability to cope with other interferences in the environment. Even if the UAV moves to other interference environments, it can still take corresponding frequency hopping actions based on the optimal frequency hopping strategy obtained from other UAVs in the same interference environment in the past, thereby possessing corresponding anti-interference capabilities for different environments. Although the aggregated network model has been optimized, each UAV still needs to undergo a small amount of local training for its specific environment to further fine-tune and optimize the frequency hopping strategy that can be adopted for that environment. This method of local secondary training helps the trained frequency hopping interference model better adapt to its own operating conditions and interference situations, enhancing the anti-interference capability and communication efficiency of the entire flight ad hoc network system. In this way, the defects mentioned in Problem 2 and Problem 3 of the existing technology can be overcome.

[0133] Based on this, during step S11, the UAV can directly utilize the pre-trained, locally preset frequency-hopping interference model to determine the target frequency-hopping strategy based on the current communication environment. That is, based on the type of interference present in the current communication environment, the corresponding anti-interference carrier frequency can be determined, and then step S12 is executed to implement the hopping strategy, performing the next communication interaction based on the determined carrier frequency. This overcomes the problem of low communication efficiency caused by the inability to adapt to environmental changes, as mentioned in Problem 2 above.

[0134] The following will combine Figures 3 to 15 The effectiveness of the frequency hopping interference model trained using the training method provided in this application in practical applications will be verified.

[0135] The embodiments of this application are applicable to three types of interference environments: comb interference, dynamic probability interference, and tracking interference. Both comb interference and dynamic probability interference have relatively fixed frequency structures. Therefore, these two types of interference can be combined with tracking interference to obtain hybrid interference, and then the anti-interference capability of the frequency hopping interference model trained by this application can be tested using the hybrid interference.

[0136] in, Figure 3 This diagram illustrates the anti-interference performance of a Q-network model trained under comb-like interference conditions using existing technology in a tracking interference environment. The horizontal axis, "step," represents each time slot, and the vertical axis, "Chosen Action," represents the actions selected by the UAV in different time slots. In the diagram, "x" represents the interfered frequency, and "o" represents the frequency selected by the model. The absence of overlap between "x" and "o" indicates that the UAV cannot select the correct frequency hopping strategy in a comb-like interference environment. Squares mark the selected frequency as being interfered with.

[0137] in, Figure 4 This diagram illustrates the anti-interference performance of a Q-network model trained under existing tracking interference conditions in a comb interference environment. The horizontal axis, 'step', represents each time slot, and the vertical axis, 'Frequency', represents the selected frequency hopping carrier frequency. In the diagram, 'x' represents the interfered frequency, and 'o' represents the frequency selected by the model. The absence of overlap between 'x' and 'o' indicates that the UAV cannot select the correct frequency hopping strategy in a comb interference environment. Squares mark the selected frequency as being interfered with.

[0138] in, Figure 5 This diagram illustrates the anti-interference performance of a Q-network model trained under dynamic probabilistic interference conditions using existing technology in a tracking interference environment. The horizontal axis, "step," represents each time slot, and the vertical axis, "Frequency," represents the selected frequency hopping carrier frequency. In the diagram, "x" represents the interfered frequency, and "o" represents the frequency selected by the model. The absence of overlap between "x" and "o" indicates that the UAV device cannot select the correct frequency hopping strategy in a comb-like interference environment. Squares mark the selected frequency as being interfered with.

[0139] in, Figure 6 This diagram illustrates the anti-interference performance of a Q-network model trained under tracking interference conditions using existing technology in a dynamic probabilistic interference environment. The broken lines in the diagram have no specific physical meaning; they are simply used to connect selected points for easier viewing of the results. The horizontal axis represents each time slot, and the vertical axis represents the frequency points in the environment. Darker squares in the diagram indicate a higher probability of interference. Figure 6 It is evident that the frequency points selected by the Q-network model trained by existing technology contain black squares, and thus cannot provide the correct frequency hopping strategy.

[0140] in, Figure 7 This diagram illustrates the anti-interference performance of a dual-depth Q-network model trained under comb-like interference conditions in a tracking interference environment. The horizontal axis represents the number of training epochs, and the vertical axis represents the average accuracy per epoch. It shows that the dual-depth Q-network model reaches stability after 100 training epochs, achieving 100% accuracy.

[0141] To further verify the anti-interference effect of the model, experiments were conducted using the trained frequency-hopping jamming model (i.e., the trained dual deep Q-network model) in mixed jamming environments of tracking jamming and comb jamming, as follows: Figure 8 As shown, Figure 8The diagram illustrates the anti-interference performance of a dual deep Q-network model trained in a comb-like interference environment using the model training method provided in this application. In the diagram, x represents the interfered frequency point, and o represents the frequency point selected by the model. The fact that x and o do not overlap indicates that the UAV device can select the correct frequency hopping strategy in the comb-like interference environment.

[0142] like Figure 9 As shown in the figure, x represents the interfered frequency point, and o represents the frequency point selected by the model. The fact that x and o do not overlap indicates that the UAV device can select the correct frequency hopping strategy in a tracking interference environment. Figure 10 As shown in the figure, x represents the frequency point being interfered with, and o represents the frequency point selected by the model. The fact that x and o do not overlap indicates that the UAV equipment can select the correct frequency hopping strategy in a mixed interference environment of comb interference and tracking interference.

[0143] Among them, such as Figure 11 As shown, the frequency-hopping anti-jamming model trained in the UAV equipment can be tested in a mixed interference environment containing dynamic probabilistic interference and tracking interference. The horizontal axis represents the number of training rounds, i.e., how many rounds of model training have been completed. The vertical axis represents the average accuracy of each training round. It can be seen that the accuracy of the model tends to stabilize after 450 training rounds, and can be stabilized at around 92%.

[0144] in, Figure 12 This diagram illustrates the anti-interference effect of the frequency-hopping interference model against dynamic probabilistic interference. Darker squares indicate a higher probability of interference. The horizontal axis represents each time slot, and the vertical axis represents the frequency points in the environment. In the diagram, a completely black square corresponding to a selected frequency point in a time slot represents complete interference, while lighter colors indicate a lower probability of interference. As can be seen from the diagram, the model almost exclusively selects the frequencies represented by light-colored squares. Figure 6 , Figure 12 , Figure 14 The points on the zigzag line represent the frequency points selected by the model in different time slots. The darker the color, the higher the probability of interference. Correspondingly, the darker the color of the square where the frequency point is located, the higher the probability of interference.

[0145] Figure 13 The diagram illustrates the anti-jamming effect of the frequency-hopping jamming model against tracking interference. The horizontal axis, 'step', represents each time slot, and the vertical axis, 'Frequency', represents the selected frequency-hopping carrier frequency. In the figure, 'x' represents the jammed frequency, and 'o' represents the frequency selected by the model. The fact that 'x' and 'o' do not overlap indicates that the UAV can select the correct frequency-hopping strategy in a tracking interference environment.

[0146] Figure 14This diagram illustrates the anti-jamming effect of the frequency-hopping jamming model in a mixed jamming environment combining dynamic probabilistic jamming and tracking jamming. The horizontal axis represents each time slot, and the vertical axis represents the frequency points in the environment. In the diagram, if the squares on the selected frequency points for the corresponding time slots are completely black, it indicates complete jamming; lighter colors indicate a lower probability of jamming. As can be seen from the diagram, the model almost always selects the frequency points of the lighter-colored squares.

[0147] Figure 15 The diagram illustrates the anti-interference performance of the dual deep Q-network model, the global model, and the initialized dual deep Q-network model, obtained using the training method of the frequency hopping anti-interference model provided in this application. The horizontal axis represents the training epochs, and the vertical axis represents the anti-interference accuracy. The curves from top to bottom represent the anti-interference performance curves for the global model, the dual deep Q-network model, and the initial dual deep Q-network model, respectively. It is evident that the global model's convergence speed is significantly higher than the other two models. This indicates that, compared to the dual deep Q-network model trained in a single environment, federated learning makes more accurate decisions and achieves higher precision in more complex interference environments.

[0148] in, Figure 15 The horizontal axis in Figure 7 , Figure 11 The horizontal axis is different in the two cases. Figure 7 , Figure 11 The middle horizontal axis refers to the number of rounds of federated learning training, while Figure 15 The horizontal axis in the diagram represents the number of training iterations for a single model.

[0149] Secondly, this application provides an adaptive frequency hopping anti-jamming device based on federated learning, which is applied to various unmanned aerial vehicle (UAV) devices in a flight ad hoc network system, such as... Figure 16 As shown, the device 160 includes:

[0150] The frequency hopping strategy determination module 1601 is used to determine the target frequency hopping strategy based on the communication environment of the UAV device using a preset frequency hopping anti-interference model.

[0151] Anti-interference module 1602 is used to execute the target frequency hopping strategy;

[0152] The flight ad hoc network system also includes: base station equipment, and the pre-set frequency hopping anti-interference model is trained through the following steps:

[0153] The UAV device trains a pre-set initial dual deep Q network model based on initial network parameters and interaction information with its environment. The interaction information with the environment includes: its own communication state information, actions taken, immediate rewards obtained, and new state information. The actions taken are carrier frequencies selected under the existing state information.

[0154] The UAV device then uploads the model parameters of the trained dual deep Q-network model to the base station device.

[0155] The base station equipment generates an aggregated network model based on the model parameters uploaded by each of the drone devices, using a federated averaging algorithm, and then sends the aggregated network model to each of the drone devices.

[0156] The UAV device receives the aggregated network model sent by the base station device, performs secondary learning training on the dual deep Q network model based on the aggregated network model, and determines the dual deep Q network model obtained from the secondary learning training as the preset frequency hopping anti-interference model.

[0157] Optionally, the dual-depth Q-network model includes: a behavior network and a target Q-value generation network, wherein the behavior network is used to select the action to be taken, and the target Q-value generation network is used to generate a Q-value based on the action selected by the behavior network; the step of training the preset initial dual-depth Q-network model by the UAV device based on the initialized network parameters and the interaction information with the environment includes:

[0158] Training sample data is obtained from a preset experience replay pool. The training sample data is the interaction information between the UAV equipment and its environment collected in the past.

[0159] The training sample data is input into the preset initial dual-depth Q-network model. The behavior network selects different carrier frequencies under different state information, and the target Q-value generation network outputs the predicted Q-value based on the carrier frequency selected by the behavior network.

[0160] The weights of the behavior network are updated based on the error between the predicted Q-value and the Q-value label until the error between the predicted Q-value and the Q-value label converges, wherein the Q-value label is the training label of the dual deep network model.

[0161] Optionally, updating the weights of the behavioral network based on the error between the predicted Q-value and the Q-value label includes:

[0162] Based on the error between the predicted Q-value and the Q-value label, the weights of the behavioral network are updated using the following weight update equation:

[0163] Q(s,a)←Q(s,a)+α[r+γmax a′ Q(s ′ ,a ′ )-Q(s,a)]

[0164] Where Q(s) ′ ,a ′ Let Q(s,a) be the predicted Q-value, Q(s,a) be the Q-value label, s be the current channel state of the UAV, a be the frequency used by the UAV in its current state s, α be the model's learning rate, and γ be a discount factor used to measure the importance of the reward. ′ For the UAV equipment to be in the channel state of the next time slot, a ′ The frequency selected for the next time slot, r is the frequency at which the UAV device is in s ′ The instantaneous reward obtained.

[0165] Optionally, the step of training a pre-set initial dual-depth Q-network model by the UAV device based on initial network parameters and interaction information with the environment further includes:

[0166] According to the preset training rounds, the parameters of the target Q-value generation network are updated based on the updated parameters of the behavior network.

[0167] Optionally, the step of generating an aggregated network model by the base station device based on the model parameters uploaded by each of the UAV devices and using a federated averaging algorithm includes:

[0168] The model parameters uploaded by each of the aforementioned UAV devices are received, and the received model parameters are input into the following aggregation equation to generate the aggregation network model:

[0169]

[0170] Where ω is the weight parameter of the model, t is the time slot, N is the total number of UAV devices, k1 is each UAV device in the first interference environment, k2 is each UAV device in the second interference environment, and n k1 Let n be the number of drones in the first interference environment. k2 Let be the number of drones in the second interference environment, 1k be the model parameters of the drones in the first interference environment, and 2k be the model parameters of the drones in the second interference environment. The weight parameters of the model in the first disturbance environment at time slot t+1 are... These are the weight parameters of the model in the second disturbance environment at time slot t+1.

[0171] Optionally, the step of receiving the aggregated network model sent by the base station device from the UAV device, and performing secondary learning training on the dual deep Q-network model based on the aggregated network model, includes:

[0172] The global model parameters of the aggregated network model are used as the initial model parameters for the dual deep Q-network model trained in the second learning stage.

[0173] Based on the initial model parameters and the interaction information with the environment, the dual deep Q-network model is trained until the carrier frequency output by the dual deep Q-network model meets the preset anti-interference conditions, wherein the preset anti-interference conditions are determined according to the environment. The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in this application all comply with relevant laws and regulations and do not violate public order and good morals.

[0174] The names of the messages or information exchanged between multiple devices in the embodiments of this application are for illustrative purposes only and are not intended to limit the scope of these messages or information.

[0175] Thirdly, exemplary embodiments of this application also provide an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor. The memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to cause the electronic device to perform a method according to an embodiment of this application.

[0176] An exemplary embodiment of this application also provides a non-transitory computer-readable storage medium storing a computer program, wherein the computer program, when executed by a computer's processor, is used to cause the computer to perform a method according to an embodiment of this application.

[0177] An exemplary embodiment of this application also provides a computer program product, including a computer program, wherein, when executed by a computer's processor, the computer program is used to cause the computer to perform a method according to an embodiment of this application.

[0178] refer to Figure 17The present invention describes a structural block diagram of an electronic device 1700 that can serve as a server or client of this application, which is an example of a hardware device that can be applied to various aspects of this application. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the application described and / or claimed herein.

[0179] like Figure 17 As shown, the electronic device 1700 includes a computing unit 1701, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1702 or a computer program loaded from a storage unit 1708 into a random access memory (RAM) 1703. The RAM 1703 may also store various programs and data required for the operation of the electronic device 1700. The computing unit 1701, ROM 1702, and RAM 1703 are interconnected via a bus 1704. An input / output (I / O) interface 1705 is also connected to the bus 1704.

[0180] Multiple components in electronic device 1700 are connected to I / O interface 1705, including: input unit 1706, output unit 1707, storage unit 1708, and communication unit 1709. Input unit 1706 can be any type of device capable of inputting information to electronic device 1700. Input unit 1706 can receive input digital or character information and generate key signal inputs related to user settings and / or function control of the electronic device. Output unit 1707 can be any type of device capable of presenting information and may include, but is not limited to, a display, speaker, video / audio output terminal, vibrator, and / or printer. Storage unit 1708 may include, but is not limited to, disks and optical discs. Communication unit 1709 allows electronic device 1700 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks, and may include, but is not limited to, modems, network cards, infrared communication devices, wireless communication transceivers, and / or chipsets, such as Bluetooth™ devices, WiFi devices, WiMax devices, cellular communication devices, and / or the like.

[0181] The computing unit 1701 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1701 performs the various methods and processes described above. For example, in some embodiments, the aforementioned federated learning-based adaptive frequency hopping anti-jamming method can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 1708. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 1700 via ROM 1702 and / or communication unit 1709. In some embodiments, the computing unit 1701 can be configured to perform the aforementioned federated learning-based adaptive frequency hopping anti-jamming method by any other suitable means (e.g., by means of firmware).

[0182] The program code used to implement the methods of this application may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0183] In the context of this application, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0184] As used in this application, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, device, and / or apparatus (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0185] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0186] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0187] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other.

Claims

1. A federated learning based adaptive frequency hopping anti-jamming method, characterized in that, The method is applied to various unmanned aerial vehicle (UAV) devices in a flight ad hoc network system, and the method includes: Using a pre-set frequency hopping anti-interference model, the target frequency hopping strategy is determined based on the communication environment of the UAV device, and the target frequency hopping strategy is executed. The flight ad hoc network system also includes: base station equipment, and the pre-set frequency hopping anti-interference model is trained through the following steps: The UAV device trains a pre-set initial dual deep Q network model based on initial network parameters and interaction information with its environment. The interaction information with the environment includes: its own communication state information, actions taken, immediate rewards obtained, and new state information. The actions taken are carrier frequencies selected under the existing state information. The UAV device then uploads the model parameters of the trained dual deep Q-network model to the base station device. The base station equipment generates an aggregated network model based on the model parameters uploaded by each of the drone devices, using a federated averaging algorithm, and then sends the aggregated network model to each of the drone devices. The UAV device receives the aggregated network model sent by the base station device, performs secondary learning training on the dual deep Q network model based on the aggregated network model, and determines the dual deep Q network model obtained from the secondary learning training as the preset frequency hopping anti-interference model. The dual-depth Q-network model includes: a behavior network and a target Q-value generation network, wherein the behavior network is used to select the action to be taken, and the target Q-value generation network is used to generate a Q-value based on the action selected by the behavior network; the training of the pre-set initial dual-depth Q-network model by the UAV device based on the initialized network parameters and the interaction information with the environment includes: Training sample data is obtained from a preset experience replay pool. The training sample data is the interaction information between the UAV equipment and its environment collected in the past. The training sample data is input into the preset initial dual-depth Q-network model. The behavior network selects different carrier frequencies under different state information, and the target Q-value generation network outputs the predicted Q-value based on the carrier frequency selected by the behavior network. The weights of the behavior network are updated based on the error between the predicted Q-value and the Q-value label until the error between the predicted Q-value and the Q-value label converges, wherein the Q-value label is the training label of the dual deep Q-network model; the parameters of the target Q-value generation network are updated according to the updated parameters of the behavior network in a preset training cycle. The process of generating an aggregated network model by the base station equipment based on the model parameters uploaded by each of the UAV equipment and using a federated averaging algorithm includes: The model parameters uploaded by each of the aforementioned UAV devices are received, and the received model parameters are input into the following aggregation equation to generate the aggregation network model: in, These are the weight parameters of the model. For time slots, The total number of drone devices. For each drone device in the first interference environment, For each drone device in the second interference environment, The number of drones in the first interference environment. The number of drones in the second interference environment. These are the model parameters for the UAV equipment in the first interference environment. The model parameters for the UAV equipment in the second interference environment. The weight parameters of the model in the first disturbance environment at time slot t+1 are... The weight parameters of the model in the second interference environment at time slot t+1; the first interference environment and the second interference environment are different types of interference environments, including: comb interference, dynamic probability interference, tracking interference and mixed interference, and the mixed interference includes: interference obtained by mixing the comb interference and the tracking interference, and interference obtained by mixing the dynamic probability interference and the tracking interference.

2. The method of claim 1, wherein, The step of updating the weights of the behavior network based on the error between the predicted Q-value and the Q-value label includes: Based on the error between the predicted Q-value and the Q-value label, the weights of the behavioral network are updated using the following weight update equation: + ] in, To predict the Q value, Here, is the Q-value label, s is the current channel state of the UAV, and a is the frequency point adopted by the UAV in its current state s. The learning rate of the model. This is a discount factor used to measure the importance of the reward. The channel state of the drone equipment in the next time slot, The frequency selected for the next time slot, r is the frequency at which the UAV device is located. The instantaneous reward obtained.

3. The method according to claim 1, characterized in that, The step of receiving the aggregated network model sent by the base station device from the UAV device, and performing secondary learning training on the dual deep Q network model based on the aggregated network model, includes: The global model parameters of the aggregated network model are used as the initial model parameters for the dual deep Q-network model trained in the second learning stage. Based on the initial model parameters and the interaction information with the environment, the dual deep Q network model is trained until the carrier frequency output by the dual deep Q network model meets the preset anti-interference conditions, wherein the preset anti-interference conditions are determined according to the environment.

4. An adaptive frequency hopping anti-interference device based on federated learning, characterized in that, The device is applied to various unmanned aerial vehicle (UAV) devices in a flight ad hoc network system, and the device includes: The frequency hopping strategy determination module is used to determine the target frequency hopping strategy based on the communication environment of the UAV device using a preset frequency hopping anti-interference model. An anti-interference module is used to execute the target frequency hopping strategy; The flight ad hoc network system also includes: base station equipment, and the pre-set frequency hopping anti-interference model is trained through the following steps: The UAV device trains a pre-set initial dual deep Q network model based on initial network parameters and interaction information with its environment. The interaction information with the environment includes: its own communication state information, actions taken, immediate rewards obtained, and new state information. The actions taken are carrier frequencies selected under the existing state information. The UAV device then uploads the model parameters of the trained dual deep Q-network model to the base station device. The base station equipment generates an aggregated network model based on the model parameters uploaded by each of the drone devices, using a federated averaging algorithm, and then sends the aggregated network model to each of the drone devices. The UAV device receives the aggregated network model sent by the base station device, performs secondary learning training on the dual deep Q network model based on the aggregated network model, and determines the dual deep Q network model obtained from the secondary learning training as the preset frequency hopping anti-interference model. The dual-depth Q-network model includes: a behavior network and a target Q-value generation network, wherein the behavior network is used to select the action to be taken, and the target Q-value generation network is used to generate a Q-value based on the action selected by the behavior network; the training of the pre-set initial dual-depth Q-network model by the UAV device based on the initialized network parameters and the interaction information with the environment includes: Training sample data is obtained from a preset experience replay pool. The training sample data is the interaction information between the UAV equipment and its environment collected in the past. The training sample data is input into the preset initial dual-depth Q-network model. The behavior network selects different carrier frequencies under different state information, and the target Q-value generation network outputs the predicted Q-value based on the carrier frequency selected by the behavior network. The weights of the behavior network are updated based on the error between the predicted Q-value and the Q-value label until the error between the predicted Q-value and the Q-value label converges, wherein the Q-value label is the training label of the dual deep Q-network model; the parameters of the target Q-value generation network are updated according to the updated parameters of the behavior network in a preset training cycle. The process of generating an aggregated network model by the base station equipment based on the model parameters uploaded by each of the UAV equipment and using a federated averaging algorithm includes: The model parameters uploaded by each of the aforementioned UAV devices are received, and the received model parameters are input into the following aggregation equation to generate the aggregation network model: in, These are the weight parameters of the model. For time slots, The total number of drone devices. For each drone device in the first interference environment, For each drone device in the second interference environment, The number of drones in the first interference environment. The number of drones in the second interference environment. These are the model parameters for the UAV equipment in the first interference environment. The model parameters for the UAV equipment in the second interference environment. The weight parameters of the model in the first disturbance environment at time slot t+1 are... The weight parameters of the model in the second interference environment at time slot t+1; the first interference environment and the second interference environment are different types of interference environments, including: comb interference, dynamic probability interference, tracking interference and mixed interference, and the mixed interference includes: interference obtained by mixing the comb interference and the tracking interference, and interference obtained by mixing the dynamic probability interference and the tracking interference.

5. The apparatus of claim 4, wherein, The step of receiving the aggregated network model sent by the base station device from the UAV device, and performing secondary learning training on the dual deep Q network model based on the aggregated network model, includes: The global model parameters of the aggregated network model are used as the initial model parameters for the dual deep Q-network model trained in the second learning stage. Based on the initial model parameters and the interaction information with the environment, the dual deep Q network model is trained until the carrier frequency output by the dual deep Q network model meets the preset anti-interference conditions, wherein the preset anti-interference conditions are determined according to the environment.

6. An electronic device, comprising: The electronic device includes: Processor; and Stored program memory, The program includes instructions that, when executed by the processor, cause the processor to perform the method according to any one of claims 1-3.

7. A non-transitory computer-readable storage medium having stored thereon computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-3.

Images