A non-expert data-assisted meta-reinforcement learning method for frequency-hopping jamming resource allocation

Abstract

The present disclosure relates to a non-expert data-assisted meta-reinforcement learning method for frequency-hopping interference resource allocation, comprising: constructing a frequency-hopping interference resource allocation problem based on a communication adversarial scenario; modeling the frequency-hopping interference resource allocation problem as a Markov decision process; randomly initializing a meta-strategy to obtain an initial policy network; and meta-training the initial policy network based on the MAML algorithm. The meta-training includes: in a task-specific adaptation phase, optimizing the initial policy network using a reinforcement learning loss function and a behavioral cloning loss function based on non-expert data to obtain a task-specific policy and calculate the reinforcement learning loss for each interference task; in a meta-optimization phase, minimizing the reinforcement learning loss for all interference tasks to obtain an optimized allocation policy; and fine-tuning the optimized allocation policy based on the new interference task. The present disclosure combines reinforcement learning and behavioral cloning to extract useful information from non-expert data, enabling efficient strategy exploration.

Description

Technical Field

[0001] The present disclosure relates to the field of wireless communication technology, and in particular to a non-expert data-assisted meta-reinforcement learning frequency hopping interference resource allocation method. Background Art

[0002] In the field of wireless communications, wireless sensor networks are widely used in military communications. With the development of spread-spectrum communication technology, frequency-hopping communication, with its advantages such as a wide spectrum occupancy range and extremely high channel hopping rate, is highly resistant to traditional jamming methods such as tracking jamming and frequency sweeping jamming. Currently, the most effective countermeasure to frequency-hopping communication is to employ partial-band blocking jamming. However, to maximize the advantages of partial-band blocking jamming, the key is to address the interference resource allocation issue involved in this approach. Specifically, determining the appropriate center jamming frequency and bandwidth for different jamming devices is crucial, achieving interference suppression against frequency-hopping targets while minimizing the use of jamming resources.

[0003] Among related technologies, deep reinforcement learning methods have demonstrated strong decision-making intelligence and efficiency in the field of communication jamming decision-making. Although some research progress has been made on frequency-hopping communication jamming decision-making, they all require the design of a sophisticated jamming reward function. This not only requires specialized domain knowledge, but the reward function also typically contains many hyperparameters, which often require extensive repeated experiments to determine. Therefore, related technologies have not simultaneously considered two key issues: how to improve the generalization of jamming strategies and how to avoid the cumbersome design of a sophisticated jamming reward function.

[0004] Therefore, it is necessary to provide a new technical solution to improve one or more problems existing in the above solutions.

[0005] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute prior art known to ordinary technicians in the field. Summary of the Invention

[0006] The purpose of the embodiments of the present disclosure is to provide a non-expert data-assisted meta-reinforcement learning frequency hopping interference resource allocation method, which combines reinforcement learning and behavior cloning to extract useful information from non-expert data and achieve efficient strategy exploration.

[0007] According to an embodiment of the present disclosure, a non-expert data-assisted meta-reinforcement learning frequency hopping interference resource allocation method is provided, characterized by comprising:

[0008] Construct the frequency hopping jamming resource allocation problem based on the communication confrontation scenario;

[0009] Modeling the frequency hopping interference resource allocation problem as a Markov decision process;

[0010] Randomly initialize the meta-strategy to obtain the initial policy network;

[0011] Performing meta-training on the initial policy network based on the MAML algorithm, wherein the meta-training includes a specific task adaptation phase and a meta-optimization phase;

[0012] In the specific task adaptation stage, the initial policy network is optimized and trained based on non-expert data using a reinforcement learning loss function and a behavior cloning loss function to obtain a specific task strategy, and the reinforcement learning loss of each interference task is calculated based on the specific task strategy;

[0013] In the meta-optimization stage, the reinforcement learning loss of all interference tasks is minimized, and the optimized allocation strategy is obtained;

[0014] Fine-tune the meta-trained optimized allocation policy based on the new distractor task.

[0015] In an exemplary embodiment of the present disclosure, in the specific task adaptation stage, based on non-expert data, the initial policy network is optimized and trained using a reinforcement learning loss function and a behavior cloning loss function to obtain a specific task policy, and the reinforcement learning loss of each interference task is calculated based on the specific task policy, including the following steps:

[0016] Randomly sample multiple interference tasks, and cyclically perform the following steps on each interference task:

[0017] Using the initial strategy network to interact with each of the interference tasks to obtain first training data;

[0018] Calculating parameters of a specific task strategy using the non-expert data and the first training data;

[0019] According to the specific task strategy, the TRPO algorithm is used to calculate the reinforcement learning loss of each interference task.

[0020] In an exemplary embodiment of the present disclosure, calculating the parameters of a specific task strategy using the non-expert data and the first training data includes:

[0021] Determine the reinforcement learning loss function, the formula is as follows:

[0022]

[0023] in, V(s t ) represents the value function, represents the reinforcement learning loss function, represents the first training data, θ o represents the parameters of the initial policy network, represents the initial policy network, H represents the length of a trajectory, γ represents the discount factor, and λ represents the GAE parameter;

[0024] The behavior cloning loss function is determined as follows:

[0025]

[0026] in, represents the behavior cloning loss function, represents non-expert data, θ o represents the parameters of the initial policy network, represents the initial policy network;

[0027] The parameters of the specific task strategy are obtained by deriving the parameters of the initial strategy network using the reinforcement learning loss function and the behavior cloning loss function. The formula is as follows:

[0028]

[0029] Among them, θ i θ represents the parameters of a specific task strategy, o represents the parameters of the initial policy network, ⊙ represents the Hadamard product, α represents the adaptive learning rate vector, represents the gradient of the initial policy network, w rl Represents the first loss weight, w bc represents the second loss weight.

[0030] In an exemplary embodiment of the present disclosure, according to the specific task strategy, the TRPO algorithm is used to calculate the reinforcement learning loss of each interference task, including the following steps:

[0031] Using the specific task strategy to interact with the plurality of interference tasks to obtain verification data;

[0032] Based on the verification data, the reinforcement learning loss is calculated using the TRPO algorithm.

[0033] In an exemplary embodiment of the present disclosure, in the meta-optimization stage, minimizing the reinforcement learning loss of all interfering tasks and obtaining an optimized allocation strategy includes the following steps:

[0034] Minimize the reinforcement learning loss of all interference tasks through formula (4):

[0035]

[0036] Among them, θ i θ represents the parameters of a specific task strategy, o represents the parameters of the initial policy network, ⊙ represents the Hadamard product, α represents the adaptive learning rate vector, represents the gradient of the initial policy network, Represents verification data;

[0037] The TRPO algorithm is used to solve the formula (4) to obtain the optimized allocation strategy.

[0038] In an exemplary embodiment of the present disclosure, the fine-tuning stage includes the following steps:

[0039] Using the optimized allocation strategy to interact with the new interference task to obtain second training data and example data corresponding to the new interference task;

[0040] According to the second training data and the example data, a gradient descent step is performed on the allocation strategy after meta-training optimization to obtain fine-tuned and updated strategy network parameters.

[0041] In an exemplary embodiment of the present disclosure, in the meta-training, it is assumed that the actions sampled from the non-expert data have an advantage value relative to the actions sampled from the initial policy network. The advantage value can be described as:

[0042]

[0043] Wherein, ω represents the value of the advantage, represents the initial policy network, represents the action sampled from non-expert data, a t represents the actions sampled from the initial policy network, Indicates the advantage that the action sampled from non-expert data can obtain over other actions in the state, Indicates the advantage that the action sampled from the initial policy network can obtain over other actions in the state, represents the interference task distribution, represents the tasks sampled from the interfering task distribution.

[0044] In an exemplary embodiment of the present disclosure, modeling the frequency hopping interference resource allocation problem as a Markov decision process includes:

[0045] defining the action space, state space, and reward function of the Markov decision process;

[0046] A discrete Markov decision process is constructed according to the action space, the state space, and the reward function.

[0047] In an exemplary embodiment of the present disclosure, the reward function can be described as:

[0048]

[0049] Among them, r(s t ,a t ) represents the reward function, e n (t) represents the transient interference effect of the currently constructed interference scheme on the nth frequency hopping frequency set, a t represents the action of the interferer, s t represents the state of the environment, κ represents the interference coverage coefficient, represents the frequency hopping frequency set, Indicates the number of frequency hopping frequency sets.

[0050] In an exemplary embodiment of the present disclosure, in the step of constructing a frequency hopping interference resource allocation problem according to a communication countermeasure scenario, the frequency hopping interference resource allocation problem can be expressed as follows after formulating:

[0051]

[0052] Where sgn(·) is the sign function. When the independent variable x>0, sgn(x)=1. represents the interference plus noise signal ratio of the frequency hopping channel at time t, is the total number of channels in the nth frequency hopping frequency set, Indicates that all interfering nodes are connected with f u,n (t) is the overall JNSR generated by the hopping channel of the center frequency, κ is the interference coverage factor, is the maximum interference power, is the maximum signal power, is the set of central interference frequencies, is the set of interference bandwidths.

[0053] The technical solutions provided by the present disclosure may have the following beneficial effects:

[0054] In the embodiments of the present disclosure, through the above method, on the one hand, by meta-training the initial policy network based on the MAML algorithm, initial network parameters with good generalization ability are obtained, and then through fine-tuning steps, a small amount of interaction data of the new interference task is combined to perform gradient descent for several steps, so that the optimized allocation strategy can be quickly adapted to the new task and the generalization of the interference strategy is improved; on the other hand, in the specific task adaptation stage of meta-training, the initial policy network is improved based on non-expert data by jointly utilizing reinforcement learning and behavioral cloning, which can achieve efficient optimization of the interference strategy in a sparse interference reward environment and realize efficient strategy exploration.

[0055] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] The accompanying drawings are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification, serve to explain the principles of the present disclosure. Obviously, the drawings described below are only some embodiments of the present disclosure, and those skilled in the art can derive other drawings based on these drawings without inventive effort.

[0057] Figure 1 A flowchart showing the steps of a non-expert data-assisted meta-reinforcement learning frequency hopping interference resource allocation method in an exemplary embodiment of the present disclosure is shown;

[0058] Figure 2 A schematic diagram showing a comparison of cumulative interference rewards obtained by different methods in an exemplary embodiment of the present disclosure during a meta-training process of a simulation experiment;

[0059] Figure 3 A schematic diagram showing a comparison of the cumulative interference rewards obtained by the method proposed in this application in an exemplary embodiment of the present disclosure when facing different tasks in the fine-tuning phase of a simulation experiment;

[0060] Figure 4 A schematic diagram showing a comparison of the jamming success rates obtained by the method proposed in the present application in an exemplary embodiment of the present disclosure when faced with a previously encountered task in a fine-tuning phase of a simulation experiment;

[0061] Figure 5 A schematic diagram showing a comparison of the interference success rates obtained by the method proposed in this application in an exemplary embodiment of the present disclosure when faced with an unseen task in the fine-tuning phase of a simulation experiment;

[0062] Figure 6 A schematic diagram showing a comparison of the cumulative interference rewards of the method proposed in this application in an ablation experiment in an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0063] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be embodied in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete and will fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0064] In addition, the accompanying drawings are merely schematic illustrations of the present disclosure and are not necessarily drawn to scale. Identical reference numerals in the figures denote identical or similar parts, and thus repetitive descriptions thereof will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities that do not necessarily correspond to physically or logically separate entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0065] This example embodiment first provides a non-expert data-assisted meta-reinforcement learning frequency hopping interference resource allocation method. Figure 1 As shown in , the method may include the following steps:

[0066] Step S101: constructing a frequency hopping interference resource allocation problem according to a communication countermeasure scenario;

[0067] Step S102: Modeling the frequency hopping interference resource allocation problem as a Markov decision process;

[0068] Step S103: Randomly initialize the meta-strategy to obtain an initial strategy network;

[0069] Step S104: performing meta-training on the initial policy network based on the MAML algorithm, wherein the meta-training includes a specific task adaptation phase and a meta-optimization phase;

[0070] In the specific task adaptation phase, the initial policy network is optimized and trained using reinforcement learning and behavior cloning based on non-expert data to obtain a specific task strategy, and the reinforcement learning loss of each interference task is calculated based on the specific task strategy;

[0071] In the meta-optimization stage, the reinforcement learning loss of all interference tasks is minimized, and the optimized allocation strategy is obtained;

[0072] Step S105: fine-tune the allocation strategy optimized by meta-training according to the new interference task.

[0073] Through the above method, on the one hand, by meta-training the initial policy network based on the MAML algorithm, initial network parameters with good generalization ability are obtained. Then, through fine-tuning steps and combining a small amount of interaction data from the new interference task with several steps of gradient descent, the optimized allocation strategy can be quickly adapted to the new task, thereby improving the generalization of the interference strategy. On the other hand, in the specific task adaptation stage of meta-training, the initial policy network is improved based on non-expert data by jointly utilizing reinforcement learning and behavioral cloning, which can achieve efficient optimization of the interference strategy in a sparse interference reward environment and realize efficient strategy exploration.

[0074] Below, we will refer to Figures 1 to 6 Each step of the above method in this exemplary embodiment is described in more detail.

[0075] In one embodiment, the communication confrontation scenario in step S101 is a ground-to-air many-to-many communication confrontation scenario. In this communication confrontation scenario, the user adopts frequency hopping spread spectrum communication and uses N frequency hopping frequency sets to build N frequency hopping communication links. To indicate that, Represents all frequency hopping channels under the nth frequency set, Indicates the center frequency of the i-th channel in the n-th frequency set used by the user, The total transmit power of each frequency hopping communication link is Evenly distributed on all frequency hopping channels under the frequency hopping frequency set used by the link, that is, Among them, b u represents the bandwidth of the frequency hopping channel, and U(f) represents the power spectral density function of the communication signal.

[0076] Specifically, in the above communication confrontation scenario, the jammer uses M ground jammer nodes to jam the user's N frequency hopping communication links. Represents the index of the interference node. Each interference node can be in the interference bandwidth set Select an interference bandwidth from and select a center interference frequency To form an interference frequency band on the spectrum, that is, The transmission power of the interference node is evenly distributed in this interference frequency band, that is, Where J(f) is the power spectral density function of the interference signal, is the maximum transmit power of the interference node. Considering the power density function of the noise is N(f), the center frequency of the mth interference node can be calculated by formula (8): The jamming-plus-noise-to-signal ratio (JNSR) generated by the frequency hopping channel is:

[0077]

[0078] in, Represents the indicator function, when The interference frequency is centered on The interference frequency band of the interference bandwidth can completely cover As the center frequency, b u When the frequency hopping channel is the channel bandwidth, otherwise G u represents the channel gain of the communication link, G j represents the channel gain of the interfering link, F b is the filtering loss coefficient, ρ is the polarization loss coefficient; L u and L j represent the path propagation losses of the communication signal and the interference signal respectively.

[0079] It should be noted that the path propagation loss L of the above communication signal is u and the path propagation loss L of the interference signal j , can be calculated by formula (9):

[0080]

[0081] Where d represents the signal propagation distance, and f represents the signal carrier frequency.

[0082] It should also be noted that, in this embodiment, d u Indicates the propagation distance of the communication signal, using d j Indicates the propagation distance of the interference signal.

[0083] Specifically, in the aforementioned communication countermeasure scenario, the interference coverage coefficient κ is further introduced. For a specific frequency hopping communication link, when more than κ of all frequency hopping channels in the frequency hopping frequency set used by it are interfered with, the frequency hopping communication link can be considered to be completely suppressed. From this, it can be deduced that the formula for successfully determining the interference caused by M interfering nodes on the nth frequency hopping frequency set is:

[0084]

[0085] Where sgn(·) is the sign function. When the independent variable x>0, sgn(x)=1. represents the interference plus noise signal ratio of the frequency hopping channel at time t, is the total number of channels in the nth frequency hopping frequency set, Indicates that all interfering nodes are connected with f u,n (t) is the overall JNSR generated by the hopping channel of the center frequency, and κ is the interference coverage factor.

[0086] It should be noted that the jammer's goal is to reasonably determine the central jamming frequency and jamming bandwidth combination of the M jamming nodes to interfere with as many frequency hopping links as possible. Therefore, in the above communication confrontation scenario, the frequency hopping interference resource allocation problem can be formulated based on this goal and expressed as the following optimization problem:

[0087]

[0088] Where sgn(·) is the sign function. When the independent variable x>0, sgn(x)=1. represents the interference plus noise signal ratio of the frequency hopping channel at time t, is the total number of channels in the nth frequency hopping frequency set, Indicates that all interfering nodes are connected with f u,n (t) is the overall JNSR generated by the hopping channel of the center frequency, κ is the interference coverage factor, is the maximum interference power, is the maximum signal power, is the set of central interference frequencies, is the set of interference bandwidths;

[0089] It should also be noted that for the above optimization problem, the solution space size is Obviously, the solution space grows exponentially with the number of interfering nodes M. Traditional optimization algorithms cannot cope with such a large solution space. In addition, when users are interfered with maliciously, they will take anti-interference measures to protect the communication quality. It is assumed that after being interfered with, users will activate another set of new frequency sets for these communication links. In this new set of frequencies, the spectrum distribution of the frequency-hopping channels has changed. Therefore, it is a new interference task for the interferer, and the above optimization problem needs to be solved again, which consumes a lot of computing resources and is not smart enough.

[0090] In this embodiment, in order to reasonably and optimally allocate interference resources of multiple interfering devices and achieve simultaneous interference on multiple frequency hopping frequency sets, this embodiment models the frequency hopping interference resource allocation problem as a Markov Decision Process (MDP).

[0091] Furthermore, step S102 may include: defining an action space, a state space, and a reward function of a Markov decision process, and constructing a discrete Markov decision process according to the action space, the state space, and the reward function.

[0092] Specifically, the action is defined as the combination of the central interference frequency and the interference bandwidth, and the action space A can be expressed as:

[0093]

[0094] in, and They represent the central interference frequency and interference bandwidth selected by the tth interference node, and t∈{0,1,L,M-1}, A represents the action space, a t Indicates the action of the interferer.

[0095] Specifically, after each interfering node decides on an interference plan, it immediately calculates the interference effects of all interfering nodes that have previously completed decisions, including the interfering node, and uses this transient interference effect as the state of the next interfering node. Therefore, the state space is defined as:

[0096] s t =[e0(t),e1(t),L,e n (t),L,e N-1 (t)]∈S(12)

[0097] in, S is the state space, s t Indicates the state of the environment.

[0098] Specifically, the reward function is the key to solving the above optimization problem using reinforcement learning. In this application, a rough binary reward is directly used based on whether the final optimization goal is achieved or not. The reward function can be described as:

[0099]

[0100] Among them, r(s t ,a t ) represents the reward function, e n (t) represents the transient interference effect of the currently constructed interference scheme on the nth frequency hopping frequency set, a t represents the action of the interferer, s t represents the state of the environment, κ represents the interference coverage coefficient, represents the frequency hopping frequency set, Indicates the number of frequency hopping frequency sets.

[0101] It should be noted that the reward function used in this application is very simple for the purpose of demonstration. For comparison, the expression of a professional reward function designed in the prior art is given:

[0102]

[0103] Comparing Formula (6) and Formula (13), it can be clearly seen that Formula (13) contains more manual experience, and the design and parameter adjustment process is very cumbersome. However, this sophisticated reward function represents a dense reward environment and requires professional domain knowledge as support. It also requires a large number of verification experiments to determine the characteristic parameters in the reward function, which undoubtedly requires a large amount of manpower and computing resources. The reward function provided by this application is more concise and intuitive, and easy to understand, but it also creates a sparse reward environment. Since the rewards are only 0 and 1, the network's policy gradients will be the same in the initial stage, resulting in the inability to improve the policy, which increases the difficulty of interference policy optimization.

[0104] It should be noted that when the frequency hopping interference resource allocation problem is modeled as a Markov decision process, the same set of interference strategy networks is used to make decisions for each interference node in a time-division manner to meet the memoryless feature of the Markov decision process.

[0105] Regarding the simple reward function adopted in the above step S102, in order to achieve efficient optimization of interference decisions in the face of new tasks in a binary sparse reward environment, this application, after step S103, through step S104, in the specific task adaptation stage of meta-training, jointly utilizes reinforcement learning and behavioral cloning to improve the initial policy network based on non-expert data, so as to achieve efficient optimization of interference strategies in a sparse interference reward environment.

[0106] For example, MAML is a multi-task meta-learning algorithm that can use different models and quickly adapt between different tasks. Therefore, MAML can be applied to many fields, including classification, regression, and reinforcement learning. Reinforcement Learning (RL) is a paradigm in the field of machine learning. Without prior knowledge, the intelligent agent takes actions and interacts with the environment through trial and error to obtain environmental rewards, and then adjusts its own behavior strategy based on the environmental rewards. Through repeated iterations, it eventually converges to the optimal or suboptimal strategy for the current environment. TRPO stands for Trust Region Policy Optimization. The TRPO algorithm is an algorithm for reinforcement learning that aims to ensure the stability of learning by limiting the size of policy updates.

[0107] This example, building on the MAML algorithm's training framework, improves the generalization of the allocation policy by simultaneously optimizing the total loss of multiple interfering tasks during meta-training. Using the MAML algorithm's training process as a reference, we designed two optimization steps: meta-training and fine-tuning. During meta-training, we train an optimized allocation policy. During fine-tuning, we only need to fine-tune the parameters of the meta-trained allocation policy to adapt it to new tasks.

[0108] Optionally, the following steps may be included before step S103:

[0109] Input the number of interference nodes M and the interference bandwidth set Maximum interference power Maximum signal power Interference task collection and non-expert data and other parameters;

[0110] Set the maximum number of meta-training rounds, the number of distractor tasks sampled per meta-training, the number of trajectories sampled from each distractor task, the learning rate during meta-training, and the learning rate during fine-tuning.

[0111] In one embodiment, the meta-training process in step S104 is performed multiple times, and the task-specific adaptation phase and the meta-optimization phase are performed for each meta-training cycle. In each meta-training iteration, the task-specific adaptation phase is performed first, and then the meta-optimization phase is performed.

[0112] In one embodiment, in step S103, after a random initialization process, the parameters of the meta-strategy are randomly initialized to the parameters θ0 of the initial policy network, and the learning rate α of the meta-training process is initialized. This embodiment uses θ0 as the optimization starting point to meta-train the initial policy network.

[0113] Furthermore, in the specific task adaptation stage, based on non-expert data, the initial policy network is optimized and trained using a reinforcement learning loss function and a behavior cloning loss function to obtain a specific task strategy, and the reinforcement learning loss of each interference task is calculated based on the specific task strategy, including the following steps:

[0114] Step S1041: randomly sample T interference tasks, and cyclically perform the following steps on each of the interference tasks:

[0115] Step S1042: using the initial strategy network to interact with each of the interference tasks to obtain first training data;

[0116] Step S1043: Calculating parameters of a specific task strategy using the non-expert data and the first training data;

[0117] Step S1044: Calculate the reinforcement learning loss of each interference task using the TRPO algorithm according to the specific task strategy.

[0118] In one embodiment, step S1043 calculates the parameters of the specific task strategy using the non-expert data and the first training data, which is achieved by the following steps:

[0119] Determine the reinforcement learning loss function, the formula is as follows:

[0120]

[0121] in, V(s t ) represents the value function, represents the reinforcement learning loss function, represents the first training data, θ o represents the parameters of the initial policy network, represents the initial policy network, H represents the length of a trajectory, γ represents the discount factor, and λ represents the GAE parameter;

[0122] Calculate the behavior cloning loss function, the formula is as follows:

[0123]

[0124] in, represents the behavior cloning loss function, represents non-expert data, θ o represents the parameters of the initial policy network, represents the initial policy network;

[0125] The parameters of the specific task strategy are obtained by deriving the parameters of the initial strategy network using the reinforcement learning loss function and the behavior cloning loss function. The formula is as follows:

[0126]

[0127] Among them, θ i θ represents the parameters of a specific task strategy, o represents the parameters of the initial policy network, ⊙ represents the Hadamard product, α represents the adaptive learning rate vector, represents the gradient of the initial policy network, w rl Represents the first loss weight, w bc represents the second loss weight.

[0128] In order to efficiently optimize the initial policy network even in a sparsely interfering reward environment, reinforcement learning and behavioral cloning are used to improve the initial policy network based on non-expert data during the aforementioned task-specific adaptation phase. Since the MAML algorithm is compatible with any model trained using gradient descent, the loss from reinforcement learning and the loss from behavioral cloning can be directly incorporated into the loss function to provide a meaningful policy gradient. Specifically, reinforcement learning is performed using the first training data obtained by interacting with the initial policy network and each interfering task, and policy adaptation based on reinforcement learning is enhanced through behavioral cloning of non-expert data, thereby guiding the policy update for the specific task in the right direction under sparse rewards. After performing a one-step gradient descent, the aforementioned task-specific policy is obtained.

[0129] In one embodiment, step S1044 calculates the reinforcement learning loss of each interference task using the TRPO algorithm according to the specific task strategy, and is implemented by the following steps:

[0130] Utilizing the specific task strategy to interact with T interference tasks to obtain verification data;

[0131] Based on the verification data, the reinforcement learning loss is calculated using the TRPO algorithm.

[0132] The reinforcement learning loss of each interference task calculated through the above steps makes it easy to directly minimize the reinforcement learning loss of all interference tasks in the meta-optimization stage, thereby obtaining an allocation strategy that meets the goal of minimizing the reinforcement learning loss of all interference tasks. This strategy is the optimized allocation strategy obtained in the meta-training process.

[0133] It should be noted that in the task-specific adaptation phase of meta-training, in order to improve learning efficiency, the parameters θ of the initial policy network are specifically o A learnable adaptive learning rate vector α is configured and set to be equal to θ o Vectors with the same dimension, let α and θ o Perform meta-optimization together, that is, when θ o When optimized, α can be adaptively adjusted. Compared with the fixed scalar learning rate set in the prior art, this learnable adaptive learning rate vector α can make the parameters of the policy network more efficiently learned.

[0134] It should also be noted that the vector The direction of represents the update direction of the initial policy network, the vector The length of represents the learning rate of the initial policy network. Through this improvement, the parameter update of the initial policy network will not rigidly follow the gradient direction, but the parameters θ of the initial policy network o Each parameter in is configured with an independent update direction and learning rate, which can accelerate the optimization of the initial policy network.

[0135] This embodiment provides non-expert data Where p is the index of the trajectory. It should be noted that this embodiment makes the following assumptions for non-expert data:

[0136] In the meta-training, the actions sampled from the non-expert data are set Relative to the action a sampled from the initial policy network t has an advantage value, which can be described as:

[0137]

[0138] Wherein, ω represents the value of the advantage, represents the initial policy network, represents the action sampled from non-expert data, a t represents the actions sampled from the initial policy network, Indicates the advantage that the action sampled from non-expert data can obtain over other actions in the state, Indicates the advantage that the action sampled from the initial policy network can obtain over other actions in the state, represents the interference task distribution, represents the tasks sampled from the interfering task distribution.

[0139] It should be understood that for the above assumptions, since the initial policy network is untrained and close to a random policy, the decision experience of a non-professional interference decision maker can be regarded as non-expert data. Since the decision experience of a non-professional interference decision maker is always better than a random policy, the above assumptions are easy to meet in practice.

[0140] Furthermore, since the reinforcement learning loss of each interference task has been calculated based on the verification data using the TRPO algorithm in step S1044, in the meta-optimization phase, the reinforcement learning loss of all interference tasks can be minimized to obtain an optimized allocation strategy, which can specifically include the following sub-steps:

[0141] Step S1045: Minimize the reinforcement learning loss of all interference tasks using formula (4):

[0142]

[0143] Among them, θ i θ represents the parameters of a specific task strategy, o represents the parameters of the initial policy network, ⊙ represents the Hadamard product, α represents the adaptive learning rate vector, represents the gradient of the initial policy network, Represents verification data;

[0144] Step S1046: Solve the formula (4) using the TRPO algorithm to obtain the optimized allocation strategy.

[0145] In the embodiments of the present disclosure, the TRPO algorithm is used to solve formula (4), and through multiple iterative optimizations, an allocation strategy and learning rate vector with good generalization performance can be obtained. It should be noted that the process of solving formula (4) is a standard TRPO solution process, so the solution result is directly given here, as shown in the following formulas (14) to (16):

[0146]

[0147] Where l represents the minimum value that satisfies the KL divergence constraint, and l∈{0,1,2,L,L}, L is the maximum number of backtracking steps, σ represents the backtracking coefficient, and θ o′ represents the updated policy parameters, δ represents the trust region radius, represents the policy gradient, The Hessian matrix representing the gradient, represents the intermediate transition term, is the mathematical expectation symbol, which means that In the sampling trajectory, τ represents the trajectory, and H represents the length of the trajectory.

[0148] In this embodiment, the allocation strategy obtained through meta-training can adapt to new interference tasks by performing several gradient descent steps based on the strategy parameters and learning rate vector trained in the meta-training process.

[0149] In one embodiment, the fine-tuning stage of step S105 may include the following steps:

[0150] Step S1051: Using the optimized allocation strategy and the new interference task τ new Interacting to obtain second training data and example data corresponding to the new interference task;

[0151] Step S1052: performing gradient descent processing on the allocation strategy optimized by meta-training according to the second training data and the example data to obtain fine-tuned updated strategy network parameters.

[0152] In step S1052, the allocation strategy after meta-training optimization can be subjected to gradient descent processing using the following formula (17):

[0153]

[0154] Among them, θ new represents the policy network parameters after fine-tuning the update, represents the second training data, represents the reinforcement learning loss function corresponding to the second training data, represents sample data, Represents the behavior cloning loss function corresponding to the example data.

[0155] Furthermore, the following steps may be included before step S1051:

[0156] Input the optimized allocation policy after meta-training and the new distractor task τ that has never been seen new .

[0157] After the allocation strategy network in the embodiment of the present disclosure undergoes two stages of training, namely meta-training and fine-tuning, the obtained strategy network parameters θ new In the new interference task τ new Demonstrated excellent adaptability.

[0158] The specific algorithm of the frequency hopping interference resource allocation method based on meta-deep reinforcement learning proposed in this application is shown in Table 1 below:

[0159] Table 1 Frequency hopping interference resource allocation algorithm based on Meta-TRPO algorithm

[0160]

[0161]

[0162] In the embodiment of the present application, in order to further verify the effect of the non-expert data-assisted meta-reinforcement learning frequency hopping interference resource allocation method proposed in the present application, the following simulation experiment was conducted:

[0163] The parameters of this simulation experiment are set as follows:

[0164] The user uses N = 5 frequency hopping communication links. When the user finds that the communication quality is poor, the frequency hopping frequency set used by these communication links can be changed. In the simulation experiment, it is assumed that the center frequency of all frequency hopping channels in each frequency set will be shifted downward by S. n , and the degree of deviation of each frequency set follows a uniform distribution S n ~U(0,1), so the interference task can be formalized as and Then the corresponding frequency group becomes The jammer is equipped with M = 12 jamming nodes, and the available jamming bandwidth set is It is assumed that after the user changes the frequency hopping frequency set used, the jammer can promptly use electronic warfare support technology to detect the center frequencies of all the changed frequency hopping channels. is 46dBm, 82dBm, b u 25kHz, G j 18dB, G u 22dB, F b is 1, ρ is 0.8, κ is 0.3, d u 180KM, d j The distance is 250KM and N(f) is -164dBm / Hz.

[0165] The hyperparameter settings for this simulation experiment are as follows:

[0166] All methods used a three-layer multilayer perceptron with 128 neurons in each hidden layer, using the ReLU activation function and the Adam optimizer for optimization. The training task was 8, and the number of sampled interaction trajectories, K, was 16. The learning rate during fine-tuning was 0.001.

[0167] In order to verify the excellent performance of the method proposed in this application, the following is an illustration of the comparison of various methods, which are as follows:

[0168] Example method: Policy generated by a partially trained (non-converged) LSTM-A3C model under the constraints of the specialized reward function expressed in Equation (13).

[0169] Expert method: The policy generated by a fully trained and converged LSTM-A3C model under the expert reward function expressed in formula (13).

[0170] Meta-TRPO-dense method: Using the Meta-TRPO-dense model, training is performed under the constraints of the professional reward function expressed in formula (13). Dense can be understood as a data processing technology based on matrix operations.

[0171] Meta-TRPO-sparse method: Using the Meta-TRPO-sparse model, training is performed under the constraints of the rough reward function expressed in formula (6). Sparse can be understood as an approximate sparse regression algorithm.

[0172] Meta-BC method: Before the task adaptation stage, an improved MAML algorithm of behavior cloning (BC) is performed on the example data and trained under the rough reward function expressed in formula (6).

[0173] EMRLD method: Utilize the meta-reinforcement learning algorithm based on EMRLD and train under the constraints of the rough reward function expressed in formula (6).

[0174] Continuous spectrum blocking interference method: A frequency hopping interference method in which the interference bandwidth is continuous and the central interference frequency is switched in a sweeping manner. Each interference node selects the maximum interference bandwidth, and the interference frequency bands of each interference node are connected into a longer and continuous interference interval, and then the central interference frequency is selected in a sweeping manner.

[0175] The evaluation indicators for comparing the performance of various methods are as follows:

[0176] Cumulative interference reward: The cumulative interference reward is the sum of the interference rewards obtained by all interference nodes in a interference test.

[0177] Interference success rate: The interference success rate is the proportion of all frequency hopping links that can be successfully interfered with simultaneously in 100 interference tests.

[0178] All methods are evaluated under dense rewards and repeated three times with different random seeds. The mean and standard deviation of the three experiments are calculated as the final results.

[0179] Analysis of the results of this simulation experiment:

[0180] First, the cumulative interference rewards of each method during meta-training are compared. Figure 2 As shown in . Figure 2 It can be seen that the proposed method shows the fastest convergence speed and the largest cumulative interference reward without the need for a professional interference reward function. Although the proposed method relies on the guidance of examples in the early stage of meta-training, it can quickly exceed the performance of the examples. The meta-TRPO-sparse method can hardly optimize the interference strategy when it is in a sparse interference reward, and the optimization speed of the interference strategy of the Meta-TRPO-dense method is still relatively slow when it is in a dense interference reward. The Meta-BC method has a faster convergence speed by imitating examples, but imitating examples is not enough to fully learn the useful information in the examples. It can be seen that the cumulative interference reward that the Meta-BC method can achieve when it converges is still lower than the method proposed in this application. In addition, the traditional continuous spectrum blocking interference method can hardly achieve an effective interference effect on the frequency hopping link because it ignores the cooperation of different interference nodes in deciding the interference scheme. It should be noted that all methods cannot reach the level of experts, because the methods based on MAML variants still need to be further improved through subsequent fine-tuning to further improve performance.

[0181] Next, we further demonstrate the performance of the method proposed in this application in the fine-tuning stage. We divide the new interference tasks into tasks that have been seen in meta-training and tasks that have not been seen. Figure 3 It can be seen that compared with unseen tasks, the proposed method can adapt relatively faster to reach the level of experts for seen tasks. Figure 4 and Figure 5 The success rate of the proposed method against interference of each frequency hopping link is given in detail. Figure 4 As can be seen from the figure, the proposed algorithm only needs 50 simple gradient descent steps to achieve an average interference success rate of 91% in the tasks seen. Figure 5 As can be seen in Figure 3, the proposed algorithm only needs 50 simple gradient descent steps to achieve an average interference success rate of 82% in unseen tasks.

[0182] The following table gives the intuitive statistical values ​​of various evaluation indicators of the method proposed in this application in the fine-tuning stage. When facing unseen tasks, when the number of training trajectories reaches 800 times, a relatively considerable cumulative interference reward and interference success rate can be achieved. When facing seen tasks, when the number of training trajectories reaches 80 times, a fairly considerable cumulative interference reward and interference success rate can be achieved.

[0183] Table 2 Intuitive statistical values ​​of each evaluation index in the fine-tuning stage

[0184] Number of trajectories for training Cumulative Interference Reward Interference success rate Unseen missions 800 213.28±58.9 0.82±0.24 Seen tasks 80 232.89±26.61 0.91±0.11 expert - 240.00 1.00 Example - 79.00 0.25

[0185] Finally, an ablation experiment related to the proposed algorithm was conducted, and the results are shown in Figure 6 As shown in . Figure 6 It can be seen that adding behavioral cloning loss to the loss function can significantly improve the cumulative interference reward obtained by the proposed method, and using adaptive updates instead of a fixed learning rate can also improve the reward. In addition, combining hot start with behavioral cloning can greatly improve the convergence speed of the proposed method, which is guided by useful information in the example.

[0186] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the appended claims.

Claims

1. A non-expert data-assisted meta-reinforcement learning frequency hopping interference resource allocation method, characterized in that: include: Construct the frequency hopping jamming resource allocation problem based on the communication confrontation scenario; Modeling the frequency hopping interference resource allocation problem as a Markov decision process; Randomly initialize the meta-strategy to obtain the initial policy network; Performing meta-training on the initial policy network based on the MAML algorithm, wherein the meta-training includes a specific task adaptation phase and a meta-optimization phase; In the specific task adaptation stage, the initial policy network is optimized and trained based on non-expert data using a reinforcement learning loss function and a behavior cloning loss function to obtain a specific task strategy, and the reinforcement learning loss of each interference task is calculated based on the specific task strategy; In the meta-optimization stage, the reinforcement learning loss of all interference tasks is minimized, and the optimized allocation strategy is obtained; Fine-tune the allocation strategy optimized by meta-training based on the new distractor task; In the specific task adaptation stage, based on non-expert data, the initial policy network is optimized and trained using a reinforcement learning loss function and a behavior cloning loss function to obtain a specific task strategy, and the reinforcement learning loss of each interference task is calculated based on the specific task strategy, including the following steps: Randomly sample multiple interference tasks, and cyclically perform the following steps on each interference task: Using the initial strategy network to interact with each of the interference tasks to obtain first training data; Calculating parameters of a specific task strategy using the non-expert data and the first training data; According to the specific task strategy, the reinforcement learning loss of each interference task is calculated using the TRPO algorithm; The calculating of the parameters of the specific task strategy using the non-expert data and the first training data includes: The behavior cloning loss function is determined as follows: （2） in, represents the behavior cloning loss function, represents non-expert data, represents the parameters of the initial policy network, represents the initial policy network; In the meta-optimization stage, the reinforcement learning loss of all interference tasks is minimized by formula (4): （4） in, Parameters representing a specific task strategy, represents the parameters of the initial policy network, represents the Hadamard product, represents the adaptive learning rate vector, represents the gradient of the initial policy network, Represents verification data.

2. The non-expert data-assisted meta-reinforcement learning frequency hopping interference resource allocation method according to claim 1, characterized in that: The calculating parameters of a specific task strategy using the non-expert data and the first training data includes: Determine the reinforcement learning loss function, the formula is as follows: （1） in, , represents the value function, , represents the reinforcement learning loss function, represents the first training data, represents the parameters of the initial policy network, represents the initial policy network, H represents the length of a trajectory, represents the discount factor, Indicates GAE parameters; The parameters of the specific task strategy are obtained by deriving the parameters of the initial strategy network using the reinforcement learning loss function and the behavior cloning loss function. The formula is as follows: （3） in, Parameters representing a specific task strategy, represents the parameters of the initial policy network, represents the Hadamard product, represents the adaptive learning rate vector, represents the gradient of the initial policy network, represents the first loss weight, represents the second loss weight.

3. The non-expert data-assisted meta-reinforcement learning frequency hopping interference resource allocation method according to claim 2, characterized in that: According to the specific task strategy, the TRPO algorithm is used to calculate the reinforcement learning loss of each interference task, including the following steps: Using the specific task strategy to interact with the plurality of interference tasks to obtain verification data; Based on the verification data, the reinforcement learning loss is calculated using the TRPO algorithm.

4. The non-expert data-assisted meta-reinforcement learning frequency hopping interference resource allocation method according to claim 3, characterized in that: In the meta-optimization stage, the reinforcement learning loss of all interference tasks is minimized and the optimized allocation strategy is obtained, including the following steps: The TRPO algorithm is used to solve the formula (4) to obtain the optimized allocation strategy.

5. The non-expert data-assisted meta-reinforcement learning frequency hopping interference resource allocation method according to claim 1, characterized in that: The fine-tuning stage includes the following steps: Using the optimized allocation strategy to interact with the new interference task to obtain second training data and example data corresponding to the new interference task; According to the second training data and the example data, a gradient descent step is performed on the allocation strategy after meta-training optimization to obtain fine-tuned and updated strategy network parameters.

6. The non-expert data-assisted meta-reinforcement learning frequency hopping interference resource allocation method according to claim 1, characterized in that: In the meta-training, it is assumed that the actions sampled from the non-expert data have an advantage value relative to the actions sampled from the initial policy network. The advantage value can be described as: （5） in, represents the value of the advantage, represents the initial policy network, represents actions sampled from non-expert data, represents the actions sampled from the initial policy network, Indicates the advantage that the action sampled from non-expert data can obtain over other actions in the state, Indicates the advantage that the action sampled from the initial policy network can obtain over other actions in the state, represents the interference task distribution, represents the tasks sampled from the interfering task distribution.

7. The non-expert data-assisted meta-reinforcement learning frequency hopping interference resource allocation method according to claim 1, characterized in that: Modeling the frequency hopping interference resource allocation problem as a Markov decision process includes: defining the action space, state space, and reward function of the Markov decision process; A discrete Markov decision process is constructed according to the action space, the state space, and the reward function.

8. The non-expert data-assisted meta-reinforcement learning frequency hopping interference resource allocation method according to claim 7, characterized in that: The reward function can be described as: （6） in, represents the reward function, represents the transient interference effect of the currently constructed interference scheme on the nth frequency hopping frequency set, Indicates the action of the interfering party, Indicates the state of the environment, represents the interference coverage coefficient, represents the frequency hopping frequency set, Indicates the number of frequency hopping frequency sets.

9. The non-expert data-assisted meta-reinforcement learning frequency hopping interference resource allocation method according to claim 1, characterized in that: In the step of constructing a frequency hopping interference resource allocation problem according to the communication countermeasure scenario, the frequency hopping interference resource allocation problem can be expressed as follows after formulating: （7） in, is a symbolic function, when the independent variable hour, ; represents the interference plus noise signal ratio of the frequency hopping channel at time t, is the total number of channels in the nth frequency hopping frequency set, Indicates that all interfering nodes are The overall JNSR generated by the hopping channel for the center frequency, represents the interference coverage coefficient, is the maximum interference power, is the maximum signal power, is the set of central interference frequencies, is the set of interference bandwidths.

Images