Non-expert data assisted meta reinforcement learning frequency hopping interference resource allocation method

Abstract

The embodiment of the invention relates to a non-expert data assisted meta reinforcement learning frequency hopping interference resource allocation method. The method comprises the following steps: constructing a frequency hopping interference resource allocation problem according to a communication confrontation scene; modeling a frequency hopping interference resource allocation problem as a Markov decision process; randomly initializing the element strategy to obtain an initial strategy network; meta-training is carried out on the initial strategy network based on an MAML algorithm, and the meta-training comprises the steps that in the specific task adaptation stage, based on non-expert data, optimization training is carried out on the initial strategy network through a reinforcement learning loss function and a behavior cloning loss function, a specific task strategy is obtained, and reinforcement learning loss of all interference tasks is calculated; in the meta-optimization stage, the reinforcement learning loss of all interference tasks is minimized, and an optimized distribution strategy is obtained; and carrying out fine adjustment on the optimized distribution strategy according to the new interference task. According to the method, useful information is extracted from non-expert data in combination with reinforcement learning and behavior cloning, and efficient strategy exploration is realized.

Description

Technical Field

[0001] The present disclosure relates to the field of wireless communication technology, and in particular to a non-expert data-assisted meta-reinforcement learning frequency hopping interference resource allocation method. Background Art

[0002] In the field of wireless communication technology, wireless sensor networks are widely used in the field of military communications. With the development of spread spectrum communication technology, frequency hopping communication can resist traditional interference methods such as tracking interference and frequency sweeping interference by virtue of its advantages such as wide spectrum occupancy range and extremely high channel hopping rate. The current effective measure to deal with frequency hopping communication is to adopt the partial frequency band blocking interference method. However, if the advantages of partial frequency band blocking interference are to be maximized, the key is to solve the interference resource allocation problem involved in the partial frequency band blocking interference method, that is, to decide the appropriate central interference frequency and interference bandwidth for different interference devices, so as to achieve interference suppression of frequency hopping targets under the premise of consuming as little interference resources as possible.

[0003] In the related technologies, deep reinforcement learning methods have demonstrated strong decision-making intelligence and efficiency in the field of communication interference decision-making. Although certain research progress has been made on the problem of frequency-hopping communication interference decision-making, it is necessary to design a sophisticated interference reward function, which not only requires professional domain knowledge as support, but also the reward function usually contains many hyperparameters, which often require a large number of repeated experiments to determine. Therefore, the related technologies do not simultaneously consider two key issues: how to improve the generalization of interference strategies and how to avoid the cumbersome design of sophisticated interference reward functions.

[0004] Therefore, it is necessary to provide a new technical solution to improve one or more problems existing in the above solutions.

[0005] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0006] The purpose of the disclosed embodiments is to provide a non-expert data-assisted meta-reinforcement learning frequency hopping interference resource allocation method, which combines reinforcement learning and behavior cloning to extract useful information from non-expert data and achieve efficient strategy exploration.

[0007] According to an embodiment of the present disclosure, a non-expert data-assisted meta-reinforcement learning frequency hopping interference resource allocation method is provided, which is characterized by comprising:

[0008] Construct the frequency hopping jamming resource allocation problem based on the communication confrontation scenario;

[0009] Modeling the frequency hopping interference resource allocation problem as a Markov decision process;

[0010] Randomly initialize the meta-strategy to get the initial strategy network;

[0011] Performing meta-training on the initial policy network based on the MAML algorithm, wherein the meta-training includes a specific task adaptation phase and a meta-optimization phase;

[0012] Wherein, in the specific task adaptation stage, based on non-expert data, the initial strategy network is optimized and trained using a reinforcement learning loss function and a behavior cloning loss function to obtain a specific task strategy, and the reinforcement learning loss of each interference task is calculated based on the specific task strategy;

[0013] In the meta-optimization stage, the reinforcement learning losses of all interference tasks are minimized, and an optimized allocation strategy is obtained;

[0014] Fine-tune the allocation strategy optimized by meta-training based on the new distractor task.

[0015] In an exemplary embodiment of the present disclosure, in the specific task adaptation stage, based on non-expert data, the initial strategy network is optimized and trained using a reinforcement learning loss function and a behavior cloning loss function to obtain a specific task strategy, and the reinforcement learning loss of each interference task is calculated based on the specific task strategy, including the following steps:

[0016] A plurality of interference tasks are randomly sampled, and the following steps are performed cyclically for each of the interference tasks:

[0017] Using the initial strategy network to interact with each of the interference tasks to obtain first training data;

[0018] Calculating parameters of a specific task strategy using the non-expert data and the first training data;

[0019] According to the specific task strategy, the TRPO algorithm is used to calculate the reinforcement learning loss of each interference task.

[0020] In an exemplary embodiment of the present disclosure, the calculating the parameters of a specific task strategy by using the non-expert data and the first training data includes:

[0021] Determine the reinforcement learning loss function, the formula is as follows:

[0022]

[0023] in, V(s t ) represents the value function, represents the reinforcement learning loss function, represents the first training data, θ o represents the parameters of the initial policy network, represents the initial policy network, H represents the length of a trajectory, γ represents the discount factor, and λ represents the GAE parameter;

[0024] The behavior cloning loss function is determined as follows:

[0025]

[0026] in, represents the behavior cloning loss function, represents non-expert data, θ o represents the parameters of the initial policy network, represents the initial policy network;

[0027] The parameters of the specific task strategy are obtained by using the reinforcement learning loss function and the behavior cloning loss function to derive the parameters of the initial strategy network. The formula is as follows:

[0028]

[0029] Among them, θ i represents the parameters of a specific task strategy, θ o represents the parameters of the initial policy network, ⊙ represents the Hadamard product, α represents the adaptive learning rate vector, represents the gradient of the initial policy network, w rl represents the first loss weight, w bc represents the second loss weight.

[0030] In an exemplary embodiment of the present disclosure, according to the specific task strategy, the reinforcement learning loss of each interference task is calculated using the TRPO algorithm, including the following steps:

[0031] Using the specific task strategy to interact with the plurality of interference tasks to obtain verification data;

[0032] Based on the verification data, the reinforcement learning loss is calculated using the TRPO algorithm.

[0033] In an exemplary embodiment of the present disclosure, in the meta-optimization stage, minimizing the reinforcement learning loss of all interfering tasks and obtaining an optimized allocation strategy includes the following steps:

[0034] The reinforcement learning loss of all interference tasks is minimized by formula (4):

[0035]

[0036] Among them, θ i represents the parameters of a specific task strategy, θ o represents the parameters of the initial policy network, ⊙ represents the Hadamard product, α represents the adaptive learning rate vector, represents the gradient of the initial policy network, Represents verification data;

[0037] The TRPO algorithm is used to solve the formula (4) to obtain the optimized allocation strategy.

[0038] In an exemplary embodiment of the present disclosure, the fine-tuning stage includes the following steps:

[0039] Using the optimized allocation strategy to interact with the new interference task to obtain second training data and example data corresponding to the new interference task;

[0040] According to the second training data and the example data, a gradient descent step is performed on the allocation strategy after meta-training optimization to obtain fine-tuned and updated policy network parameters.

[0041] In an exemplary embodiment of the present disclosure, in the meta-training, it is assumed that the actions sampled from the non-expert data have an advantage value relative to the actions sampled from the initial policy network, and the advantage value can be described as:

[0042]

[0043] Where ω represents the value of the advantage, represents the initial policy network, represents the actions sampled from non-expert data, a t represents the actions sampled from the initial policy network, Indicates the advantage that the action sampled from non-expert data can obtain over other actions in the state, Indicates the advantage that the action sampled from the initial policy network can obtain over other actions in the state, represents the interference task distribution, represents the tasks sampled from the interfering task distribution.

[0044] In an exemplary embodiment of the present disclosure, modeling the frequency hopping interference resource allocation problem as a Markov decision process includes:

[0045] defining an action space, a state space, and a reward function for the Markov decision process;

[0046] A discrete Markov decision process is constructed according to the action space, the state space and the reward function.

[0047] In an exemplary embodiment of the present disclosure, the reward function can be described as:

[0048]

[0049] Among them, r(s t ,a t ) represents the reward function, e n (t) represents the transient interference effect of the currently constructed interference scheme on the nth frequency hopping frequency set, a t represents the action of the interferer, s t represents the state of the environment, κ represents the interference coverage coefficient, represents the frequency hopping frequency set, Indicates the number of frequency hopping frequency sets.

[0050] In an exemplary embodiment of the present disclosure, in the step of constructing a frequency hopping interference resource allocation problem according to a communication confrontation scenario, the frequency hopping interference resource allocation problem can be expressed as follows after being formulated:

[0051]

[0052] Where sgn(·) is a sign function, when the independent variable x>0, sgn(x)=1; represents the interference plus noise signal ratio of the frequency hopping channel at time t, is the total number of channels in the nth frequency hopping frequency set, Indicates that all interfering nodes are connected with f u,n (t) is the overall JNSR generated by the hopping channel of the center frequency, κ is the interference coverage factor, is the maximum interference power, is the maximum signal power, is the set of central interference frequencies, is the set of interference bandwidths.

[0053] The technical solution provided by the present disclosure may have the following beneficial effects:

[0054] In the embodiments of the present disclosure, through the above method, on the one hand, by meta-training the initial policy network based on the MAML algorithm, the initial network parameters with good generalization ability are obtained, and then through the fine-tuning step, a small amount of interaction data of the new interference task is combined to perform a gradient descent of several steps, so that the optimized allocation strategy can be quickly adapted to the new task and the generalization of the interference strategy is improved; on the other hand, in the specific task adaptation stage of meta-training, the initial policy network is improved based on non-expert data by combining reinforcement learning and behavior cloning, which can achieve efficient optimization of the interference strategy in a sparse interference reward environment and achieve efficient strategy exploration.

[0055] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0057] Figure 1 A flowchart showing the steps of the non-expert data-assisted meta-reinforcement learning frequency hopping interference resource allocation method in an exemplary embodiment of the present disclosure;

[0058] Figure 2 A schematic diagram showing a comparison of cumulative interference rewards obtained by different methods in an exemplary embodiment of the present disclosure during a meta-training process of a simulation experiment;

[0059] Figure 3 A schematic diagram showing a comparison of the cumulative interference rewards obtained by the method proposed in the present application in an exemplary embodiment of the present disclosure when facing different tasks in the fine-tuning stage of a simulation experiment;

[0060] Figure 4 A schematic diagram showing a comparison of the interference success rates obtained by the method proposed in the present application in an exemplary embodiment of the present disclosure when facing a task that has been seen in the fine-tuning stage of a simulation experiment;

[0061] Figure 5 A schematic diagram showing a comparison of the interference success rates obtained by the method proposed in the present application in an exemplary embodiment of the present disclosure when facing an unseen task in the fine-tuning stage of a simulation experiment;

[0062] Figure 6 A schematic diagram showing a comparison of the cumulative interference rewards of the method proposed in the present application in an exemplary embodiment of the present disclosure in an ablation experiment. DETAILED DESCRIPTION

[0063] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the disclosure will be more comprehensive and complete and to fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0064] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0065] In this example implementation, a non-expert data-assisted meta-reinforcement learning frequency hopping interference resource allocation method is first provided. Figure 1 As shown in , the method may include the following steps:

[0066] Step S101: construct a frequency hopping interference resource allocation problem according to a communication confrontation scenario;

[0067] Step S102: Modeling the frequency hopping interference resource allocation problem as a Markov decision process;

[0068] Step S103: randomly initialize the meta-strategy to obtain an initial strategy network;

[0069] Step S104: performing meta-training on the initial policy network based on the MAML algorithm, wherein the meta-training includes a specific task adaptation phase and a meta-optimization phase;

[0070] Wherein, in the specific task adaptation stage, based on non-expert data, the initial strategy network is optimized and trained using reinforcement learning and behavior cloning to obtain a specific task strategy, and the reinforcement learning loss of each interference task is calculated based on the specific task strategy;

[0071] In the meta-optimization stage, the reinforcement learning losses of all interference tasks are minimized, and an optimized allocation strategy is obtained;

[0072] Step S105: fine-tune the allocation strategy optimized by meta-training according to the new interference task.

[0073] Through the above method, on the one hand, by meta-training the initial policy network based on the MAML algorithm, the initial network parameters with good generalization ability are obtained, and then through the fine-tuning step, a small amount of interaction data of the new interference task is combined to perform several steps of gradient descent, so that the optimized allocation strategy can quickly adapt to the new task and improve the generalization of the interference strategy; on the other hand, in the specific task adaptation stage of meta-training, the initial policy network is improved based on non-expert data by jointly using reinforcement learning and behavior cloning, which can achieve efficient optimization of the interference strategy in a sparse interference reward environment and realize efficient strategy exploration.

[0074] Next, we will refer to Figures 1 to 6 Each step of the above method in this example implementation is described in more detail.

[0075] In one embodiment, the communication confrontation scenario in step S101 is a ground-to-air many-to-many communication confrontation scenario. In this communication confrontation scenario, the user adopts frequency hopping spread spectrum communication and uses N frequency hopping frequency sets to build N frequency hopping communication links. To indicate that, represents all frequency hopping channels under the nth frequency set, represents the center frequency of the ith channel in the nth frequency set used by the user, The total transmission power of each frequency hopping communication link is Evenly distributed on all frequency hopping channels under the frequency hopping frequency set used by the link, that is, Among them, b u represents the bandwidth of the frequency hopping channel, and U(f) represents the power spectral density function of the communication signal.

[0076] Specifically, in the above communication confrontation scenario, the jammer uses M ground jammers to carry out jamming attacks on the user's N frequency hopping communication links. Represents the index of the interfering node. Each interfering node can be in the interference bandwidth set Select an interference bandwidth and select a center interference frequency To form an interference frequency band on the spectrum, that is The transmission power of the interfering node is evenly distributed in this interference frequency band, that is, Where J(f) is the power spectral density function of the interference signal, is the maximum transmit power of the interfering node. Considering the power density function of the noise is N(f), the power density of the mth interfering node to the center frequency can be calculated by formula (8): The Jamming-plus-noise-to-signal ratio (JNSR) generated by the frequency hopping channel is:

[0077]

[0078] in, Represents the indicator function. The interference frequency is the center frequency. The interference frequency band of the interference bandwidth can completely cover is the center frequency, b u When the frequency hopping channel is the channel bandwidth, otherwise G u represents the channel gain of the communication link, G j represents the channel gain of the interfering link, F b is the filtering loss coefficient, ρ is the polarization loss coefficient; L u and L j Represent the path propagation loss of communication signal and interference signal respectively.

[0079] It should be noted that the path propagation loss L of the above communication signal is u and the path propagation loss L of the interference signal j , can be calculated by formula (9):

[0080]

[0081] Where d represents the signal propagation distance and f represents the signal carrier frequency.

[0082] It should also be noted that in this embodiment, d u Indicates the propagation distance of the communication signal, using d j Indicates the propagation distance of the interference signal.

[0083] Specifically, in the above communication confrontation scenario, the interference coverage coefficient κ is further introduced. For a specific frequency hopping communication link, when more than κ of all frequency hopping channels under the frequency hopping frequency set used by it are interfered, the frequency hopping communication link can be considered to be completely suppressed. It can be deduced that the interference success judgment formula formed by M interference nodes on the nth frequency hopping frequency set is:

[0084]

[0085] Where sgn(·) is a sign function, when the independent variable x>0, sgn(x)=1; represents the interference plus noise signal ratio of the frequency hopping channel at time t, is the total number of channels in the nth frequency hopping frequency set, Indicates that all interfering nodes are connected with f u,n (t) is the overall JNSR generated by the hopping channel of the center frequency, and κ represents the interference coverage factor.

[0086] It should be noted that the goal of the jammer is to reasonably decide the combination of the central jamming frequency and the jamming bandwidth of the M jamming nodes to interfere with as many frequency hopping links as possible. Therefore, in the above communication confrontation scenario, the frequency hopping interference resource allocation problem can be formulated based on this goal and expressed as the following optimization problem:

[0087]

[0088] Where sgn(·) is a sign function, when the independent variable x>0, sgn(x)=1; represents the interference plus noise signal ratio of the frequency hopping channel at time t, is the total number of channels in the nth frequency hopping frequency set, Indicates that all interfering nodes are connected with f u,n (t) is the overall JNSR generated by the hopping channel of the center frequency, κ is the interference coverage factor, is the maximum interference power, is the maximum signal power, is the set of central interference frequencies, is the set of interference bandwidths;

[0089] It should also be noted that for the above optimization problem, the solution space size is Obviously, the solution space grows exponentially with the number of interfering nodes M. Traditional optimization algorithms cannot cope with such a large solution space. In addition, when users are interfered with maliciously, they will take anti-interference measures to protect the communication quality. It is assumed that after being interfered with, users will enable another set of new frequency sets for these communication links. In this new set of frequencies, the spectrum distribution of the frequency-hopping channels has changed. Therefore, it is a new interference task for the interferer, and it is necessary to solve the above optimization problem again, which is very computationally intensive and not smart enough.

[0090] In this embodiment, in order to reasonably and optimally allocate interference resources of multiple interference devices and implement simultaneous interference of multiple frequency hopping frequency sets, this embodiment models the frequency hopping interference resource allocation problem as a Markov decision process (MDP).

[0091] Furthermore, step S102 may include: defining an action space, a state space, and a reward function of a Markov decision process, and constructing a discrete Markov decision process according to the action space, the state space, and the reward function.

[0092] Specifically, the action is defined as the combination of the central interference frequency and the interference bandwidth, and the action space A can be expressed as:

[0093]

[0094] in, and They represent the central interference frequency and interference bandwidth selected by the tth interference node, respectively, and t∈{0,1,L,M-1}, A represents the action space, a t Indicates the action of the interferer.

[0095] Specifically, after each interfering node decides on an interference plan, it immediately calculates the interference effects of all interfering nodes that have completed decisions before, including the interfering node, and uses the transient interference effect as the state of the next interfering node. Thus, the state space is defined as:

[0096] s t =[e 0 (t),e 1 (t),L,e n (t),L,e N-1 (t)]∈S(12)

[0097] in, S is the state space, s t Indicates the state of the environment.

[0098] Specifically, the reward function is the key to solving the above optimization problem by using reinforcement learning. In this application, a rough binary reward based on whether the final optimization goal is achieved or not is directly used. The reward function can be described as:

[0099]

[0100] Among them, r(s t ,a t ) represents the reward function, e n (t) represents the transient interference effect of the currently constructed interference scheme on the nth frequency hopping frequency set, a t represents the action of the interferer, s t represents the state of the environment, κ represents the interference coverage coefficient, represents the frequency hopping frequency set, Indicates the number of frequency hopping frequency sets.

[0101] It should be noted that the reward function used in this application is very simple in order to show that it is very simple. For comparison, the expression of a professional reward function designed in the prior art is given:

[0102]

[0103] Comparing formula (6) and formula (13), it can be clearly seen that formula (13) contains more manual experience, and the design and parameter adjustment process is very cumbersome, but this kind of fine reward function represents a dense reward environment, which needs to be supported by professional domain knowledge, and needs to be determined through a large number of verification experiments. The characteristic parameters in the reward function undoubtedly require a lot of manpower and computing resources. The reward function provided by this application is more concise and intuitive, easy to understand, but at the same time it will form a sparse reward environment; since the rewards are only 0 and 1, the network's policy gradients will be the same in the initial stage, resulting in the inability to improve the strategy, which increases the difficulty of interference strategy optimization.

[0104] It should be noted that when the frequency hopping interference resource allocation problem is modeled as a Markov decision process, the same set of interference strategy networks is used to make decisions for each interference node in a time-division manner to meet the memoryless characteristics of the Markov decision process.

[0105] Regarding the simple reward function adopted in the above step S102, in order to achieve efficient optimization of interference decisions in the face of new tasks in a binary sparse reward environment, this application, after step S103, through step S104, in the specific task adaptation stage of meta-training, jointly utilizes reinforcement learning and behavior cloning to improve the initial policy network based on non-expert data, so as to achieve efficient optimization of interference strategies in a sparse interference reward environment.

[0106] For example, MAML is a multi-task meta-learning algorithm that can use different models and quickly adapt between different tasks. Therefore, MAML can be applied to many fields, including classification, regression, and reinforcement learning. Reinforcement Learning (RL) is a paradigm in the field of machine learning. Without prior knowledge, the agent takes actions and interacts with the environment through trial and error to obtain environmental rewards, and then adjusts its own behavior strategy according to the environmental rewards. Through repeated iterations, it finally converges to the optimal or suboptimal strategy for the current environment. TRPO stands for Trust Region Policy Optimization. The TRPO algorithm is an algorithm for reinforcement learning that aims to ensure the stability of learning by limiting the size of policy updates.

[0107] Based on the training framework of the MAML algorithm, this embodiment improves the generalization ability of the allocation strategy by simultaneously optimizing the total loss of multiple interference tasks during meta-training. With reference to the training process of the MAML algorithm, two optimization processes, meta-training and fine-tuning, are designed. In the meta-training process, we train to obtain an optimized allocation strategy, and then in the fine-tuning stage, we only need to perform a few fine-tuning steps on the parameters of the allocation strategy optimized by meta-training to adapt to new tasks.

[0108] Optionally, the following steps may be included before step S103:

[0109] Input the number of interference nodes M and the interference bandwidth set Maximum interference power Maximum signal power Distracting Task Sets and Non-Expert Data And other parameters;

[0110] Set the maximum number of meta-training rounds, the number of distractor tasks sampled per meta-training, the number of trajectories sampled from each distractor task, the learning rate during meta-training, and the learning rate during the fine-tuning phase.

[0111] In one embodiment, the meta-training process in step S104 is performed multiple times, and the specific task adaptation phase and the meta-optimization phase are performed for each meta-training cycle, and in each meta-training iteration, the specific task adaptation phase is performed first, and then the meta-optimization phase is performed.

[0112] In one embodiment, in step S103, after a random initialization process, the parameters of the meta-strategy are randomly initialized to the parameters θ of the initial strategy network. 0 , and initialize the learning rate α of the meta-training process. In this embodiment, θ 0 To optimize the starting point, the initial policy network is meta-trained.

[0113] Furthermore, in the specific task adaptation stage, based on non-expert data, the initial strategy network is optimized and trained using a reinforcement learning loss function and a behavior cloning loss function to obtain a specific task strategy, and the reinforcement learning loss of each interference task is calculated based on the specific task strategy, including the following steps:

[0114] Step S1041: randomly sample T interference tasks, and cyclically perform the following steps on each of the interference tasks:

[0115] Step S1042: using the initial strategy network to interact with each of the interference tasks to obtain first training data;

[0116] Step S1043: Calculating parameters of a specific task strategy using the non-expert data and the first training data;

[0117] Step S1044: According to the specific task strategy, the reinforcement learning loss of each interference task is calculated using the TRPO algorithm.

[0118] In one embodiment, step S1043 uses the non-expert data and the first training data to calculate the parameters of the specific task strategy, which is achieved by the following steps:

[0119] Determine the reinforcement learning loss function, the formula is as follows:

[0120]

[0121] in, V(s t ) represents the value function, represents the reinforcement learning loss function, represents the first training data, θ o represents the parameters of the initial policy network, represents the initial policy network, H represents the length of a trajectory, γ represents the discount factor, and λ represents the GAE parameter;

[0122] Calculate the behavior cloning loss function, the formula is as follows:

[0123]

[0124] in, represents the behavior cloning loss function, represents non-expert data, θ o represents the parameters of the initial policy network, represents the initial policy network;

[0125] The parameters of the specific task strategy are obtained by using the reinforcement learning loss function and the behavior cloning loss function to derive the parameters of the initial strategy network. The formula is as follows:

[0126]

[0127] Among them, θ i represents the parameters of a specific task strategy, θ o represents the parameters of the initial policy network, ⊙ represents the Hadamard product, α represents the adaptive learning rate vector, represents the gradient of the initial policy network, w rl represents the first loss weight, w bc represents the second loss weight.

[0128] In order to efficiently optimize the initial policy network in a sparse interference reward environment, in the above-mentioned task-specific adaptation stage, reinforcement learning and behavior cloning are used together to improve the initial policy network based on non-expert data. Since the MAML algorithm is compatible with any model trained using gradient descent, the loss of reinforcement learning and the loss of behavior cloning can be directly merged into the loss function to provide a meaningful policy gradient. Specifically, reinforcement learning is performed using the first training data obtained by the interaction between the initial policy network and each interference task, and reinforcement learning-based policy adaptation is enhanced by behavioral cloning of non-expert data, thereby guiding the policy update of the specific task in the right direction under sparse rewards. After performing a step of gradient descent, the above-mentioned specific task policy can be obtained.

[0129] In one embodiment, step S1044 calculates the reinforcement learning loss of each interference task using the TRPO algorithm according to the specific task strategy, and is implemented by the following steps:

[0130] Using the specific task strategy to interact with T interference tasks to obtain verification data;

[0131] Based on the verification data, the reinforcement learning loss is calculated using the TRPO algorithm.

[0132] The reinforcement learning loss of each interference task calculated through the above steps makes it easy to directly minimize the reinforcement learning loss of all interference tasks in the meta-optimization stage, thereby obtaining an allocation strategy that meets the goal of minimizing the reinforcement learning loss of all interference tasks. This strategy is the optimized allocation strategy obtained in the meta-training process.

[0133] It should be noted that in the specific task adaptation stage of meta-training, in order to improve learning efficiency, the parameters θ of the initial policy network are specially o A learnable adaptive learning rate vector α is configured and set to be equal to θ o Vectors with the same dimension, let α and θ o Perform meta-optimization together, that is, when θ o When optimized, α can be adjusted adaptively. Compared with the fixed scalar learning rate set in the prior art, this learnable adaptive learning rate vector α can make the parameters of the policy network learn more efficiently.

[0134] It should also be noted that the vector The direction of represents the update direction of the initial policy network, the vector The length of represents the learning rate of the initial policy network. With this improvement, the parameter update of the initial policy network will not rigidly follow the gradient direction, but the parameters θ of the initial policy network oEach parameter in is configured with an independent update direction and learning rate, which can accelerate the optimization of the initial policy network.

[0135] This example provides non-expert data Where p is the index of the trajectory. It should be noted that this embodiment makes the following assumptions about non-expert data:

[0136] In the meta-training, the actions sampled from the non-expert data are set Relative to the action a sampled from the initial policy network t has an advantage value, which can be described as:

[0137]

[0138] Where ω represents the value of the advantage, represents the initial policy network, represents the actions sampled from non-expert data, a t represents the actions sampled from the initial policy network, Indicates the advantage that the action sampled from non-expert data can obtain over other actions in the state, Indicates the advantage that the action sampled from the initial policy network can obtain over other actions in the state, represents the interference task distribution, represents the tasks sampled from the distribution of distractor tasks.

[0139] It should be understood that for the above assumptions, since the initial policy network is untrained and close to a random policy, the decision experience of a non-professional interference decision maker can be regarded as non-expert data, because the decision experience of a non-professional interference decision maker is always better than a random policy, so the above assumptions are easy to meet in practice.

[0140] Furthermore, since the reinforcement learning loss of each interference task has been calculated based on the verification data by using the TRPO algorithm through the aforementioned step S1044, therefore, in the meta-optimization stage, the reinforcement learning loss of all interference tasks can be minimized directly to obtain the optimized allocation strategy, which can specifically include the following sub-steps:

[0141] Step S1045: Minimize the reinforcement learning loss of all interference tasks by formula (4):

[0142]

[0143] Among them, θ i represents the parameters of a specific task strategy, θ orepresents the parameters of the initial policy network, ⊙ represents the Hadamard product, α represents the adaptive learning rate vector, represents the gradient of the initial policy network, Represents verification data;

[0144] Step S1046: using the TRPO algorithm to solve the formula (4) to obtain the optimized allocation strategy.

[0145] In the embodiment of the present disclosure, the TRPO algorithm is used to solve formula (4), and the allocation strategy and learning rate vector with good generalization performance can be obtained through multiple iterative optimization. It should be noted that the process based on the above-mentioned solution of formula (4) is a standard TRPO solution process, so the solution result is directly given here, as shown in the following formulas (14) to (16):

[0146]

[0147] Where l represents the minimum value that satisfies the KL divergence constraint, and l∈{0,1,2,L,L}, L is the maximum number of backtracking steps, σ represents the backtracking coefficient, and θ o ′ represents the updated policy parameters, δ represents the trust region radius, represents the policy gradient, The Hessian matrix representing the gradient, represents the intermediate transition term, is the mathematical expectation symbol, which means that In the sampling trajectory, τ represents the trajectory, and H represents the length of the trajectory.

[0148] In this embodiment, the allocation strategy obtained by meta-training can adapt to new interference tasks by performing a number of gradient descent steps based on the strategy parameters and learning rate vectors trained in the meta-training process.

[0149] In one embodiment, in the fine-tuning stage of step S105, the following steps may be included:

[0150] Step S1051: Using the optimized allocation strategy and the new interference task τ new Interacting to obtain second training data and example data corresponding to the new interference task;

[0151] Step S1052: According to the second training data and the example data, a gradient descent step is performed on the allocation strategy after meta-training optimization to obtain fine-tuned and updated strategy network parameters.

[0152] In step S1052, the allocation strategy after meta-training optimization can be subjected to gradient descent processing by the following formula (17):

[0153]

[0154] Among them, θ new represents the policy network parameters after fine-tuning the update, represents the second training data, represents the reinforcement learning loss function corresponding to the second training data, represents sample data, Represents the behavior cloning loss function corresponding to the example data.

[0155] Furthermore, the following steps may be included before step S1051:

[0156] Input the optimized allocation policy after meta-training and the new distractor task τ that has never been seen new .

[0157] After the allocation strategy network in the embodiment of the present disclosure undergoes two stages of training, namely, meta-training and fine-tuning, the obtained strategy network parameters θ new In the new interference task τ new Demonstrated excellent adaptability.

[0158] The specific algorithm of the frequency hopping interference resource allocation method based on meta-deep reinforcement learning proposed in this application is shown in Table 1 below:

[0159] Table 1 Frequency hopping interference resource allocation algorithm based on Meta-TRPO algorithm

[0160]

[0161]

[0162] In the embodiment of the present application, in order to further verify the effect of the non-expert data-assisted meta-reinforcement learning frequency hopping interference resource allocation method proposed in the present application, the following simulation experiment was carried out:

[0163] The parameters of this simulation experiment are set as follows:

[0164] The user uses N = 5 frequency hopping communication links in total. When the user finds that the communication quality is poor, the frequency hopping frequency set used by these communication links can be changed. In the simulation experiment, it is assumed that the center frequency of all frequency hopping channels in each frequency set will be shifted downward by S as a whole. n , and the degree of deviation of each frequency set follows a uniform distribution S n ~U(0,1), so the interference task can be formalized as and The corresponding frequency group becomes The jammer is equipped with M = 12 jammer nodes, and the available jammer bandwidth set is It is assumed that after the user changes the frequency hopping frequency set used, the jammer can promptly use electronic warfare support technology to detect the center frequencies of all changed frequency hopping channels. 46dBm, is 82dBm, b u 25kHz, G j 18dB, G u 22dB, F b is 1, ρ is 0.8, κ is 0.3, d u 180KM, d j It is 250KM and N(f) is -164dBm / Hz.

[0165] The hyperparameter settings for this simulation experiment are as follows:

[0166] All methods use a three-layer multilayer perceptron with 128 neurons in each hidden layer, and use the ReLU activation function and the Adam optimizer to optimize the neural network. The number of training tasks is 8, and the number of sampled interaction trajectories K is 16. The learning rate in the fine-tuning stage is 0.001.

[0167] In order to verify the excellent performance of the method proposed in this application, the following is explained by comparing various methods, and the various methods are as follows:

[0168] Example method: Policy generated by a partially trained (non-converged) LSTM-A3C model under the professional reward function expressed in Equation (13).

[0169] Expert method: The policy generated by a fully trained and converged LSTM-A3C model under the expert reward function expressed in formula (13).

[0170] Meta-TRPO-dense method: Using the Meta-TRPO-dense model, training is performed under the professional reward function expressed in formula (13). Dense can be understood as a data processing technology based on matrix operations.

[0171] Meta-TRPO-sparse method: Using the Meta-TRPO-sparse model, training is performed under the rough reward function expressed in formula (6). Sparse can be understood as an approximate sparse regression algorithm.

[0172] Meta-BC method: Before the task adaptation stage, an improved MAML algorithm of behavior cloning (BC) is performed on the example data and trained under the rough reward function expressed in formula (6).

[0173] EMRLD method: Utilize the meta-reinforcement learning algorithm based on EMRLD and train under the constraints of the rough reward function expressed in formula (6).

[0174] Continuous spectrum blocking interference method: a frequency hopping interference method in which the interference bandwidth is continuous and the central interference frequency is switched in a sweeping manner. Each interference node selects the maximum interference bandwidth, and connects the interference frequency bands of each interference node into a longer and continuous interference interval, and then selects the central interference frequency in a sweeping manner.

[0175] The evaluation indicators for comparing the performance of various methods are as follows:

[0176] Cumulative interference reward: The cumulative interference reward is the sum of the interference rewards obtained by all interference nodes in a interference test.

[0177] Interference success rate: The interference success rate is the proportion of all frequency hopping links that can be successfully interfered with simultaneously in 100 interference tests.

[0178] All methods are evaluated under dense rewards and repeated three times with different random seeds. The mean and standard deviation of the three experiments are calculated as the final results.

[0179] The results of this simulation experiment are analyzed as follows:

[0180] First, we compared the cumulative interference rewards of various methods during meta-training. Figure 2 As shown in . Figure 2 It can be seen that the proposed method shows the fastest convergence speed and the largest cumulative interference reward without the need for a professional interference reward function. Although the proposed method relies on the guidance of examples in the early stage of meta-training, it can quickly exceed the performance of the examples. The meta-TRPO-sparse method can hardly optimize the interference strategy when it is in a sparse interference reward, and the optimization speed of the interference strategy of the Meta-TRPO-dense method is still relatively slow when it is in a dense interference reward. The Meta-BC method has a faster convergence speed by imitating the examples, but the imitation of the examples is not enough to fully learn the useful information in the examples. It can be seen that the cumulative interference reward that the Meta-BC method can achieve when it converges is still lower than the method proposed in this application. In addition, the traditional continuous spectrum blocking interference method can hardly achieve an effective interference effect on the frequency hopping link because it ignores the cooperation of different interference nodes in deciding the interference scheme. It should be noted that all methods cannot reach the level of experts, because the methods based on MAML variants still need to be further improved through subsequent fine-tuning.

[0181] Next, the performance of the method proposed in this application in the fine-tuning stage is further demonstrated. The new interference tasks are divided into tasks seen in meta-training and tasks that have not been seen. Figure 3 It can be seen that the proposed method can adapt relatively faster to reach near-expert levels for seen tasks compared to unseen tasks. Figure 4 and Figure 5 The success rate of the proposed method against interference of each frequency hopping link is given in detail. Figure 4 As can be seen from the figure, the proposed algorithm only needs 50 simple gradient descent steps to achieve an average interference success rate of 91% in the tasks seen. Figure 5 As can be seen in Figure 2, the proposed algorithm only needs 50 simple gradient descent steps to achieve an average interference success rate of 82% in unseen tasks.

[0182] The following table gives the intuitive statistical values ​​of various evaluation indicators of the method proposed in this application in the fine-tuning stage. When facing unseen tasks, when the number of training trajectories reaches 800 times, a relatively considerable cumulative interference reward and interference success rate can be achieved. When facing seen tasks, when the number of training trajectories reaches 80 times, a considerable cumulative interference reward and interference success rate can be achieved.

[0183] Table 2 Intuitive statistical values ​​of each evaluation index in the fine-tuning stage

[0184] Number of trajectories to train Cumulative Interference Reward Interference success rate Unseen missions 800 213.28±58.9 0.82±0.24 Missions seen 80 232.89±26.61 0.91±0.11 expert - 240.00 1.00 Example - 79.00 0.25

[0185] Finally, an ablation experiment related to the proposed algorithm was conducted, and the results are shown in Figure 6 As shown in . Figure 6 It can be seen that adding behavioral cloning loss to the loss function can greatly improve the cumulative interference reward obtained by the proposed method, and using adaptive updates instead of fixed learning rates can also improve the rewards. In addition, combining hot start with behavioral cloning can greatly improve the convergence speed of the proposed method, thanks to the guidance of useful information in the example.

[0186] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any modification, use or adaptation of the present disclosure, which follows the general principles of the present disclosure and includes common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the appended claims.

Claims

1. A non-expert data-assisted meta-reinforcement learning frequency hopping interference resource allocation method, characterized in that: include: Construct the frequency hopping jamming resource allocation problem based on the communication confrontation scenario; Modeling the frequency hopping interference resource allocation problem as a Markov decision process; Randomly initialize the meta-strategy to get the initial strategy network; Performing meta-training on the initial policy network based on the MAML algorithm, wherein the meta-training includes a specific task adaptation phase and a meta-optimization phase; Wherein, in the specific task adaptation stage, based on non-expert data, the initial strategy network is optimized and trained using a reinforcement learning loss function and a behavior cloning loss function to obtain a specific task strategy, and the reinforcement learning loss of each interference task is calculated based on the specific task strategy; In the meta-optimization stage, the reinforcement learning losses of all interference tasks are minimized, and an optimized allocation strategy is obtained; Fine-tune the allocation strategy optimized by meta-training based on the new distractor task.

2. The non-expert data-assisted meta-reinforcement learning frequency hopping interference resource allocation method according to claim 1, characterized in that: In the specific task adaptation stage, based on non-expert data, the initial strategy network is optimized and trained using a reinforcement learning loss function and a behavior cloning loss function to obtain a specific task strategy, and the reinforcement learning loss of each interference task is calculated based on the specific task strategy, including the following steps: A plurality of interference tasks are randomly sampled, and the following steps are performed cyclically for each of the interference tasks: Using the initial strategy network to interact with each of the interference tasks to obtain first training data; Calculating parameters of a specific task strategy using the non-expert data and the first training data; According to the specific task strategy, the TRPO algorithm is used to calculate the reinforcement learning loss of each interference task.

3. The non-expert data-assisted meta-reinforcement learning frequency hopping interference resource allocation method according to claim 2, characterized in that: The calculating the parameters of the specific task strategy by using the non-expert data and the first training data comprises: Determine the reinforcement learning loss function, the formula is as follows: in, V(s t ) represents the value function, represents the reinforcement learning loss function, represents the first training data, θ o represents the parameters of the initial policy network, represents the initial policy network, H represents the length of a trajectory, γ represents the discount factor, and λ represents the GAE parameter; The behavior cloning loss function is determined as follows: in, represents the behavior cloning loss function, represents non-expert data, θ o represents the parameters of the initial policy network, represents the initial policy network; The parameters of the specific task strategy are obtained by using the reinforcement learning loss function and the behavior cloning loss function to derive the parameters of the initial strategy network. The formula is as follows: Among them, θ i represents the parameters of a specific task strategy, θ o represents the parameters of the initial policy network, ⊙ represents the Hadamard product, α represents the adaptive learning rate vector, represents the gradient of the initial policy network, w rl represents the first loss weight, w bc represents the second loss weight.

4. The non-expert data-assisted meta-reinforcement learning frequency hopping interference resource allocation method according to claim 3, characterized in that: According to the specific task strategy, the reinforcement learning loss of each interference task is calculated using the TRPO algorithm, including the following steps: Using the specific task strategy to interact with the plurality of interference tasks to obtain verification data; Based on the verification data, the reinforcement learning loss is calculated using the TRPO algorithm.

5. The non-expert data-assisted meta-reinforcement learning frequency hopping interference resource allocation method according to claim 4, characterized in that: In the meta-optimization stage, the reinforcement learning loss of all interference tasks is minimized, and the optimized allocation strategy is obtained, including the following steps: The reinforcement learning loss of all interference tasks is minimized by formula (4): Among them, θ i represents the parameters of a specific task strategy, θ o represents the parameters of the initial policy network, ⊙ represents the Hadamard product, α represents the adaptive learning rate vector, represents the gradient of the initial policy network, Represents verification data; The TRPO algorithm is used to solve the formula (4) to obtain the optimized allocation strategy.

6. The non-expert data-assisted meta-reinforcement learning frequency hopping interference resource allocation method according to claim 1, characterized in that: The fine-tuning stage includes the following steps: Using the optimized allocation strategy to interact with the new interference task to obtain second training data and example data corresponding to the new interference task; According to the second training data and the example data, a gradient descent step is performed on the allocation strategy after meta-training optimization to obtain fine-tuned and updated policy network parameters.

7. The non-expert data-assisted meta-reinforcement learning frequency hopping interference resource allocation method according to claim 1, characterized in that: In the meta-training, it is assumed that the actions sampled from the non-expert data have an advantage value relative to the actions sampled from the initial policy network, and the advantage value can be described as: Where ω represents the value of the advantage, represents the initial policy network, represents the actions sampled from non-expert data, a t represents the actions sampled from the initial policy network, Indicates the advantage that the action sampled from non-expert data can obtain over other actions in the state, Indicates the advantage that the action sampled from the initial policy network can obtain over other actions in the state, represents the interference task distribution, represents the tasks sampled from the interfering task distribution.

8. The non-expert data-assisted meta-reinforcement learning frequency hopping interference resource allocation method according to claim 1, characterized in that: The method of modeling the frequency hopping interference resource allocation problem as a Markov decision process includes: defining an action space, a state space, and a reward function for the Markov decision process; A discrete Markov decision process is constructed according to the action space, the state space and the reward function.

9. The non-expert data-assisted meta-reinforcement learning frequency hopping interference resource allocation method according to claim 8, characterized in that: The reward function can be described as: Among them, r(s t ,a t ) represents the reward function, e n (t) represents the transient interference effect of the currently constructed interference scheme on the nth frequency hopping frequency set, a t represents the action of the interferer, s t represents the state of the environment, κ represents the interference coverage coefficient, represents the frequency hopping frequency set, Indicates the number of frequency hopping frequency sets.

10. The non-expert data-assisted meta-reinforcement learning frequency hopping interference resource allocation method according to claim 1, characterized in that: In the step of constructing a frequency hopping interference resource allocation problem according to the communication confrontation scenario, the frequency hopping interference resource allocation problem can be expressed as follows after being formulated: Where sgn(·) is a sign function. When the independent variable x>0, sgn(x)=1; represents the interference plus noise signal ratio of the frequency hopping channel at time t, is the total number of channels in the nth frequency hopping frequency set, Indicates that all interfering nodes are connected with f u,n (t) is the overall JNSR generated by the hopping channel of the center frequency, κ is the interference coverage factor, is the maximum interference power, is the maximum signal power, is the set of central interference frequencies, B J is the set of interference bandwidths.

Images