O2AJFH-based dual-hopping frequency hopping pattern intelligent decision-making method

Abstract

The invention discloses an O2AJFH-based double-hopping frequency hopping pattern intelligent decision-making method, which designs a double-hopping frequency hopping pattern and an O2AJFH algorithm based on an offline-online framework, firstly realizes primary frequency change by dynamically adjusting a frequency range, and then realizes secondary frequency change by optimizing continuous values of frequency hopping frequency, frequency hopping rate and instantaneous bandwidth, so that the frequency hopping pattern can be determined according to the frequency hopping pattern and the O2AJFH algorithm, and the frequency hopping pattern and the O2AJFH algorithm can be determined according to the frequency hopping pattern and the O2AJFH algorithm. The generated double-hopping frequency hopping pattern is used for intelligently deciding the double-hopping frequency hopping pattern, the problems that the frequency point hopping randomness is not strong enough or sometimes interference cannot be effectively avoided, the flexibility and randomness of the frequency hopping pattern are limited to a certain degree and the like are solved, and the performance of the frequency hopping pattern is better than that of a three-change frequency hopping pattern.

Description

Technical Field

[0001] This invention relates to the field of intelligent anti-interference decision-making technology in frequency hopping communication, specifically to an intelligent decision-making method for frequency hopping patterns based on O2AJFH dual-transition. This intelligent decision-making method for frequency hopping patterns is an intelligent anti-interference decision-making method based on an offline-online framework algorithm to adjust the parameters of the "dual-transition" frequency hopping pattern. Background Technology

[0002] With the increasing complexity of electromagnetic interference environments, traditional frequency-hopping communication technology faces severe challenges. Traditional frequency-hopping technology typically pre-sets fixed frequency-hopping parameters between the transmitter and receiver. Its anti-interference strategy is relatively fixed and lacks flexibility, essentially relying on blind anti-interference measures and failing to guarantee communication reliability in complex and ever-changing interference environments. In contrast, the "dual-variable" frequency-hopping pattern more effectively avoids interference by dynamically adjusting the hopping rate and frequency spacing. However, it still suffers from insufficient randomness in frequency hopping, making it difficult to flexibly avoid strong interference environments.

[0003] To address the aforementioned issues, the "three-variable" frequency hopping technology enhances the anti-interference and anti-interception capabilities of the frequency hopping system by dynamically adjusting the hopping frequency, rate, and instantaneous bandwidth. The hopping frequency is a continuous value, while the hopping rate and instantaneous bandwidth are discrete values. However, selecting hopping frequencies only within a fixed or small frequency range may result in insufficient randomness in frequency hopping or sometimes an inability to effectively avoid interference. Furthermore, because the hopping rate and instantaneous bandwidth are preset as discrete values, the flexibility and randomness of the hopping pattern are somewhat limited. Figure 1 The spectral distributions of conventional frequency hopping signals, "three-change" frequency hopping signals, and "double-change" frequency hopping signals are presented.

[0004] Therefore, the "double frequency hopping" pattern, which achieves continuous frequency change and continuous random change of frequency hopping rate and instantaneous bandwidth through a second frequency hopping, is of great significance for improving the anti-interference performance of frequency hopping communication systems.

[0005] Deep reinforcement learning, by combining the powerful representational capabilities of deep neural networks, can approximate the value functions of states and actions and perform effective mappings in high-dimensional state-action spaces, thereby solving complex decision-making problems. Currently, deep reinforcement learning algorithms used for intelligent decision-making on frequency-hopping patterns are all online algorithms. Their policy performance is poor in the early stages of online training, and the frequency-hopping signals are easily tracked by interfering parties. Therefore, further research is needed on multi-parameter intelligent decision-making for frequency-hopping signals using deep reinforcement learning based on an offline-online reinforcement learning framework. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to address the limitations of existing intelligent anti-interference decision-making systems and improve the anti-interference and anti-interception capabilities of frequency hopping communication systems.

[0007] To address the aforementioned technical problems, the present invention provides a frequency hopping pattern intelligent decision-making method based on O2AJFH dual-transition, comprising the following steps: A module 1 for primary frequency changes is established using pseudo-random sequences and the improved ID3QN algorithm of D3QN. A module 2 for secondary frequency changes is established by optimizing the continuous values ​​of frequency hopping frequency, frequency hopping rate and instantaneous bandwidth using the improved IDSAC-T algorithm. Based on the perceived spectrum state, estimate the main parameters of the interference and predict the spectrum state within the time period and frequency band; Module 1 generates the initial frequency range of the h-th hop of the frequency hopping signal based on the set pseudo-random sequence. The initial frequency range is within the frequency band. The Agent obtains the initial state from the environment based on the initial frequency range and uses the ID3QN algorithm to make intelligent decisions on actions. The decision frequency range is obtained by shifting the frequency range of the h-th hop signal upward or downward, thus realizing a one-time frequency change. Module 2 obtains the second state from the environment based on the decision frequency range, and makes a decision on the action according to the IDSAC-T algorithm. After the action interacts with the environment, the next state and reward are generated. The states, actions and rewards corresponding to the two modules are stored in the corresponding experience pools respectively. Offline training phase: From the experience pool Uniform sampling Based on experience, calculate the loss function, update the network parameters of module one, and then update the target network parameters every preset number of steps. From the experience pool Uniform sampling Based on this experience, the gradient of the critic network is calculated, and the parameters of the estimated critic network and the target network in module two are updated. Then, every [percentage missing]... Step, using The empirical update module 2 estimates the network parameters and target network parameters of the Actor, and then updates the regularization coefficients every [period]. The target entropy is updated during each training iteration; Online training phase: After training the initial policy, the network parameters obtained during the offline training phase are used to initialize the number of training rounds and the total number of training rounds of the algorithm. Initialize the number of hops, input the first state into module one, select the first action based on the action of module one, and execute the first action to obtain the second state; The second state is input into the Actor estimation network of module two to obtain the second action. The second action is executed to obtain the next state of the first state, and the first reward and the second reward are calculated. Store the first state, first action, first reward, and the next state of the first state into the experience pool. From the experience pool Uniform sampling Based on experience, calculate the loss function, update the network parameters of module one, and then update the target network parameters every preset number of steps. The next state of the first state is input into module one. The action selection strategy of module one is used to obtain the next action of the first action. The next action of the first action is executed to obtain the previous state of the second state. The second state, the second action, the second reward and the previous state of the second state are stored in the experience pool. From the experience pool Uniform sampling Based on this experience, the gradient of the critic network is calculated, and the parameters of the estimated critic network and the target network in module two are updated. Then, every [percentage missing]... Step, using The empirical update module 2 estimates the network parameters and target network parameters of the Actor, and then updates the regularization coefficients every [period]. The target entropy is updated during each training iteration; If the previous state in the second state is not the termination state, then execution begins from the step in which the first and second rewards were calculated during the online training phase. If the number of training rounds of the algorithm is not [missing information], then [missing information]. Then, execution begins after initializing the hop count during the online training phase, continuing until the algorithm reaches the required training rounds. The algorithm ends, yielding the following results: fixed interference environment and corresponding initial frequency hopping pattern, fixed interference environment and corresponding steady-state frequency hopping pattern, dynamic interference environment and corresponding initial frequency hopping pattern, dynamic interference environment and corresponding steady-state frequency hopping pattern, intelligent dynamic interference environment and corresponding initial frequency hopping pattern, and intelligent dynamic interference environment and corresponding steady-state frequency hopping pattern.

[0008] Preferably, the loss function is calculated using the following formula:

[0009] in, It is a loss function that makes a conservative estimate of the unknown state-action pair; It is a loss function that makes an optimistic estimate of the state-action pair; This is a tradeoff factor used to control the weighting relationship between the two estimates in the overall loss function; The true behavioral strategy for an unknown state-action pair can only be approximated by existing data in the experience pool. The objective function is... As a reward discount factor; Indicates the first Network parameters of a Q-network; Indicates the first The network parameters of the Q target network.

[0010] Preferably, the formulas for updating the estimated network parameters and the target network parameters by the Actor are as follows:

[0011] in, Estimate network parameters for the Actor. For the target network parameters, Estimate the network's learning rate for the Actor.

[0012] Preferably, the regularization coefficient update formula is as follows:

[0013] in, The regularization coefficient is . The learning rate is the regularization coefficient.

[0014] Preferably, the target entropy update formula is as follows:

[0015] in, , Let be the learning rate of the target entropy, and let the initial target entropy be . , Let be the target entropy.

[0016] Preferably, the reward is calculated based on an instant reward function:

[0017]

[0018] in, For the first Jumping SJNR, For the first The dwell time of the frequency-hopping signal. , For the first The instantaneous bandwidth of the jump To control The constant of the magnitude, To control The constant of the magnitude, To control The constant of the magnitude, Instantaneous bandwidth Interference power in the environment For the set threshold, For the first The jump is used as the first reward in guiding Module 1 to select the optimal action. For the first The jump is used as a second reward to guide Module 2 in selecting the optimal action.

[0019] This invention proposes an intelligent decision-making method for frequency hopping patterns based on O2AJFH with dual-hopping transitions. It designs a "dual-hopping" frequency hopping pattern and an O2AJFH algorithm based on an offline-online framework. First, it dynamically adjusts the frequency range to achieve a "first" frequency change. Then, it optimizes the continuous values ​​of the hopping frequency, hopping rate, and instantaneous bandwidth to achieve a "second" frequency change, generating a "dual-hopping" frequency hopping pattern. This method intelligently decides on the "dual-hopping" frequency hopping pattern, solving problems such as insufficient randomness of frequency hopping transitions or the inability to effectively avoid interference, and limitations on the flexibility and randomness of the frequency hopping pattern. It outperforms the "three-change" frequency hopping pattern. Attached Figure Description

[0020] Figure 1 Frequency domain model diagrams of interference spectrum, conventional frequency hopping signal spectrum, "three-change" frequency hopping signal spectrum, and "double-change" frequency hopping signal spectrum provided in embodiments of the present invention; Figure 2 A schematic diagram of the corresponding structure of a frequency hopping pattern intelligent decision-making method based on dual hopping transitions of O2AJFH provided in an embodiment of the present invention; Figure 3 Schematic diagrams of a fixed interference environment and its corresponding initial frequency hopping pattern and a fixed interference environment and its corresponding steady-state frequency hopping pattern provided in embodiments of the present invention; Figure 4 A schematic diagram of the dynamic interference environment and the corresponding initial frequency hopping pattern and the dynamic interference environment and the corresponding steady-state frequency hopping pattern provided in the embodiments of the present invention; Figure 5 Schematic diagrams of the intelligent dynamic interference environment and corresponding initial frequency hopping pattern and the intelligent dynamic interference environment and corresponding steady-state frequency hopping pattern provided in the embodiments of the present invention; Figure 6 The diagrams show the fixed interference environment and corresponding performance curves, the dynamic interference environment and corresponding performance curves, and the intelligent dynamic interference environment and corresponding performance curves provided in the embodiments of the present invention. Detailed Implementation

[0021] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. Furthermore, it should be understood that after reading the teachings of this invention, those skilled in the art can make various alterations or modifications to the invention, and these equivalent forms also fall within the scope defined by the appended claims.

[0022] This invention provides a frequency hopping pattern intelligent decision-making method based on O2AJFH dual-transition, comprising two modules: Module 1 utilizes a pseudo-random sequence and the improved algorithm ID3QN[1] to achieve a "one-time" frequency change, ensuring frequency randomness and effectively avoiding complex interference.

[0023] D3QN[1] References:] M. Gök. Dynamic Path Planning Via Dueling DoubleDeep Q-Network (D3QN) with Prioritized Experience Replay[J]. Applied SoftComputing, 2024, 158: 111503.

[0024] Module 2 uses the improved IDSAC-T[2] algorithm to optimize the continuous values ​​of frequency hopping, frequency hopping rate and instantaneous bandwidth, realize the "secondary" frequency change, and increase the system flexibility.

[0025] IDSAC-T[2] References: J. Duan, et al. Distributional Soft Actor-Criticwith Three Refinements[J]. IEEE Transactions on Pattern Analysis and MachineIntelligence, 2025, 1-12.

[0026] The O2AJFH algorithm incorporates offline pre-training and online fine-tuning, improving its decision-making performance in the initial stage. Furthermore, to enable the O2AJFH algorithm to converge to a better strategy, the IDSAC-T algorithm introduces a method for dynamically adjusting the target entropy. The "double-hop" frequency-hopping pattern generated by the method of this invention can generate an effective frequency-hopping strategy in the initial stage and effectively avoid fixed, intelligent, and dynamic interference, demonstrating better adaptability and anti-interference capabilities.

[0027] The specific steps are as follows: Step 1: Based on the sensed spectrum state, estimate the main parameters of the interference and predict the time period. and frequency band The spectrum state within; Step 2: Set the following parameters: signal power Gaussian white noise power Pseudo-random sequences Frequency hopping rate Instantaneous channel bandwidth Frequency range offset , frequency range offset direction reward function threshold Initialize the experience pool using offline datasets and The number of parallel Q-networks is The number of samples was and Tuning parameters ; Learning rate of Module 1 The learning rates of the Critic network and Actor network in Module 2 are respectively and "Soft update" coefficient Regularization coefficient learning rate Reward Discount Factor ; Initialize regularization coefficients Target entropy ; Initialize the Q network parameters of module one and ; In Module 2, the Critic network parameters are initialized. and Actor network parameters and .

[0028] Specifically as follows: The O2AJFH algorithm consists of two modules within an offline-online framework. In the offline training phase, the O2AJFH algorithm pre-trains the network parameters of both modules using only a fixed dataset from the experience pools, thereby optimizing the network parameters to obtain optimal results. In the online phase, online training and decision-making are performed alternately. First, the fixed data from the two experience pools is used to initialize the corresponding online experience pools, and the optimized network parameters are used as the initial network parameters for the online phase.

[0029] Module 1 generates the initial frequency range of the h-th hop of the frequency hopping signal based on the pseudo-random sequence L. , .

[0030] Agent based on initial frequency range Get state from the environment .

[0031] in, This represents the frequency range at the current time corresponding to the h-th jump. The maximum value of the internal interference power after normalization. And it utilizes the ID3QN algorithm for intelligent decision-making. = ,in and These represent the offset and offset direction of the frequency range of the h-th hop signal, respectively. This represents a discrete set of offsets. By shifting the frequency range of the h-th hop upwards or downwards, complex interference can be initially avoided, improving the system's anti-interference and anti-interception capabilities.

[0032] when When the frequency range is shifted towards lower frequencies, the upper and lower bounds of the frequency range are:

[0033] in, The bandwidth of the frequency range of each hop signal. Similarly, when When the frequency range is shifted towards higher frequencies, the upper and lower bounds of the frequency range are:

[0034] in, Module 1 implements the first frequency jump.

[0035] Module 2 determines the frequency range based on the decisions made in Module 1. Get state from the environment ,in , This represents the frequency range at the current time corresponding to the h-th jump. The maximum value of the internal interference power after normalization. The IDSAC-T algorithm is then applied to intelligently decide the action for the h-th jump. = ,in For frequency hopping, For frequency hopping rate, This refers to the instantaneous bandwidth. After interacting with the environment, the next state is generated. and rewards , Next, the states, actions, and rewards corresponding to the two modules are stored in their respective experience pools to continue training their network parameters.

[0036] Offline training phase: Step 3: From the experience pool Uniform sampling Based on experience, calculate the loss function and use it to update the network parameters of Module 1. Subsequently, the target network parameters are updated every 50 steps. The details are as follows: In Module 1, the Agent draws from the experience pool. Uniform sampling Based on experience, the loss function can be calculated using the following formula:

[0037] in, It is a loss function that makes a conservative estimate of the unknown state-action pair; It is a loss function that makes an optimistic estimate of the state-action pair; This is a tradeoff factor used to control the weighting relationship between the two estimates in the overall loss function; The true behavioral strategy for an unknown state-action pair can only be approximated by existing data in the experience pool. The objective function is... As a reward discount factor; Indicates the first Network parameters of a Q-network; Indicates the first The network parameters of the target Q network are then updated using the following formula. :

[0038] in, For learning rate, Let be the number of Q-networks. Then, every 50 steps, update the target network parameters using the following formula: ;

[0039] Step 4: From the experience pool Uniform sampling This experience will help update the Critic estimation network parameters in Module 2. and target network parameters Subsequently, every Step, using this Actor estimation network parameters in Module 2 of the empirical update. and target network parameters Then update the regularization coefficients. Every The target entropy is updated during the next training iteration. The details are as follows: In Module Two, from the experience pool Uniform sampling One empirical method is to calculate the gradient update of the critic network using the following formula:

[0040] The regularization coefficients of the control policy entropy are as follows:

[0041]

[0042]

[0043]

[0044]

[0045]

[0046] .

[0047] The Critic estimated network parameters are then updated based on the following formula. :

[0048] in, This involves estimating the learning rate of the Critic network. The parameters of the Critic target network in Module 2 are updated according to the following formula. .

[0049]

[0050] in, This is the "soft update" coefficient, and Subsequently, every Step, using this Based on group experience, the Actor estimates network parameters in Module 2 are updated according to the following formula. and target network parameters .

[0051]

[0052] in, Estimate the network's learning rate for the Actor. Simultaneously, update the regularization coefficients using the following formula. :

[0053] in, The learning rate is the regularization coefficient. Subsequently, every [percentage missing]... During the training iteration, the target entropy is updated according to the following formula: :

[0054] in, , Let be the learning rate of the target entropy, and let the initial target entropy be . , The threshold value is the target entropy.

[0055] Continue performing steps 3 and 4 until a better initial strategy is obtained.

[0056] Online training phase: The network parameters obtained during the offline training phase are used to initialize the network parameters during the online training phase. Step 5: Number of training rounds for the algorithm Total number of training rounds for the algorithm ; Step 6: Initialize hop count ; Step 7: Set the status The input is fed into Module 1, where the action selection strategy of Module 1 is used to obtain the action. Execute actions Get the state The details are as follows: In the O2AJFH algorithm, the state Input into Module 1, and select the action according to the following formula. Execute actions Get the state The action selection strategy for Module 1 is obtained as follows:

[0057] in, The number of parallel Q-networks, For the first Parameters of a Q-network The tuning parameters determine the degree of optimistic exploration.

[0058] Step 8: Set the status The input to the Actor estimation network in Module 2 yields the action. ,implement Get the next state Calculate rewards and rewards The details are as follows: To maximize the signal-to-interference-plus-noise ratio (SINR), the instantaneous reward function is designed as follows:

[0059]

[0060] in, For the first Jumping SJNR, For the first The dwell time of the frequency-hopping signal. , For the first The instantaneous bandwidth of the jump To control The constant of the magnitude, To control The constant of the magnitude, To control The constant of the magnitude, Instantaneous bandwidth Interference power in the environment For the set threshold, For the first The jump is used as a reward to guide Module 1 in selecting the optimal action. For the first The jump is used as a reward to guide Module 2 in selecting the optimal action.

[0061] Step 9, Store in experience pool ,like The current experience is full; old experiences will be replaced in the order they were generated. Step 10: From the experience pool Uniform sampling Group experience is used to calculate the loss function, which is then used to update the network parameters of Module 1. Subsequently, the target network parameters are updated every 50 steps. The details are as follows: The specific method for step 10 in the online phase is the same as the specific method for step 3 in the offline phase. The agent draws from the experience pool. Uniform sampling Based on group experience, the loss function is calculated according to the formula, and then the network parameters of Module 1 are updated using the formula. Subsequently, the target network parameters are updated by formula every 50 steps. .

[0062] Step 11, Change the status The input is fed into Module 1, where the action selection strategy of Module 1 is used to obtain the action. Execute actions get ;Will Store in experience pool In the middle, if The current experience is full; old experiences will be replaced in the order they were generated. Step 12, from the experience pool Uniform sampling Group experience is used to update the Critic estimation network parameters in Module 2. and target network parameters Subsequently, every Step, using the above Group experience, update the Actor estimation network parameters in Module 2. and target network parameters Update the regularization coefficients. Subsequently every The target entropy is updated during the next training iteration. The details are as follows: The specific method for step 12 in the online phase is the same as the specific method for step 4 in the offline phase. The Agent draws from the experience pool. Uniform sampling Group experience, using formulas to update Critic estimated network parameters Then, the Critic target network parameters in module two are updated using the formula. Subsequently, every Step, using the above Based on group experience, the Actor estimated network parameters and target network parameters in Module 2 are updated according to the formula. Update the regularization coefficients using the formula. Subsequently, every The target entropy is updated according to the formula during the next training iteration. .

[0063] Step 13, if Not in a terminated state. , , Return to step 8; otherwise, if the algorithm reaches the required number of training rounds... If the algorithm terminates, then the algorithm completes the training rounds; otherwise, the algorithm completes the training rounds. Return to step 6.

[0064] This invention utilizes the O2AJFH algorithm for intelligent decision-making on "double-hop" frequency hopping patterns, resulting in the following frequency hopping pattern: Figure 3 (a) and 3(b), Figure 4 (a) and 4(b), Figure 5 As shown in (a) and 5(b), the small red rectangles in the figures represent the frequency band and dwell time of the frequency hopping signal in a certain time slot.

[0065] Figure 3 The interference in (a) and (b) is fixed interference. Specifically, Figure 3(a) in the diagram represents a fixed interference environment and the corresponding initial frequency hopping pattern. Figure 3 (b) in the diagram represents a fixed interference environment and the corresponding steady-state frequency hopping pattern.

[0066] Figure 4 (a) and (b) Figure 5 In (a) and (b), the wideband and narrowband interference and frequency sweeping interference are randomly and dynamically selected in different time slots. Specifically, Figure 4 (a) in the diagram represents the dynamic interference environment and the corresponding initial frequency hopping pattern. Figure 4 (b) in the diagram represents the dynamic interference environment and the corresponding steady-state frequency hopping pattern. Figure 5 (a) in the diagram represents the intelligent dynamic interference environment and the corresponding initial frequency hopping pattern. Figure 5 (b) in the diagram represents the intelligent dynamic interference environment and the corresponding steady-state frequency hopping pattern.

[0067] Figure 5 In (a) and 5(b), the small blue rectangles represent intelligent interference. As can be seen from the figures, in the initial stage, the frequency-hopping pattern effectively avoids most interference areas. In areas with significant interference, the frequency-hopping pattern is relatively small; while in areas with less interference, the pattern is relatively large. With continuous optimization of the algorithm's performance in the online stage, the frequency-hopping pattern in the steady-state stage essentially avoids interference. The O2AJFH algorithm demonstrates good adaptability and optimization capabilities in fixed, dynamic, and intelligent interference environments.

[0068] exist Figure 3 (a) and 3(b), Figure 4 (a) and 4(b), Figure 5 In the three different interference environments shown in (a) and 5(b), the O2AJFH algorithm and the online algorithm of O2AJFH were applied to intelligently decide the "double hop" frequency hopping pattern, along with three offline-online reinforcement learning algorithms: O2AJFH algorithm, ID3QN-AWAC (ID3QN and AWAC[3] combined algorithm), ID3QN-Cal-QL (ID3QN and Cal-QL[4] combined algorithm), and ID3QN-EDAC (ID3QN and EDAC[5] combined algorithm). The average signal-to-interference-plus-noise ratio (SIR) of each algorithm under the same interference environment was obtained as shown in the curves of the number of training rounds. Figure 6 As shown in (a)-(c), Figure 6 (a) in the figure represents a fixed interference environment and the corresponding performance curve. Figure 6 (b) in the figure represents the dynamic interference environment and the corresponding performance curve. Figure 6 (c) in the figure represents the intelligent dynamic interference environment and the corresponding performance curve.

[0069] ID3QN and AWAC[3] combined algorithm Reference: A. Nair, A. Gupta, M. Dalal, et al. AWAC: Accelerating Online Reinforcement Learning with Offline Datasets[J].Arxiv Preprint Arxiv:2006.09359, 2020.

[0070] ID3QN and Cal-QL[4] combined algorithm Reference: M. Nakamoto, S. Zhai, A. Singh, et al. Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuning[J]. Advances in Neural Information Processing Systems, 2024, 36:62244-62269.

[0071] EDAC[5] Combinatorial Algorithm Reference: G. An, S. Moon, HJ Kim, et al. Uncertainty-Based Offline Reinforcement Learning with Diversified Q-ensemble[J]. Advances in Neural Information Processing Systems, 2021, 34: 7436-7447.

[0072] As can be observed from the figure, compared with the O2AJFH algorithm of this invention without an offline training phase, the O2AJFH algorithm using the offline-online framework exhibits superior initial policy performance in the initial stage of online training, and its steady-state average signal-to-interference-plus-noise ratio (SNR) is significantly higher. Compared with the offline-online algorithm, the O2AJFH algorithm shows faster convergence speed and more stable performance, with a higher steady-state average SNR. O2AJFH can generate an effective frequency hopping policy in the initial stage. Furthermore, compared with the benchmark algorithm, O2AJFH has a faster convergence speed and higher steady-state performance, can adapt to different interference environments, and has stronger anti-interference capabilities.

[0073] The beneficial effects of the embodiments of the present invention are as follows: 1. The "double-hop" frequency hopping pattern proposed in this invention first achieves a "first" frequency change by dynamically adjusting the hopping frequency, ensuring frequency randomness and effectively avoiding complex interference. Then, it achieves a "second" change by optimizing and adjusting the hopping frequency, hopping rate, and instantaneous bandwidth, increasing system flexibility. The "double-hop" frequency hopping pattern has better anti-interference performance than the "triple-hop" frequency hopping pattern and can effectively avoid fixed, intelligent, and dynamic interference.

[0074] 2. This invention proposes an O2AJFH algorithm for intelligent decision-making of "double-hopping" frequency hopping patterns. The O2AJFH algorithm consists of two modules: Module 1 uses a pseudo-random sequence and the ID3QN algorithm to achieve a "single" frequency change; Module 2 uses the IDSAC-T algorithm to optimize the continuous values ​​of the frequency hopping frequency, frequency hopping rate, and instantaneous bandwidth to achieve a "second" frequency change.

[0075] 3. The O2AJFH algorithm introduces offline pre-training and online fine-tuning, which improves the algorithm's decision-making performance in the initial stage, enabling the O2AJFH algorithm to generate an effective frequency hopping strategy in the initial stage.

[0076] 4. In order to balance the exploration and utilization relationship of O2AJFH, a method of dynamically adjusting the target entropy is introduced, which reduces the exploration ratio of the agent in the later stage of training, makes full use of actions with known good results, and avoids the strategy failing to converge to the optimum due to over-exploration.

[0077] The key technical points of this invention are as follows: (1) A smart decision-making method for frequency hopping patterns based on O2AJFH with "double hopping". This invention includes two modules: Module 1 uses pseudo-random sequences and the ID3QN algorithm to achieve a "single" frequency change, ensuring frequency randomness and effectively avoiding complex interference; Module 2 uses an improved IDSAC-T algorithm to optimize the continuous values ​​of frequency hopping frequency, frequency hopping rate, and instantaneous bandwidth, achieving a "second" frequency change and increasing system flexibility. The O2AJFH algorithm introduces offline pre-training and online fine-tuning, improving the algorithm's decision-making performance in the initial stage. In addition, to enable the O2AJFH algorithm to converge to a better strategy, the IDSAC-T algorithm introduces a method for dynamically adjusting the target entropy.

[0078] (2) A “double-hop” frequency hopping pattern. This technique first utilizes a randomly optimized frequency hopping band to achieve a “first” frequency change. Subsequently, a “second” change is achieved by optimizing and adjusting the frequency hopping frequency, frequency hopping rate, and instantaneous bandwidth. The frequency hopping frequency, frequency hopping rate, and instantaneous bandwidth are continuously selected.

[0079] The "double-hop" frequency hopping pattern in this invention refers to a signal whose hopping frequency, hopping rate, and instantaneous bandwidth are continuously and randomly changing, with the hopping frequency changing twice. This enhances the randomness and flexibility of the frequency hopping pattern, further improving its anti-interference and anti-interception capabilities. Simultaneously, an improved Dueling Double DeepQ Network (ID3QN) and pseudo-random sequences are used to achieve a "single" frequency change, ensuring frequency randomness and effectively avoiding complex interference. An improved Distributional SoftActor-Critic with Three Refinements (IDSAC-T) algorithm is used to optimize the continuous values ​​of the hopping frequency, hopping rate, and instantaneous bandwidth, achieving a "double" frequency change. Furthermore, this invention employs deep reinforcement learning based on an offline-online reinforcement learning framework for intelligent decision-making regarding the "double-hop" frequency hopping pattern, improving its policy performance in the early stages of online training.

Claims

1. A frequency-hopping pattern intelligent decision-making method based on O2AJFH dual-transition, characterized in that, Includes the following steps: A module 1 for primary frequency change is established using pseudo-random sequences and the improved ID3QN algorithm. A module 2 for secondary frequency change is established using the improved IDSAC-T algorithm to optimize the continuous values ​​of frequency hopping frequency, hopping rate, and instantaneous bandwidth. Based on the sensed spectral state, the main parameters of interference are estimated, and the spectral state within the time period and frequency band is predicted. Module 1 generates the initial frequency range of the h-th hop of the frequency hopping signal based on the set pseudo-random sequence. The initial frequency range is within the frequency band. The agent obtains the initial state from the environment based on the initial frequency range and uses the ID3QN algorithm to intelligently decide on actions. The decision frequency range is obtained by shifting the frequency range of the h-th hop signal upwards or downwards by the first intelligent decision action, thus realizing the primary frequency change. Module 2 obtains the second state from the environment based on the decision frequency range and decides on actions according to the IDSAC-T algorithm. After the action interacts with the environment, the next state and reward are generated. The states, actions, and rewards corresponding to the two modules are stored in the corresponding experience pools. Offline training phase: uniform sampling from the experience pools. For each experience point, calculate the loss function, update the network parameters of module one, and then update the target network parameters every preset number of steps; uniformly sample from the experience pool. Based on this experience, the gradient of the critic network is calculated, and the parameters of the estimated critic network and the target network in module two are updated. Then, every [percentage missing]... Step, using The empirical update module 2 estimates the network parameters and target network parameters of the Actor, and then updates the regularization coefficients every [period]. The target entropy is updated during each training iteration; Online training phase: After training the initial policy, the network parameters obtained in the offline training phase are used to initialize the number of training rounds and the total number of training rounds of the algorithm; Initialize the hop count, input the first state into Module 1, select the first action based on the action selection strategy in Module 1, execute the first action to obtain the second state; input the second state into the Actor estimation network in Module 2 to obtain the second action, execute the second action to obtain the next state of the first state, calculate the first reward and the second reward; store the first state, the first action, the first reward, and the next state of the first state into the experience pool. From the experience pool Uniform sampling For each experience point, calculate the loss function, update the network parameters of Module 1, and then update the target network parameters every preset number of steps. Input the next state of the first state into Module 1, obtain the next action of the first action using the action selection strategy of Module 1, execute the next action of the first action to obtain the previous state of the second state, and store the second state, second action, second reward, and previous state of the second state into the experience pool. Uniform sampling Based on this experience, the gradient of the critic network is calculated, and the parameters of the estimated critic network and the target network in module two are updated. Then, every [percentage missing]... Step, using The empirical update module 2 estimates the network parameters and target network parameters of the Actor, and then updates the regularization coefficients every [period]. The target entropy is updated during each training iteration; If the previous state in the second state is not the termination state, then execution begins from the step in which the first and second rewards were calculated during the online training phase. If the number of training rounds of the algorithm is not [missing information], then [missing information]. Then, execution begins after initializing the hop count during the online training phase, continuing until the algorithm reaches the required training rounds. The algorithm ends, yielding the following results: fixed interference environment and corresponding initial frequency hopping pattern, fixed interference environment and corresponding steady-state frequency hopping pattern, dynamic interference environment and corresponding initial frequency hopping pattern, dynamic interference environment and corresponding steady-state frequency hopping pattern, intelligent dynamic interference environment and corresponding initial frequency hopping pattern, and intelligent dynamic interference environment and corresponding steady-state frequency hopping pattern.

2. The intelligent decision-making method for frequency hopping patterns based on dual-hopping transitions of O2AJFH as described in claim 1, characterized in that, The formula for calculating the loss function is as follows: in, It is a loss function that makes a conservative estimate of the unknown state-action pair; It is a loss function that makes an optimistic estimate of the state-action pair; This is a tradeoff factor used to control the weighting relationship between the two estimates in the overall loss function; The true behavioral strategy for an unknown state-action pair can only be approximated by existing data in the experience pool. The objective function is... As a reward discount factor; Indicates the first Network parameters of a Q-network; Indicates the first The network parameters of the Q target network.

3. The frequency hopping pattern intelligent decision-making method based on O2AJFH dual-hopping transition as described in claim 1, characterized in that, The formulas for updating the estimated network parameters and the target network parameters by the Actor are as follows: in, Estimate network parameters for the Actor. For the target network parameters, Estimate the network's learning rate for the Actor.

4. The intelligent decision-making method for frequency hopping patterns based on dual-hopping transitions of O2AJFH as described in claim 1, characterized in that, The regularization coefficient update formula is as follows: in, The regularization coefficient is . The learning rate is the regularization coefficient.

5. The frequency hopping pattern intelligent decision-making method based on O2AJFH dual-hopping transition as described in claim 1, characterized in that, The target entropy update formula is as follows: in, , Let be the learning rate of the target entropy, and let the initial target entropy be . , Let be the target entropy.

6. The frequency hopping pattern intelligent decision-making method based on O2AJFH dual-hopping transition as described in claim 1, characterized in that, The reward is calculated based on the instant reward function: in, For the first Jumping SJNR, For the first The dwell time of the frequency-hopping signal. , For the first The instantaneous bandwidth of the jump To control The constant of the magnitude, To control The constant of the magnitude, To control The constant of the magnitude, Instantaneous bandwidth Interference power in the environment For the set threshold, For the first The jump is used as the first reward in guiding Module 1 to select the optimal action. For the first The jump is used as a second reward to guide Module 2 in selecting the optimal action.

Images