Non-linear sweep frequency interference resisting communication method and system based on near-end strategy optimization

Abstract

The invention relates to a near-end strategy optimization-based non-linear sweep frequency interference resisting communication method and system. The method comprises the following steps: modeling an anti-nonlinear sweep frequency interference communication process as a Markov decision process, and constructing a deep neural network model comprising a strategy network and a value network; under the guidance of a current strategy network, a transmitter and a receiver synchronously execute a transmission channel selection decision of a previous time slot, switch to a corresponding channel to perform data communication, store interactive data of the receiver and a jammer as an experience sample into an experience pool, sample the experience sample from the experience pool, and send the sampled experience sample to the receiver; and optimizing parameters of the policy network and the value network by using a near-end policy optimization algorithm and an empirical sample, completing iterative updating of model weights by using an adaptive optimization algorithm, selecting a transmission channel of a next time slot by using the updated policy network, and feeding back the transmission channel to a transmitter. By adopting the method, the wireless communication system can autonomously learn the optimal anti-interference strategy, and the dynamic nonlinear frequency sweeping interference is efficiently inhibited.

Description

Technical Field

[0001] This application relates to the field of wireless communication technology, and in particular to a communication method and system for resisting nonlinear frequency sweep interference based on near-end strategy optimization. Background Technology

[0002] Wireless communication uses electromagnetic waves as the carrier of information transmission and has a naturally open transmission channel. This openness makes wireless communication systems highly vulnerable to various unintentional or malicious interferences. Frequency sweeping interference, a typical form of malicious interference, achieves a wide-bandwidth blocking effect with relatively low power cost by periodically and rapidly scanning various frequency points within the target frequency band. Existing frequency sweeping interference suppression methods typically adjust communication parameters such as carrier frequency to achieve interference suppression based on interference pattern identification and parameter extraction. While these methods are effective against linear frequency sweeping interference with fixed parameters, their performance significantly degrades under frequency sweeping interference with dynamically nonlinear changes in key parameters such as sweep direction and sweep rate. This is mainly because the time-varying characteristics of interference make it difficult for traditional methods to accurately model and extract interference characteristic parameters.

[0003] With the continuous improvement of the performance of digital communication devices, dynamic nonlinear frequency sweep interference has gradually become a practical application, posing a significant threat to the reliable transmission of wireless communication systems. In recent years, reinforcement learning, with its outstanding dynamic policy optimization capabilities, has demonstrated unique advantages in the field of communication interference suppression. It continuously optimizes transmission strategies through interactive learning with the interference environment, providing a new technical path to address the problem of dynamic nonlinear frequency sweep interference. Therefore, how to specifically apply reinforcement learning techniques to solve the problems of unpredictable time-frequency states, excessively large policy space dimensions, and insufficient model generalization ability under dynamic nonlinear frequency sweep interference, and achieve efficient suppression of nonlinear frequency sweep interference, thereby improving the reliability of wireless communication transmission in complex electromagnetic environments, has become a key technical problem that urgently needs to be solved. Summary of the Invention

[0004] Therefore, it is necessary to provide a nonlinear frequency sweeping interference-resistant communication method and system based on near-end strategy optimization to address the above-mentioned technical problems.

[0005] A nonlinear frequency sweeping interference-resistant communication method based on near-end strategy optimization, the method comprising: The anti-nonlinear frequency sweeping interference communication process is modeled as a Markov decision process. At each time step, the state includes the historical interference trajectory, the current transmission channel, and the distance between the current interference channel and the transmission channel. The action includes the selection of the transmission channel in the next time slot. The reward value includes the product of the penalty term corresponding to the interference avoidance effect and the penalty term corresponding to the channel switching cost. The penalty term for the interference avoidance effect is assigned a value according to the distance gradient between the current interference channel and the transmission channel. A deep neural network model is constructed, comprising a policy network and a value network; the policy network is used to output the selection probability of each transmission channel based on the current state, and the value network is used to evaluate the estimated value of the current state. Under the guidance of the current policy network, the transmitter and receiver synchronously execute the transmission channel selection decision of the previous time slot, switch to the corresponding channel for data communication, and store the interaction data between the receiver and the jammer as experience samples in the experience pool; the interaction data includes the previous state, action, reward value and current state; Experience samples are sampled from the experience pool, and the parameters of the policy network and value network are optimized using the near-end policy optimization algorithm and the experience samples. An adaptive optimization algorithm is then used to iteratively update the model weights. The updated policy network is used to select the transmission channel for the next time slot and the feedback is sent back to the transmitter.

[0006] A nonlinear frequency sweeping interference-resistant communication system based on near-end strategy optimization, the system comprising: The model building module is used to model the anti-nonlinear frequency sweeping interference communication process as a Markov decision process. At each time step, the state includes the historical interference trajectory, the current transmission channel, and the distance between the current interference channel and the transmission channel. The action includes the selection of the transmission channel in the next time slot. The reward value includes the product of the penalty term corresponding to the interference avoidance effect and the penalty term corresponding to the channel switching cost. The penalty term for the interference avoidance effect is assigned a value according to the distance gradient between the current interference channel and the transmission channel. A network construction module is used to construct a deep neural network model including a policy network and a value network; the policy network is used to output the selection probability of each transmission channel according to the current state, and the value network is used to evaluate the estimated value of the current state. The sample acquisition module is used to enable the transmitter and receiver to synchronously execute the transmission channel selection decision of the previous time slot under the guidance of the current policy network, switch to the corresponding channel for data communication, and store the interaction data between the receiver and the jammer as experience samples in the experience pool; the interaction data includes the previous state, action, reward value and current state. The network update module is used to sample experience samples from the experience pool, optimize the parameters of the policy network and the value network using the near-end policy optimization algorithm and the experience samples, and use an adaptive optimization algorithm to complete the iterative update of the model weights. The result output module is used to select the transmission channel for the next time slot using the updated policy network and feed it back to the transmitter.

[0007] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program performing the following steps: The anti-nonlinear frequency sweeping interference communication process is modeled as a Markov decision process. At each time step, the state includes the historical interference trajectory, the current transmission channel, and the distance between the current interference channel and the transmission channel. The action includes the selection of the transmission channel in the next time slot. The reward value includes the product of the penalty term corresponding to the interference avoidance effect and the penalty term corresponding to the channel switching cost. The penalty term for the interference avoidance effect is assigned a value according to the distance gradient between the current interference channel and the transmission channel. A deep neural network model is constructed, comprising a policy network and a value network; the policy network is used to output the selection probability of each transmission channel based on the current state, and the value network is used to evaluate the estimated value of the current state. Under the guidance of the current policy network, the transmitter and receiver synchronously execute the transmission channel selection decision of the previous time slot, switch to the corresponding channel for data communication, and store the interaction data between the receiver and the jammer as experience samples in the experience pool; the interaction data includes the previous state, action, reward value and current state; Experience samples are sampled from the experience pool, and the parameters of the policy network and value network are optimized using the near-end policy optimization algorithm and the experience samples. An adaptive optimization algorithm is then used to iteratively update the model weights. The updated policy network is used to select the transmission channel for the next time slot and the feedback is sent back to the transmitter.

[0008] A computer-readable storage medium having a computer program stored thereon, the computer program performing the following steps when executed by a processor: The anti-nonlinear frequency sweeping interference communication process is modeled as a Markov decision process. At each time step, the state includes the historical interference trajectory, the current transmission channel, and the distance between the current interference channel and the transmission channel. The action includes the selection of the transmission channel in the next time slot. The reward value includes the product of the penalty term corresponding to the interference avoidance effect and the penalty term corresponding to the channel switching cost. The penalty term for the interference avoidance effect is assigned a value according to the distance gradient between the current interference channel and the transmission channel. A deep neural network model is constructed, comprising a policy network and a value network; the policy network is used to output the selection probability of each transmission channel based on the current state, and the value network is used to evaluate the estimated value of the current state. Under the guidance of the current policy network, the transmitter and receiver synchronously execute the transmission channel selection decision of the previous time slot, switch to the corresponding channel for data communication, and store the interaction data between the receiver and the jammer as experience samples in the experience pool; the interaction data includes the previous state, action, reward value and current state; Experience samples are sampled from the experience pool, and the parameters of the policy network and value network are optimized using the near-end policy optimization algorithm and the experience samples. An adaptive optimization algorithm is then used to iteratively update the model weights. The updated policy network is used to select the transmission channel for the next time slot and the feedback is sent back to the transmitter.

[0009] The aforementioned anti-nonlinear frequency sweeping interference communication method and system based on near-end policy optimization models the anti-nonlinear frequency sweeping interference communication process as a Markov decision process with multi-dimensional states. By integrating historical interference trajectories, the current transmission channel, and the distance between the interference and transmission channels, it can comprehensively capture the dynamic changing trends and real-time location correlations of frequency sweeping interference. Utilizing a dual-network architecture of a policy network and a value network, the policy network outputs the channel selection probability for each state, while the value network evaluates the long-term benefits of each state. Combined with a gradient-based reward mechanism that addresses both interference avoidance and switching costs, it achieves accurate channel selection decisions, avoiding blind switching or passive interference. By storing interactive data in an experience pool, and using a near-end policy optimization algorithm to optimize the dual-network parameters and combining adaptive optimization to complete weight iteration, it can quickly adapt to nonlinear changes in the frequency sweeping direction and rate, ensuring the stability and real-time performance of policy updates. This invention enables wireless communication systems to autonomously learn optimal anti-interference strategies, efficiently suppress dynamic nonlinear frequency sweeping interference, and significantly improve the reliability of wireless communication transmission in complex electromagnetic environments. Attached Figure Description

[0010] Figure 1 This is a flowchart illustrating a nonlinear frequency sweeping interference-resistant communication method based on near-end strategy optimization in one embodiment. Figure 2 This is a time slot diagram in one embodiment; Figure 3 This is a schematic diagram of the structure of the policy network and the value network in one embodiment, wherein, Figure 3 (a) is a schematic diagram of the policy network structure. Figure 3 (b) is a schematic diagram of the value network structure; Figure 4 This is a flowchart illustrating a nonlinear frequency sweeping interference-resistant communication method based on near-end strategy optimization in a specific embodiment. Figure 5 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0011] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0012] First, we make the following assumptions: 1. A wireless communication system includes a pair of transceivers. The transceiver and receiver synchronize according to communication time slots (hereinafter referred to as "time slots"). The time slot number is represented as follows: The system's available frequency bands are: , at equal intervals Divided into The system uses three non-overlapping communication channels. To achieve interference-resistant communication, the system dynamically switches between these channels. Avoid interference signals.

[0013] 2. The receiver possesses broadband spectrum sensing, learning, and decision-making capabilities, enabling it to detect interference signals within the available frequency band in real time and feed back channel selection commands to the transmitter via a reliable control channel. The receiver is equipped with three parallel functional modules: data communication, spectrum sensing, and intelligent learning. Therefore, both the transmitter and receiver can perform data transmission, broadband spectrum sensing, and learning and decision-making in parallel during each time slot, as shown in the attached diagram. Figure 2 As shown.

[0014] 3. The jammer in The system transmits dynamic nonlinear frequency sweep interference within the frequency band. Each time slot has a certain probability of changing its sweep rate and direction. By injecting malicious interference signals into the scanned communication channel, it forces the signal-to-interference-plus-noise ratio (SJNR) of the receiving end of the wireless communication system to be less than the demodulation threshold, thereby blocking communication.

[0015] In one embodiment, such as Figure 1 As shown, a nonlinear frequency sweeping interference-resistant communication method based on near-end strategy optimization is provided, including the following steps: Step 102: The anti-nonlinear frequency sweeping interference communication process is modeled as a Markov decision process. At each time step, the state includes the historical interference trajectory, the current transmission channel, and the distance between the current interference channel and the transmission channel. The action includes the selection of the transmission channel in the next time slot. The reward value includes the product of the penalty term corresponding to the interference avoidance effect and the penalty term corresponding to the channel switching cost. The penalty term for the interference avoidance effect is assigned a value according to the distance gradient between the current interference channel and the transmission channel.

[0016] The environmental state is defined as a vector representation of the historical interference environmental state observed by the system and the current communication state. The state of each time slot is represented as follows: (1) in, Indicates the first Discretized values ​​of the interference channel for each time slot. , For the historical length parameter, Indicates the transmission channel used in the current time slot. This represents the distance characteristic component between the current interfering channel and the transmission channel. Among them, the historical interference trajectory component... ,in Indicates the first The frequency of the interference signal in each time slot, For the channel discretization function: (2) Note that when there is enough historical data ( The interference frequency corresponding to the time slot is discretized according to equation (2); when historical records are insufficient, a discrete uniform random distribution is used to complete the data. The distance characteristic components of the current interference channel and communication channel. ,in The communication channel selection (i.e., action) made for the previous time slot.

[0017] The action is defined as the next time slot transmission channel that the system can select. The action of each time slot is represented as follows: (3) The reward function is designed to guide the system in making optimal channel selection decisions under dynamic nonlinear frequency sweeping interference environments. The reward value comprehensively considers two key factors: interference avoidance effectiveness and channel handover cost, employing a product form to achieve multi-objective balance. The system, based on the... The action selected in the previous time slot is executed in this time slot. Combined with the state of the previous time slot , can be in the first The reward for each time slot is calculated using the following formula. : (4) in, For channel distance penalty term, The channel handover penalty term is calculated using the following two formulas: (5) (6) Among them, the transmission channel of the previous time slot and distance feature components From the state The current time slot transmission channel is directly derived from this. From the action The current time slot distance feature component is obtained. It can be based on the frequency of the sensed interference signal. The result calculated using equation (2) , and then combine Calculation .

[0018] Step 104: Construct a deep neural network model including a policy network and a value network. The policy network is used to output the selection probability of each transmission channel based on the current state, and the value network is used to evaluate the estimated value of the current state.

[0019] like Figure 3 As shown, a schematic diagram of a policy network and a value network is provided, wherein, Figure 3 (a) is a schematic diagram of the policy network structure. Figure 3 (b) is a schematic diagram of the value network structure, showing the construction of both a policy network and a value network. Both networks consist of an input layer, two hidden layers, and an output layer. The policy network input state vector... Output probability distribution , indicating state The probability of selecting each action is given. The number of nodes in the input layer of the policy network is... The number of output layer nodes is The number of hidden layer nodes is set to a value according to the empirical rules for deep neural networks. , This represents the rounding operation function. The weight matrix of the policy network. Size is Initialization using a uniform He distribution: starting from a uniform distribution Sampling, in which The corresponding bias matrix Size is Initialize to a small constant of 0.1; Size is Initialization using a uniform He distribution: starting from a uniform distribution Sampling, in which The corresponding bias matrix Size is Initialize to a small constant of 0.1; Size is Initialize using a scaled-down Xavier uniform distribution: from a uniform distribution Sampling, in which Corresponding bias matrix Size is It is initialized to a small negative constant of -0.01. Similarly, the value network input state vector... Output state value The corresponding number of input layer nodes is The output layer has 1 node, and the hidden layer has 1 node. Weight matrix of the value network Size is Initialized using a uniform He distribution, the corresponding bias matrix Size is Initialize to a small constant of 0.1; Size is Similarly, using He uniform distribution initialization, the corresponding bias matrix... Size is Initialize to a small constant of 0.1; Size is Initialization using a Xavier uniform distribution: starting from a uniform distribution Sampling, in which Corresponding bias matrix Size is It is initialized to 0.

[0020] Step 106: Under the guidance of the current policy network, the transmitter and receiver synchronously execute the transmission channel selection decision of the previous time slot, switch to the corresponding channel for data communication, and store the interaction data between the receiver and the jammer as experience samples in the experience pool.

[0021] The interactive data includes the previous state, action, reward value, and current state. The receiver's intelligent learning module uses the previous time slot state recorded by the spectrum sensing module. The current action performed by the data communication module Calculate according to formula (4) Thus, a complete set of empirical samples is obtained. Store it in the experience pool.

[0022] Step 108: Sample experience samples from the experience pool, optimize the parameters of the policy network and value network using the near-end policy optimization algorithm and experience samples, and use the adaptive optimization algorithm to complete the iterative update of the model weights.

[0023] The receiver's intelligent learning module completes one update of the policy network and value network of the near-end policy optimization algorithm by following the steps of experience pool sampling, calculating value error, calculating advantage function, calculating the ratio of new to old policies, calculating policy loss after pruning, calculating value function loss, and backpropagation to update network parameters.

[0024] Step 110: Select the transmission channel for the next time slot using the updated policy network and feed it back to the transmitter.

[0025] The receiver's intelligent learning module inputs the current state into the policy network of the near-end policy optimization algorithm. The selection probability of each channel in the next time slot is obtained. The channel is selected based on this probability sampling to obtain the transmission channel for the next time slot. The data is transmitted back to the transmitter via a reliable control channel, and then the transmitter will perform the same action in the next time slot to communicate with the data.

[0026] In the aforementioned anti-nonlinear frequency sweeping interference communication method based on near-end policy optimization, the anti-nonlinear frequency sweeping interference communication process is modeled as a Markov decision process with multi-dimensional states. By integrating historical interference trajectories, the current transmission channel, and the distance between the interference and transmission channels, it can comprehensively capture the dynamic changing trend and real-time location correlation of frequency sweeping interference. Utilizing a dual-network architecture of a policy network and a value network, the policy network outputs the channel selection probability for each state, while the value network evaluates the long-term benefits of each state. Combined with a gradient-based reward mechanism that addresses both interference avoidance and switching costs, accurate channel selection decisions can be achieved, avoiding blind switching or passive interference. Interactive data is stored in an experience pool, and the parameters of the dual networks are optimized using a near-end policy optimization algorithm, combined with adaptive optimization to complete weight iteration. This allows for rapid adaptation to nonlinear changes in the frequency sweeping direction and rate, ensuring the stability and real-time performance of policy updates. This invention enables wireless communication systems to autonomously learn optimal anti-interference strategies, efficiently suppress dynamic nonlinear frequency sweeping interference, and significantly improve the reliability of wireless communication transmission in complex electromagnetic environments.

[0027] In one embodiment, the method further includes: if communication has not ended, proceeding to the next time slot and returning to the step of synchronously executing the transmission channel selection decision of the previous time slot with the transmitter and receiver, continuously iterating and optimizing the policy network and value network; if communication has ended, stopping the iteration.

[0028] In one embodiment, the step of calculating the reward value includes: determining an interference avoidance effect penalty term according to a preset gradient based on the distance between the current interference channel and the transmission channel; determining a channel switching cost penalty term based on whether the current transmission channel is consistent with the previous time slot transmission channel and whether the distance between the interference channel and the transmission channel increases after the handover; and multiplying the interference avoidance effect penalty term and the channel switching cost penalty term to obtain the reward value.

[0029] In one embodiment, the step of obtaining historical interference trajectories includes: discretizing the interference signal frequencies of historical time slots, mapping the interference signal frequencies to corresponding channel numbers, and obtaining historical interference trajectories based on the discretized values ​​of the interference channels in each time slot; when the number of historical time slot records is insufficient for a preset historical length parameter, values ​​are extracted from all available channel numbers using a discrete uniform random distribution to supplement the historical interference trajectories. In this embodiment, by constructing historical interference trajectories, interference channel information from consecutive time slots can be integrated, thereby inferring the directional trend and rate change of nonlinear frequency sweeping. This design, on the one hand, achieves more accurate capture of historical interference states, avoiding the one-sidedness of single interference information; on the other hand, it can effectively predict the nonlinear change trend of interference, providing more comprehensive dynamic support for the channel selection decision of the subsequent policy network, which is conducive to the policy network outputting channel selection probabilities that are more adapted to the characteristics of interference changes, and improving the decision accuracy against nonlinear frequency sweeping interference.

[0030] In one embodiment, both the policy network and the value network include an input layer, two hidden layers, and an output layer. The number of nodes in the input layer of the policy network is consistent with the state dimension, the number of nodes in the output layer is consistent with the number of available transmission channels, and the number of nodes in the hidden layer is the square root of the product of the number of nodes in the input layer and the number of nodes in the output layer, rounded to the nearest whole number. The number of nodes in the input layer of the value network is consistent with the state dimension, the number of nodes in the output layer is 1, and the number of nodes in the hidden layer is the same as that in the policy network.

[0031] In one embodiment, optimizing the parameters of the policy network and value network using a proximal policy optimization algorithm and empirical samples includes: calculating the target value and value error based on empirical samples using the proximal policy optimization algorithm, and obtaining the advantage function based on the value error; limiting the ratio of new to old policies within a preset range to restrict the update amplitude of the policy network, and calculating the policy loss using the advantage function and the restricted ratio of new to old policies; calculating the value loss based on the target value and the estimated value of the value network; minimizing the policy loss and value loss respectively through gradient backpropagation, and adjusting the parameters of the policy network and value network; calculating the target value and value error based on empirical samples using the proximal policy optimization algorithm includes: inputting the current state from the empirical samples into the value network to obtain the estimated value of the current state, calculating the target value of the current state based on the estimated value, a preset future reward discount coefficient, and the reward value in the empirical samples; and calculating the value error by the difference between the target value of the current state and the estimated value of the previous state.

[0032] Specifically, a batch of experience samples is extracted from the experience pool P. The batch size is set to 64 in this invention.

[0033] For each sample, Input value network, get , bring in ,in This is a discount factor reflecting the importance of future rewards; in near-term policy optimization algorithms, this hyperparameter is typically set to 0.99; then input into the value network... ,get ; Calculate the value error .

[0034] This invention employs online learning of proximal strategy optimization, thus simplifying the calculation of the advantage function. Advantage function Indicates action Better than average. Indicates action It is worse than the average level.

[0035] For each sample, Input the policy network to obtain the probability of choosing each action in this state. This serves as a benchmark for subsequent comparisons. K The optimization iterations are performed 4 to 10 times (5 in this invention), and each iteration first calculates the ratio of the old and new strategies using the following formula: (7) in, This represents the parameters of the current policy network, including all weight matrices and bias matrices, which are updated in each iteration; Indicates the current parameter Below, policy network input The probability distribution of all actions output at that time.

[0036] Calculate the strategy loss after pruning using the following formula: (8) in, It is the hyperparameter pruning factor. Based on experience, this invention sets it to 0.1 to limit the update range of the strategy. Indicates will Limited to the range Inside; This indicates that the average value is calculated for all samples in the current batch.

[0037] Calculate the value loss based on the standard average error: (9) in, The parameters representing the current value network, including all weight matrices and bias matrices, are updated in each iteration; Indicates the current parameter Below, value network input The estimated value of the output at that time; This indicates that the average value is calculated for all samples in the current batch.

[0038] Backpropagation to calculate strategy loss Policy network parameters gradient and value loss Value network parameters The gradient is calculated, and then the parameters are updated using two independent Adam optimizers. and : (10) (11) in, and These are the learning rates of the policy network and the value network, respectively, used to control the magnitude of each parameter update. In this invention, they are set to 0.0001 and 0.003, respectively. The updates of the policy network and the value network are performed by calculating gradients using the backpropagation method and updating the network parameters using the Adam optimizer.

[0039] In one specific embodiment, such as Figure 4 The diagram illustrates a flowchart of a nonlinear frequency sweep interference-resistant communication method based on near-end strategy optimization. This method is applied to a wireless communication system, which includes a transmitter and a receiver. The transmitter and receiver are synchronized according to time slots, and the method includes the following steps: Step 1: Model the anti-nonlinear frequency sweeping interference communication problem as a Markov decision process; Step 2: Construct the policy network and value network for the near-end policy optimization algorithm and initialize the parameters; Step 3: The transmitter and receiver execute the action selected in the previous time slot and switch to the corresponding channel for data communication; Step 4: The receiver observes the nonlinear frequency sweep interference environment, obtains the current status, and records it; Step 5: The receiver calculates the reward corresponding to the state and action of the previous time slot and stores the obtained experience sample into the experience pool. Step 6: The receiver updates the policy network and value network parameters; Step 7: The receiver uses the policy network to select the transmission channel for the next time slot and sends the result back to the transmitter; Step 8: If the communication has not ended, proceed to the next time slot and return to step 3 until the communication ends.

[0040] It is understandable that step 1 models the communication problem as a Markov decision process. By defining a multi-dimensional state that includes historical interference trajectories, the current transmission channel, and the distance between the interference and the transmission channel, it overcomes the limitations of traditional methods that rely solely on real-time interference parameters, achieving a comprehensive perception of the sweep direction and rate change trends, and providing complete environmental information for policy learning. Step 2 constructs a dual-network architecture of a policy network and a value network, laying the foundation for dynamic policy optimization. The policy network is responsible for outputting the channel selection probability, and the value network evaluates the long-term benefits of the state, avoiding the shortsightedness of single-network decision-making. Steps 3-5 involve the transceiver synchronously performing channel switching, the receiver observing the interference environment in real time, and calculating multiple objectives. By balancing rewards and storing experience samples, the network can continuously acquire training data under real interference environments, ensuring that the decision-making strategy fits the actual scenario. Step 6 uses a near-end policy optimization algorithm to update network parameters. By limiting the policy update amplitude, decision oscillations caused by sudden interference changes are avoided. At the same time, batch sampling from the experience pool is combined to improve the efficiency of parameter updates, balancing the real-time performance and stability of policy optimization. Steps 7 and 8 allow the policy network to continuously learn and optimize the channel selection strategy through iterative iteration. Ultimately, the system can autonomously adapt to nonlinear changes in the sweep direction and rate, effectively suppressing sweep interference without prior modeling of interference characteristics, and significantly improving the reliability of wireless communication transmission in complex electromagnetic environments.

[0041] It should be understood that, although Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0042] In one embodiment, a nonlinear frequency sweep interference-resistant communication system based on near-end strategy optimization is provided, comprising: The model building module is used to model the anti-nonlinear frequency sweeping interference communication process as a Markov decision process. At each time step, the state includes the historical interference trajectory, the current transmission channel, and the distance between the current interference channel and the transmission channel. The action includes the selection of the transmission channel in the next time slot. The reward value includes the product of the penalty term corresponding to the interference avoidance effect and the penalty term corresponding to the channel switching cost. The penalty term for the interference avoidance effect is assigned according to the gradient of the distance between the current interference channel and the transmission channel. The network construction module is used to build a deep neural network model that includes a policy network and a value network. The policy network is used to output the selection probability of each transmission channel based on the current state, and the value network is used to evaluate the estimated value of the current state. The sample acquisition module is used to enable the transmitter and receiver to synchronously execute the transmission channel selection decision of the previous time slot under the guidance of the current policy network, switch to the corresponding channel for data communication, and store the interaction data between the receiver and the jammer as experience samples in the experience pool; the interaction data includes the previous state, action, reward value and current state; The network update module is used to sample experience samples from the experience pool, optimize the parameters of the policy network and value network using the near-end policy optimization algorithm and experience samples, and use an adaptive optimization algorithm to complete the iterative update of the model weights. The result output module is used to select the transmission channel for the next time slot using the updated policy network and feed it back to the transmitter.

[0043] Specific limitations regarding the near-end strategy optimization-based anti-nonlinear frequency sweep interference communication system can be found in the limitations of the near-end strategy optimization-based anti-nonlinear frequency sweep interference communication method described above, and will not be repeated here. Each module in the aforementioned near-end strategy optimization-based anti-nonlinear frequency sweep interference communication system can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in the computer device in hardware form, or stored in the memory of the computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0044] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 5 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used for communication with external terminals via a network connection. When executed by the processor, the computer program implements a near-end strategy-optimized anti-nonlinear frequency sweep interference communication method. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0045] Those skilled in the art will understand that Figure 5The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0046] In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of the method described above.

[0047] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described above.

[0048] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0049] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0050] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A communication method for resisting nonlinear frequency sweeping interference based on near-end strategy optimization, characterized in that, The method includes: The anti-nonlinear frequency sweeping interference communication process is modeled as a Markov decision process. At each time step, the state includes the historical interference trajectory, the current transmission channel, and the distance between the current interference channel and the transmission channel. The action includes the selection of the transmission channel in the next time slot. The reward value includes the product of the penalty term corresponding to the interference avoidance effect and the penalty term corresponding to the channel switching cost. The penalty term for the interference avoidance effect is assigned a value according to the distance gradient between the current interference channel and the transmission channel. A deep neural network model is constructed, comprising a policy network and a value network; the policy network is used to output the selection probability of each transmission channel based on the current state, and the value network is used to evaluate the estimated value of the current state. Under the guidance of the current policy network, the transmitter and receiver synchronously execute the transmission channel selection decision of the previous time slot, switch to the corresponding channel for data communication, and store the interaction data between the receiver and the jammer as experience samples in the experience pool; the interaction data includes the previous state, action, reward value and current state; Experience samples are sampled from the experience pool, and the parameters of the policy network and value network are optimized using the near-end policy optimization algorithm and the experience samples. An adaptive optimization algorithm is then used to iteratively update the model weights. The updated policy network is used to select the transmission channel for the next time slot and the feedback is sent back to the transmitter.

2. The method according to claim 1, characterized in that, The method further includes: If the communication has not ended, proceed to the next time slot and return to the transmitter and receiver to synchronously execute the transmission channel selection decision steps of the previous time slot, continuously iterating and optimizing the policy network and value network; If communication ends, the iteration stops.

3. The method according to claim 1, characterized in that, The steps for calculating the reward value include: Based on the distance between the current interference channel and the transmission channel, the penalty term for interference avoidance effect is determined according to a preset gradient; The channel handover cost penalty is determined based on whether the current transmission channel is consistent with the previous time slot transmission channel and whether the distance between the interference channel and the transmission channel increases after the handover. The reward value is obtained by multiplying the interference avoidance effect penalty term with the channel handover cost penalty term.

4. The method according to claim 1, characterized in that, The steps for obtaining the historical interference trajectory include: The interference signal frequency of the historical time slot is discretized and mapped to the corresponding channel number. The historical interference trajectory is obtained based on the discretized value of the interference channel of each time slot. When the number of historical time slot records is insufficient to meet the preset historical length parameter, a discrete uniform random distribution is used to extract values ​​from all available channel numbers to supplement the historical interference trajectory.

5. The method according to claim 1, characterized in that, Both the policy network and the value network include an input layer, two hidden layers, and an output layer. The number of input layer nodes in the policy network is consistent with the state dimension, the number of output layer nodes is consistent with the number of available transmission channels, and the number of hidden layer nodes is the square root of the product of the number of input layer nodes and the number of output layer nodes, rounded to the nearest whole number. The number of nodes in the input layer of the value network is the same as the number of nodes in the state dimension, the number of nodes in the output layer is 1, and the number of nodes in the hidden layer is the same as that in the policy network.

6. The method according to claim 1, characterized in that, The parameters of the policy network and value network optimized using the proximal policy optimization algorithm and the empirical samples include: The near-end strategy optimization algorithm is used to calculate the target value and value error based on the empirical samples, and the advantage function is obtained based on the value error. The ratio of new to old policies is limited to a preset range to restrict the update range of the policy network. The policy loss is calculated using the advantage function and the limited ratio of new to old policies. Calculate value loss based on the estimated value of the target value and the value network; The policy loss and value loss are minimized respectively through gradient backpropagation, and the parameters of the policy network and value network are adjusted.

7. The method according to claim 6, characterized in that, The near-end strategy optimization algorithm calculates the target value and value error based on the empirical samples, including: Input the current state from the experience sample into the value network to obtain the estimated value of the current state. Calculate the target value of the current state based on the estimated value, the preset future reward discount coefficient, and the reward value in the experience sample. The value error is calculated by the difference between the target value of the current state and the estimated value of the previous state.

8. A communication system for resisting nonlinear frequency sweeping interference based on near-end strategy optimization, characterized in that, The system includes: The model building module is used to model the anti-nonlinear frequency sweeping interference communication process as a Markov decision process. At each time step, the state includes the historical interference trajectory, the current transmission channel, and the distance between the current interference channel and the transmission channel. The action includes the selection of the transmission channel in the next time slot. The reward value includes the product of the penalty term corresponding to the interference avoidance effect and the penalty term corresponding to the channel switching cost. The penalty term for the interference avoidance effect is assigned a value according to the distance gradient between the current interference channel and the transmission channel. A network construction module is used to construct a deep neural network model including a policy network and a value network; the policy network is used to output the selection probability of each transmission channel according to the current state, and the value network is used to evaluate the estimated value of the current state. The sample acquisition module is used to enable the transmitter and receiver to synchronously execute the transmission channel selection decision of the previous time slot under the guidance of the current policy network, switch to the corresponding channel for data communication, and store the interaction data between the receiver and the jammer as experience samples in the experience pool; the interaction data includes the previous state, action, reward value and current state. The network update module is used to sample experience samples from the experience pool, optimize the parameters of the policy network and the value network using the near-end policy optimization algorithm and the experience samples, and use an adaptive optimization algorithm to complete the iterative update of the model weights. The result output module is used to select the transmission channel for the next time slot using the updated policy network and feed it back to the transmitter.

Images