A cooperative communication interference decision-making method

Abstract

The disclosed embodiment is about a collaborative communication interference decision method, which includes: establishing a collaborative decision model and providing a decision environment; based on deep reinforcement learning, an interference decision algorithm integrating advantage functions is proposed under the Actor-Critic algorithm framework, and the optimal interference solution is output according to the interference decision algorithm. Through the above-mentioned collaborative interference decision method, according to the constructed collaborative decision model, the optimal interference solution is output according to the interference decision algorithm. The interference solution provided in the disclosed embodiment can achieve the optimal utilization of interference resources and improve decision efficiency.

Description

Technical Field

[0001] The present invention relates to the field of control technology, and in particular to a collaborative communication interference decision-making method. Background Art

[0002] In the field of communication confrontation, system confrontation and coordinated interference have become the main combat application methods. How to allocate interference resources and maximize resource utilization is an important problem that needs to be solved urgently, which brings huge challenges to command decision-making. Some cognitive radio interference decision-making research based on game theory, stochastic theory and other methods has made certain progress. These studies set up a confrontation scenario between the two interference parties, derive the game benefit function, and calculate the interference parameters such as interference style and power to obtain the optimal interference strategy. Although such methods can output better results, the applicable scenarios are relatively simple and cannot meet the current multi-dimensional collaborative battlefield environment. There is still a large gap with actual combat use.

[0003] Among the related technologies, research on cognitive electronic warfare based on artificial intelligence technology has made great breakthroughs in recent years. Intelligent interference decision-making is a key link, and intelligent decision-making is generally achieved based on deep reinforcement learning technology. Deep reinforcement learning is a machine learning method that maximizes the evaluation benefit value by interacting with the intelligent agent and the environment, outputting action plans through neural network fitting, and guiding network training and updating through environmental feedback. It can give better decision results through interactive learning without prior information or with less prior information, and is widely used in military intelligence fields such as battlefield resource optimization, command and coordination control, etc. In terms of communication interference decision-making, some methods establish a multi-arm selection mechanism model and a bit error rate curve dictionary. Through dictionary sampling and algorithm calculation, the jammer can construct a bit error rate curve similar to the actual curve and learn the optimal interference strategy under three interactions; some methods apply the multi-arm selection mechanism model to obtain the most efficient power allocation interference scheme by deciding the interference signal style, data packet sending instructions, and physical layer parameters such as power level; in addition, in order to solve the problem of slow convergence of reinforcement learning algorithms in interference decision-making, some methods use equivalent parameter modeling, reduce the dimension of interference parameter selection search space, add previous interference experience information, and output the best interference strategy while shortening the system learning time; based on the overall confrontation idea, the BHJM algorithm is proposed, which can give priority to interference with frequency hopping communication targets with higher threat levels under the condition of insufficient interference resources, and output the interference scheme with the highest resource utilization. However, the above studies all give optimized interference schemes for a certain signal system or a single interference station, and cannot solve the problems of collaborative interference decision-making and resource allocation.

[0004] Therefore, it is necessary to provide a new technical solution to improve one or more problems existing in the above solutions.

[0005] It should be noted that this section is intended to provide a background or context to embodiments of the invention recited in the claims. No admission is made that the description herein is prior art by inclusion in this section. Summary of the invention

[0006] The purpose of the present disclosure is to provide a collaborative communication interference decision method, thereby overcoming one or more problems caused by limitations and defects of related technologies at least to a certain extent.

[0007] According to an embodiment of the present disclosure, a cooperative communication interference decision method is provided, the method comprising:

[0008] Establish a collaborative decision-making model and provide a decision-making environment;

[0009] Based on deep reinforcement learning, an interference decision algorithm integrating advantage function is proposed under the Actor-Critic algorithm framework, and the optimal interference plan is output according to the interference decision algorithm.

[0010] In the embodiment of the present disclosure, the basic elements required to define the collaborative decision-making model are: state space and action space; wherein the state space represents the number of frequency points where a target frequency hopping signal is not interfered with as h, and the state space S is defined as [h1,h2,...,h n ], which means the number of frequencies of all target frequency-hopping signals that are not interfered with; the action space defines that the interference action output by the decision network is A, which means the deployment position of the interference station and the interference action code corresponding to the interference direction angle. In order to reduce the decision dimension of the algorithm, an angle can be selected as the interference direction angle every a° in the range of 0° to 180°.

[0011] In the embodiment of the present disclosure, based on deep reinforcement learning, an interference decision algorithm integrating advantage function is proposed under the Actor-Critic algorithm framework, and the optimal interference solution is output according to the interference decision algorithm, including:

[0012] Through the strategy execution network perception environment state, obtain the frequency point information S of each target signal to be interfered at time t t , the strategy is used to execute the fitting operation of the network to output the interference action A of each interference station at time t t .

[0013] In the embodiment of the present disclosure, based on deep reinforcement learning, an interference decision algorithm integrating advantage function is proposed under the Actor-Critic algorithm framework, and the optimal interference solution is output according to the interference decision algorithm, including:

[0014] The value evaluation network obtains the frequency point information S of each target signal to be interfered at time t and time t+1. t and S t+1, and output the interference action A at time t respectively t The state value V(S t ) and the interference action A at time t+1 t+1 The state value V(S t+1 ) to estimate the pros and cons of the strategy of the current strategy execution network.

[0015] In the embodiment of the present disclosure, based on deep reinforcement learning, an interference decision algorithm integrating advantage function is proposed under the Actor-Critic algorithm framework, and the optimal interference solution is output according to the interference decision algorithm, including:

[0016] The reward function is set based on a global concept, and by introducing expert incentives into the reward function, an evaluation is given for the state change caused by each interfering station performing the interference action, that is, the reward value r after the expert incentives are introduced is calculated and output.

[0017] In the disclosed embodiment, the introduction of expert incentives into the reward function includes:

[0018] In the basic reward value r base Add an additional expert incentive value r exp , the reward value r after the introduction of the expert incentive is obtained, the expert incentive value r exp Provide expert guidance for subsequent decisions and internal incentives for current decisions;

[0019] Among them: Expert incentive value r exp Defined as:

[0020] r exp =N cha ×(N jam +1) (1)

[0021] The reward value r after introducing expert incentives is defined as:

[0022] r=r base +r exp (2)

[0023] Where: N cha Indicates the number of targets that have been interfered, N jam Indicates the number of targets successfully interfered by the current jammer.

[0024] In the embodiment of the present disclosure, based on deep reinforcement learning, an interference decision algorithm integrating advantage function is proposed under the Actor-Critic algorithm framework, and the optimal interference solution is output according to the interference decision algorithm, including:

[0025] The advantage function is introduced to compare the advantages and disadvantages of action value and state value; the formula of the advantage function is as follows:

[0026] A(S t ,A t )=Q(S t ,A t )=r+V(S t+1 )-V(S t ) (3)

[0027] Among them, Q(S t ,A t ) represents the action value, and the advantage function A(S t ,A t ) indicates that the interference action A is executed at time t t After that, the action value Q(S t ,A t ) compared to the state value V(S t )’s degree of advantage.

[0028] In the disclosed embodiment, the policy execution network parameters and the value evaluation network parameters are respectively trained and optimized by a training optimization module; wherein the training optimization module includes a loss function of the value evaluation network and a loss function of the policy execution network, and the formulas of the loss function of the value evaluation network and the loss function of the policy execution network are respectively;

[0029] L(θ V )=A(S t ,A t θ V ) 2 =[r+γV(S t+1 θ V )-V(S t θ V )] 2 (4)

[0030] R(θ)=A(S t ,A t θ V )logp θ (A t |S t )=[r+γV(S t+1 θ V )-V(S t θ V )]logp θ (A t |S t ) (5)

[0031] Among them, L(θV ) represents the loss function of the value evaluation network, R(θ) represents the loss function of the policy execution network, θ represents the policy execution network parameters, and θ v represents the value evaluation network parameter, γ represents the attenuation factor, and p θ represents the policy probability of the policy execution network.

[0032] In the embodiment of the present disclosure, according to the A(S) output by the value evaluation network, t ,A t θ V )Optimize the policy execution network parameters so that the policy execution network can decide better interference actions.

[0033] In the embodiment of the present disclosure, based on deep reinforcement learning, an interference decision algorithm integrating advantage function is proposed under the Actor-Critic algorithm framework. After the step of outputting the optimal interference solution according to the interference decision algorithm, it also includes: calculating the interference effect according to the interference-signal ratio, and determining whether the interference is effective according to the size of the interference-signal ratio and the target suppression coefficient. The formula of the interference-signal ratio is:

[0034]

[0035] Among them, P J is the transmission power of the jammer, P S is the transmission power of the signal transmitter; H J H is the product of the gain of the jammer transmitting antenna and the signal receiving antenna. S L is the product of the signal transmitter antenna gain and the receiving antenna gain; J and L S are the spatial losses of interference signal and communication signal transmission, respectively. It indicates that the interference bandwidth of the interference station can be aligned with the part of the target signal frequency to be interfered. Indicates the power of effective interference; Indicates whether the interference frequency band is aligned with the target signal frequency to be interfered with in the frequency domain. When the interference spectrum with frequency f is aligned with the frequency f S The frequency hopping point, then the indication value is 1, otherwise it is 0.

[0036]

[0037] The technical solution provided by the embodiments of the present invention may have the following beneficial effects:

[0038] In one embodiment of the present invention, through the above-mentioned collaborative interference decision-making method, according to the constructed collaborative decision-making model, a decision environment is provided for the interference decision-making method, and based on deep reinforcement learning, an interference decision algorithm that integrates the advantage function is proposed under the Actor-Critic algorithm architecture to output the optimal interference plan according to the interference decision algorithm. The interference plan given in the embodiment of the present disclosure can achieve optimal utilization of interference resources and improve decision-making efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 A flowchart schematically illustrates a method for cooperative communication interference decision making in an exemplary embodiment of the present disclosure;

[0040] Figure 2 A schematic diagram schematically shows the positions of the interference station and the target to be interfered in an exemplary embodiment of the present disclosure;

[0041] Figure 3 A schematic diagram schematically shows a comparison of interference success rates under different algorithms in an exemplary embodiment of the present disclosure;

[0042] Figure 4 A schematic diagram schematically shows a comparison of average reward values ​​under different algorithms in an exemplary embodiment of the present disclosure;

[0043] Figure 5 A schematic diagram schematically shows a comparison of the number of interfering stations under different algorithms in an exemplary embodiment of the present disclosure;

[0044] Figure 6 A schematic diagram schematically illustrates a comparison of optimal interference solutions under different algorithms in an exemplary embodiment of the present disclosure;

[0045] Figure 7 Schematically showing a comparison diagram of interference success rates with and without expert stimulation in an exemplary embodiment of the present disclosure;

[0046] Figure 8 A schematic diagram schematically shows a comparison of the number of interfering stations when there is expert excitation and when there is no expert excitation in an exemplary embodiment of the present disclosure;

[0047] Fig. 9 The figure schematically shows the comparison of reward values ​​when there is no expert incentive in an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0048] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the present invention will be more comprehensive and complete and fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0049] In addition, the accompanying drawings are only schematic illustrations of the present invention and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities.

[0050] This example implementation first provides a cooperative communication interference decision method. Figure 1 As shown in , the method may include:

[0051] Step S101: Establish a collaborative decision-making model to provide a decision-making environment;

[0052] Step S102: Based on deep reinforcement learning, an interference decision algorithm integrating advantage function is proposed under the Actor-Critic algorithm framework, and an optimal interference solution is output according to the interference decision algorithm.

[0053] Through the above-mentioned collaborative interference decision-making method, according to the constructed collaborative decision-making model, a decision-making environment is provided for the interference decision-making method, and based on deep reinforcement learning, an interference decision algorithm that integrates the advantage function is proposed under the Actor-Critic algorithm architecture to output the optimal interference plan according to the interference decision algorithm. The interference plan given in the embodiment of the present disclosure can achieve optimal utilization of interference resources and improve decision-making efficiency.

[0054] Next, we will refer to Figure 1 Each step of the above method of this example embodiment is described in more detail.

[0055] In step S101, a collaborative decision-making model is established to provide a decision-making environment;

[0056] Specifically, the established collaborative decision-making model is an "overall optimization, station-by-station decision-making" collaborative decision-making model. This collaborative decision-making model regards each interference station as an independent intelligent agent, and makes interference actions step by step and sequentially through the same decision-making network. The interference action includes the deployment position of the interference station and the interference direction angle; when an intelligent agent makes a decision, it executes its interference action, and inputs the target information changed by the execution of the interference action into the next intelligent agent; a centralized training method is used to optimize the interference plan as a whole. When all intelligent agents have made their decisions, the weight parameters of the decision network are updated through training until convergence.

[0057] In step S102, based on deep reinforcement learning, an interference decision algorithm integrating advantage function is proposed under the Actor-Critic algorithm framework, and an optimal interference solution is output according to the interference decision algorithm.

[0058] Specifically, the jamming decision-making algorithm includes a strategy execution network, a value evaluation network, a reward evaluation module, and a training optimization module. The collaborative decision-making algorithm can be written as Advantage Function based Jamming Collaborative Decision-making algorithm, abbreviated as AFCJD.

[0059] In one embodiment, based on deep reinforcement learning, an interference decision algorithm integrating advantage function is proposed under the Actor-Critic algorithm framework, and the optimal interference solution is output according to the interference decision algorithm, including:

[0060] Through the strategy execution network perception environment state, obtain the frequency point information S of each target signal to be interfered at time t t , the strategy is used to execute the fitting operation of the network to output the interference action A of each interference station at time t t .

[0061] Specifically, the strategy execution network can be used to sense the environmental state, that is, to obtain the frequency information S of each target signal to be interfered at time t t , interference action A t Including deployment positions and interference direction angles.

[0062] In one embodiment, the basic elements required to define the collaborative decision-making model are: state space and action space; wherein the state space represents that the number of frequency points where a target frequency hopping signal is not interfered is h, and the state space S is defined as [h1,h2,...,h n], which means the number of frequencies of all target frequency-hopping signals that are not interfered with; the action space represents the definition of the interference action output by the decision network as A, which represents the deployment position of the interference station and the interference action coding corresponding to the interference direction angle. In order to reduce the decision dimension of the algorithm, an angle can be selected as the interference direction angle every a° in the range of 0° to 180°.

[0063] Specifically, it is shown in Table 1. In order to reduce the decision dimension of the algorithm, an angle can be selected as the interference direction angle every 15° in the range of 0° to 180°, and there are 11 optional angles in total.

[0064] Table 1 Interference action coding table

[0065]

[0066]

[0067] The deployment position D and interference direction angle L can be expressed by the following formula:

[0068] D=[A / 11]+1 (8)

[0069] L=(A%11+1)×15 (9)

[0070] In one embodiment, based on deep reinforcement learning, an interference decision algorithm integrating advantage function is proposed under the Actor-Critic algorithm framework, and the optimal interference solution is output according to the interference decision algorithm, including:

[0071] The value evaluation network obtains the frequency point information S of each target signal to be interfered at time t and time t+1. t and S t+1 , and output the interference action A at time t respectively t The state value V(S t ) and the interference action A at time t+1 t+1 The state value V(S t+1 ) to estimate the pros and cons of the strategy of the current strategy execution network.

[0072] Specifically, the value evaluation network obtains the frequency information S of each target signal to be interfered at time t t , output the interference action A at time t t The state value V(S t ), and the interference action A at time t+1 t+1 The state value V(S t+1 ), estimates the quality of the current policy execution network’s strategy based on the state value of the interference action.

[0073] In one embodiment, based on deep reinforcement learning, an interference decision algorithm integrating advantage function is proposed under the Actor-Critic algorithm framework, and the optimal interference solution is output according to the interference decision algorithm, including:

[0074] The reward function is set based on the global concept, and the interference action A is executed for each interference station by introducing expert incentives in the reward function. t The state change caused by the latter is evaluated, that is, the reward value r after the expert incentive is introduced is calculated and output.

[0075] Specifically, the reward function can be used to indicate the quality of the overall jamming scheme. When all frequency hopping signals are jammed, the basic reward value r base is 80; when there is no target in the interference beam, the basic reward value r base is -15; otherwise the base reward value is r base is 0, the basic reward value is r base It can be expressed as follows:

[0076]

[0077] In one embodiment, the introducing expert incentive into the reward function includes:

[0078] In the basic reward value r base Add an additional expert incentive value r exp , the reward value r after the introduction of the expert incentive is obtained, the expert incentive value r exp Provide expert guidance for subsequent decisions and internal incentives for current decisions;

[0079] Among them: Expert incentive value r exp Defined as:

[0080] r exp =N cha ×(N jam +1) (1)

[0081] The reward value r after introducing expert incentives is defined as:

[0082] r=r base +r exp (2)

[0083] Where: N cha Indicates the number of targets that have been interfered, N jam Indicates the number of targets successfully interfered by the current jammer.

[0084] Specifically, in deep reinforcement learning problems, the reward value is generally determined only based on whether the round task is completed or the round is won or lost, but this will lead to sparse reward problems, making it difficult for the decision algorithm to converge. base Add an additional expert incentive value r exp , so that the expert incentive value r exp It can continuously guide the agent to update the strategy in the direction of maximizing the cumulative value of the reward value r; base and the expert incentive value r exp The sum of the values ​​is the reward value r after embedding the expert incentive. Expert incentive value r exp It forms expert guidance for subsequent decisions and internal incentives for current decisions, as shown in formula (1), N cha The value of the expert incentive is different. exp Also different, the number of targets that have been interfered with N cha The larger the value, the closer it is to interfering with the overall goal. exp The larger the value, the larger the reward value r. Since obtaining a larger reward value r is the learning goal of the agent, the closer the interference is to the total goal, the greater the expert incentive value r. exp The stronger the incentive effect, the more expert guidance the agent can provide.

[0085] In one embodiment, based on deep reinforcement learning, an interference decision algorithm integrating advantage function is proposed under the Actor-Critic algorithm framework, and the optimal interference solution is output according to the interference decision algorithm, including:

[0086] The advantage function is introduced to compare the advantages and disadvantages of action value and state value; the formula of the advantage function is as follows:

[0087] A(S t ,A t )=Q(S t ,A t )=r+V(S t+1 )-V(S t ) (3)

[0088] Among them, Q(S t ,A t ) represents the action value, and the advantage function A(S t ,A t ) indicates that the interference action A is executed at time t t After that, the action value Q(S t ,A t ) compared to the state value V(S t ) is superior to the others. Specifically, the superiority function A(St ,A t ) indicates that the interference action A is executed at time t t After that, the action value Q(S t ,A t ) compared to the state value V(S t ) advantage, V(S t ) also represents the average action value of action A. If A(S t ,A t ) is greater than zero, it means that the interference action A t is better than the average action A. If the advantage function is less than zero, it means that the interference action A t Not as good as average action A.

[0089] In one embodiment, the policy execution network parameters and the value evaluation network parameters are respectively trained and optimized by a training optimization module, wherein the training optimization module includes a loss function of the value evaluation network and a loss function of the policy execution network, and the formulas of the loss function of the value evaluation network and the loss function of the policy execution network are respectively;

[0090] L(θ V )=A(S t ,A t θ V ) 2 =[r+γV(S t+1 θ V )-V(S t θ V )] 2 (4)

[0091] R(θ)=A(S t ,A t θ V )logp θ (A t |S t )=[r+γV(S t+1 θ V )-V(S t θ V )]logp θ (A t |S t ) (5)

[0092] Among them, L(θ V ) represents the loss function of the value evaluation network, R(θ) represents the loss function of the policy execution network, θ represents the policy execution network parameters, and θ v represents the value evaluation network parameter, γ represents the attenuation factor, and p θ represents the policy probability of the policy execution network.

[0093] Specifically, the loss function of the value evaluation network improves the accuracy of the network to the value evaluation network through continuous training, giving the strategy execution network a more accurate training target. Among them, the strategy execution network parameters and the value evaluation network parameters are weight parameters.

[0094] In one embodiment, the overall process of the AFCJD algorithm is shown in Table 2:

[0095] Table 2 AFCJD algorithm

[0096]

[0097]

[0098]

[0099] In one embodiment, according to the output of the value assessment network A(S t ,A t θ V )Optimize the policy execution network parameters so that the policy execution network can make better interference actions.

[0100] Specifically, the policy execution network decides on a better interference action, which is the output interference plan. In the algorithm, the hidden layers in the policy execution network and the value evaluation network use fully connected neural networks. The output layer of the policy execution network uses the Softmax function, and the output layer of the value evaluation network has no activation function. The remaining activation functions are all ReLu functions.

[0101] In one embodiment, based on deep reinforcement learning, an interference decision algorithm integrating advantage function is proposed under the Actor-Critic algorithm framework. After the step of outputting the optimal interference solution according to the interference decision algorithm, the method further includes: calculating the interference effect according to the interference-signal ratio, and determining whether the interference is effective according to the size of the interference-signal ratio and the target suppression coefficient. The formula of the interference-signal ratio is:

[0102]

[0103] Among them, P J is the transmission power of the jammer, P S is the transmission power of the signal transmitter; H J H is the product of the gain of the jammer transmitting antenna and the signal receiving antenna. S L is the product of the signal transmitter antenna gain and the receiving antenna gain; J and L S are the spatial losses of interference signal and communication signal transmission, respectively. It indicates that the interference bandwidth of the interference station can be aligned with the part of the target signal frequency to be interfered. Indicates the power of effective interference; Indicates whether the interference frequency band is aligned with the target signal frequency to be interfered with in the frequency domain. When the interference spectrum with frequency f is aligned with the frequency f S The frequency hopping point, then the indication value is 1, otherwise it is 0.

[0104]

[0105] Specifically, the interference effect of the interference scheme outputted is calculated according to the interference-to-signal ratio. When the interference-to-signal ratio is greater than the target suppression coefficient, the interference is effective, otherwise, the interference is ineffective.

[0106] In system electronic warfare, different deployment of jamming resources will have different effects on the jamming effect of the entire system. For example, jamming stations deployed at different locations can obtain different jamming effects for the same target, or when a jamming station can interfere with multiple targets at the same time, jamming different targets will also affect the task allocation of the remaining resources. Therefore, the difficulty of coordinated jamming lies in how to reasonably allocate the jamming resources of multiple stations to maximize their jamming efficiency. When the jamming station is aimed at multiple targets, the jamming task is usually allocated according to the threat level of the target in practice. To simplify the scene, the threat level of the target is judged by the distance between the station and the target. The closer the distance, the greater the threat, and the farther the distance, the smaller the threat. That is, when aiming at multiple targets, the jamming station gives priority to jamming the target closest to it. This embodiment starts with the deployment position and jamming target of the jamming station, pre-sets the optional positions, and realizes the selection of the target by changing the jamming direction angle of each jamming station. The deployment position and jamming direction angle of each jamming station can be called its jamming scheme, and the training of the algorithm is used to optimize the output of the jamming scheme with the highest resource utilization.

[0107] This embodiment is further described below in conjunction with a specific simulation example.

[0108] In order to evaluate the performance of the AFCJD algorithm, it is compared with the DQL algorithm and the DDNN algorithm. When the DQL algorithm and the DDNN algorithm are used in anti-interference communication scenarios, they can be analogized into collaborative interference decision-making algorithms for application in this embodiment. At the same time, by comparing the decision-making effects of the AFCJD algorithm and the algorithm without the expert incentive reward mechanism, the advantage of the expert incentive reward mechanism in improving the algorithm decision-making performance is evaluated.

[0109] (1) Scene and parameter settings

[0110] According to communication reconnaissance and various intelligence, 20 targets to be interfered in the current airspace are obtained, and their airspace positions are roughly represented in coordinate form; a total of 6 frequency hopping channels are used, and the number of frequency hopping points is 30, 65, 130, 65, 30, and 130 respectively. The specific parameters are shown in Table 3. According to long-term intelligence or reconnaissance intelligence, the jammer knows that the signal transmission power of each communication target is 200W.

[0111] Table 3 Reconnaissance target information

[0112]

[0113]

[0114] Six positions are preset, with coordinates of [100,336], [40,182], [65,219], [30,565], [70,425], and [100,456]. There are 30 jammers available, and the maximum jammer power of each jammer is 50kW. The positions of the jammers and the targets to be jammed are distributed as follows: Figure 2 shown.

[0115] The parameter settings of the AFCJD algorithm are shown in Table 4. A total of 30,000 simulation rounds are set. In order to make the algorithm converge better, the learning rate is set to a step-by-step change. S is the interference success rate per 300 rounds. S When it is greater than 0.8, the training frequency of the neural network is reduced. The Actor network is trained once every 10 steps and the Critic network is trained once every 50 steps to reduce the probability of the algorithm converging to the local optimum.

[0116] Table 4 Algorithm parameter settings

[0117]

[0118] (2) Comparative analysis of interference resource utilization

[0119] If the interference scheme decided in a certain round can interfere with all target signals, the interference scheme is considered to be effective. The average scheme efficiency of every 300 rounds is used to represent the interference success rate. When the interference success rate reaches 100%, the algorithm is considered to have converged to the optimal state and the training is completed. First, the interference success rates of the three algorithms are compared. In order to improve the exploration and utilization efficiency of the algorithms, the number of available interference stations for the DQL algorithm and the DDNN algorithm can be increased to 35.

[0120] from Figure 3It can be seen that the AFCJD algorithm proposed in this paper converges the fastest, with an average success rate of 100% in about 14,000 rounds, while the DDNN algorithm and the DQL algorithm can only converge to a jamming success rate close to 100% in about 30,000 rounds. From the comparison of jamming success rates, it can be concluded that the AFCJD algorithm proposed in this embodiment converges the fastest and can provide a usable jamming solution in the least number of simulation rounds.

[0121] Take the average reward value of the three algorithms every 300 rounds for comparison, such as Figure 4 As shown, it can be seen that the reward value of the AFCJD algorithm proposed in this embodiment is relatively large from the beginning of training, and gradually increases during the continuous training process until the algorithm converges and stops training. The training trend is similar to the trend of the interference success rate. The average reward value of the other two algorithms is relatively low in the early stage of training, and the first 1000 rounds are all less than 0, indicating that the performance of the algorithm is poor in the early stage of training and it is impossible to output an effective solution; similar to the training trend of the interference success rate, as the training deepens, the average reward value of the other two algorithms continues to increase, and the decision-making ability gradually increases until the algorithm converges. From the comparison of the average reward value, it can be seen that the decision-making ability of the AFCJD algorithm in this paper improves rapidly and the decision-making efficiency is high, which is about 50% higher than that of the DDNN algorithm and the DQL algorithm.

[0122] Here, a rule-based decision algorithm is added for comparison. This algorithm does not rely on any intelligent calculation method and assigns interference actions to interference stations in sequence according to the interference action number. If the action satisfies the condition of formula (7) after calculation, the action is valid and executed; otherwise, the next action is selected in sequence until a valid action appears. When all targets can be interfered, the combination of interference actions of each station is the interference plan.

[0123] Calculate the average number of interference stations required for all effective interference schemes in every 300 rounds, and compare the number of interference stations required for schemes given by different algorithms. Figure 5 It can be seen that the interference scheme given by the rule-based decision algorithm requires about 28 interference stations to suppress all 20 targets; while the DDNN algorithm and the DQL algorithm require about 26 interference stations to suppress all 20 targets after convergence, and the AFCJD algorithm proposed in this embodiment only requires about 25 interference stations to suppress all targets after convergence. It can be seen that after using the intelligent algorithm, an interference scheme that saves interference resources can be obtained, and the AFCJD algorithm in this embodiment has a faster decision speed and a much higher decision efficiency than the other two algorithms.

[0124] As training progresses, the jamming scheme will continue to be optimized, but the average number of jamming stations cannot reflect the resource utilization of the optimal jamming scheme. Figure 6It reflects the comparison of the number of interference stations required for the optimal solutions decided by the four algorithms. Among them, the AFCJD algorithm only needs 24 interference stations at least to suppress all targets, which can improve resource utilization by 8% compared with the DDNN algorithm and the DQL algorithm. Compared with the rule-based decision-making algorithm, the AFCJD algorithm can improve resource utilization by 15%. Since the rule-based algorithm has no intelligent computing link, the advantage of the AFCJD algorithm is more obvious, which also shows that the collaborative interference decision-making method based on the intelligent algorithm can achieve decision-making effects that general algorithms cannot achieve.

[0125] In summary, the AFCJD algorithm proposed in this paper converges to the optimal interference plan faster than the DDNN algorithm and the DQL algorithm, and the decision-making efficiency is improved by more than 50%; and the resource utilization rate of the optimal plan is higher, which can save 8% of interference resources, so the AFCJD algorithm is better for collaborative interference decision-making.

[0126] In addition, the AFCJD algorithm proposed in this embodiment is an on-policy algorithm that can directly use the output action of the decision network and the feedback reward of the environment to train the network; the DDNN algorithm and the DQL algorithm belong to the off-policy algorithm of the DQN type, which requires the state, action and other parameters of each decision to be stored as samples in the experience pool, and then sampled from the experience pool to train the decision network. The sampling efficiency of the off-policy algorithm directly determines the effectiveness and training efficiency of the algorithm. Through the above comparison, it can also be inferred that the on-policy algorithm such as AFCJD has a higher decision-making efficiency than the off-policy algorithm in the context of interference decision-making.

[0127] (3) Analysis of the impact of embedded expert incentive and reward mechanism on decision-making results

[0128] The embedded expert incentive reward mechanism is essentially a reward project. This internal incentive can break through the training boundary of the algorithm itself, give the agent more space to explore environmental information, and improve the decision-making efficiency of the algorithm. This embodiment compares the decision-making effect of the AFCJD algorithm with the algorithm without the expert incentive reward mechanism to illustrate the advantage of the expert incentive reward mechanism in enhancing the decision-making performance of the algorithm. When there is no expert incentive reward mechanism algorithm, when all frequency hopping signals are interfered, the reward value r is 80; when there is no target in the interference beam, r is -15; otherwise r is 0.

[0129] from Figure 7 and Figure 8It can be seen that the training trends of the two algorithms are the same in the first 6000 rounds. Both the average jamming success rate and the average number of jamming stations are constantly converging and the effects are comparable. After 6000 rounds, the AFCJD algorithm continues to converge until the average jamming success rate reaches 100%. The convergence speed of the algorithm without the expert incentive reward mechanism decreases after 6000 rounds. The synthesis success rate reaches 90% in 18000 rounds and fluctuates in a large range, with no trend of continued convergence. Fig. 9 The figure shows the change in reward value of the algorithm without expert incentive reward mechanism. The training trend of the algorithm can be seen more clearly. After 18,000 rounds, the algorithm can no longer decide on a result with a higher reward value due to its relatively weak exploration ability, and a short period of overfitting occurs.

[0130] In summary, compared with the algorithm without expert incentive reward mechanism, the AFCJD algorithm has stronger exploration ability, can output better decision results, and has faster and more stable training convergence. At the same time, it can be concluded that the embedded expert incentive reward mechanism can improve the algorithm's exploration ability, improve the algorithm's decision-making ability and improve the algorithm's decision-making efficiency.

[0131] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine different embodiments or examples described in this specification.

[0132] Those skilled in the art will readily appreciate other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses or adaptations of the present invention, which follow the general principles of the present invention and include common knowledge or customary techniques in the art that are not disclosed by the present invention. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present invention are indicated by the appended claims.

Claims

1. A cooperative communication interference decision method, characterized in that: The method includes: A collaborative decision-making model is established to provide a decision-making environment; wherein the basic elements required to define the collaborative decision-making model are: state space and action space; wherein the state space represents the number of frequency points of a target frequency hopping signal that are not interfered with as h, and the state space S is defined as [h1, h2, ..., h n ], which means the number of frequencies of all target frequency hopping signals that are not interfered with; the action space defines that the interference action output by the decision network is A, which means the deployment position of the interference station and the interference action code corresponding to the interference direction angle. In order to reduce the decision dimension of the algorithm, an angle can be selected as the interference direction angle every a° in the range of 0° to 180°; Based on deep reinforcement learning, an interference decision algorithm integrating advantage function is proposed under the Actor-Critic algorithm framework, and the optimal interference plan is output according to the interference decision algorithm; Among them, based on deep reinforcement learning, an interference decision algorithm integrating advantage function is proposed under the Actor-Critic algorithm framework, and the optimal interference solution is output according to the interference decision algorithm, including: Through the strategy execution network perception environment state, obtain the frequency point information S of each target signal to be interfered at time t t , the strategy is used to execute the fitting operation of the network to output the interference action A of each interference station at time t t ; The value evaluation network obtains the frequency point information S of each target signal to be interfered at time t and time t+1. t and S t+1 , and output the interference action A at time t respectively t The state value V(S t ) and the interference action A at time t+1 t+1 The state value V(S t+1 ), to estimate the pros and cons of the strategy of the current strategy execution network; The reward function is set based on the global concept, and the interference action A is executed for each interference station by introducing expert incentives in the reward function. t The state change caused by the expert incentive is evaluated, that is, the reward value r after the expert incentive is introduced is calculated and output; wherein the expert incentive is introduced into the reward function, including: In the basic reward value r base Add an additional expert incentive value r exp , the reward value r after the introduction of the expert incentive is obtained, and the expert incentive value r exp Provide expert guidance for subsequent decisions and internal incentives for current decisions; Among them: Expert incentive value r exp Defined as: r exp =N cha ×(N jam+ 1) (1) The reward value r after introducing expert incentives is defined as: r＝r base +r exp (2) Where: N cha Indicates the number of targets that have been interfered, N jam Indicates the number of targets successfully interfered by the current interference station; The advantage function is introduced to compare the advantages and disadvantages of action value and state value; as mentioned above, the formula of the advantage function is: A(S t ,A t )=Q(S t ,A t )=r+V(S t+1 )-V(S t ) (3) Among them, Q(S t , A t ) represents the action value, and the advantage function A(S t , A t ) indicates that the interference action A is executed at time t t After that, the action value Q(S t , A t ) compared to the state value V(S t )’s degree of advantage; The policy execution network parameters and the value evaluation network parameters are respectively trained and optimized by the training optimization module; wherein the training optimization module includes the loss function of the value evaluation network and the loss function of the policy execution network, and the formulas of the loss function of the value evaluation network and the loss function of the policy execution network are respectively; L(θ V )=A(S t ,A t ;θ V ) 2 =[r+γV(S t+1 ;θ V )-V(S t ;θ V )] 2 (4) R(θ)=A(S t ,A t ;θ V )logp θ (A t |S t )=[r+γV(S t+1 ;θ V )-V(S t ;θ V )]log p θ (A t |S t ) (5) Among them, L(θ V ) represents the loss function of the value evaluation network, R(θ) represents the loss function of the policy execution network, θ represents the policy execution network parameters, and θ v represents the value evaluation network parameter, γ represents the attenuation factor, and p θ represents the policy probability of the policy execution network; According to the output of the value evaluation network A(S t , A t θ V )Optimize the policy execution network parameters so that the policy execution network can decide better interference actions.

2. The cooperative communication interference decision method according to claim 1, characterized in that: Based on deep reinforcement learning, an interference decision algorithm integrating advantage function is proposed under the Actor-Critic algorithm framework. After the step of outputting the optimal interference solution according to the interference decision algorithm, the method further includes: calculating the interference effect according to the interference-signal ratio, and determining whether the interference is effective according to the size of the interference-signal ratio and the target suppression coefficient. The formula of the interference-signal ratio is: Among them, P J is the transmission power of the jammer, P S is the transmission power of the signal transmitter; H J H is the product of the gain of the jammer transmitting antenna and the signal receiving antenna. S L is the product of the signal transmitter antenna gain and the receiving antenna gain; J and L S are the spatial losses of interference signal and communication signal transmission, respectively. It indicates that the interference bandwidth of the interference station can be aligned with the part of the target signal frequency to be interfered. Indicates the power of effective interference; Indicates whether the interference frequency band is aligned with the target signal frequency to be interfered with in the frequency domain. When the interference spectrum with frequency f is aligned with the frequency f S The frequency hopping point, then the indication value is 1, otherwise it is 0.

Images