A multi-objective fusion reward function design method for interference resource allocation

Abstract

A multi-objective fusion reward function design method for interference resource allocation, involving the interdisciplinary fields of deep reinforcement learning and communication adversarial technology, specifically includes the following steps: S1, constructing a composite structure total reward expression; S2, designing six weighted reward components, each corresponding to one of the six key tactical objectives of interference resource allocation; S3, introducing tactical constraint penalty terms, which are directly added to the total reward in negative form, and are used to constrain the agent's behavior that violates actual combat rules; S4, designing a forward-looking guidance reward term; S5, establishing a hierarchical reward assignment system that matches tactical value; S6, calculating the total reward through a step-by-step reward calculation process. The reward function architecture of this invention supports flexible addition and subtraction of reward components and adjustment of weights, and can quickly customize and adapt reward functions according to different combat scenarios to meet diverse mission requirements.

Description

Technical Field

[0001] This invention relates to the field of deep reinforcement learning and communication adversarial technology, and in particular to a multi-objective fusion reward function design method for interference resource allocation. Background Technology

[0002] As is well known, in the training of interference resource allocation models based on deep reinforcement learning, the design of the reward function directly determines the learning direction of the agent and the quality of the final policy. The reward function is the most difficult part to design among the three elements of reinforcement learning (state, action, reward). It plays a key role in transforming abstract tactical objectives into quantifiable numerical signals. A well-designed reward function can guide the agent to quickly learn a policy that meets expectations, while an improperly designed reward function may lead to policy deviation, difficulty in convergence, or even training failure.

[0003] Existing reward function design methods have the following prominent problems in tasks that interfere with resource allocation: First, the objectives are too singular. Existing methods typically only consider the single metric of suppression success rate, neglecting multi-dimensional tactical requirements such as resource consumption efficiency, priority assurance, and allocation rationality. For example, designing rewards solely based on "successful suppression is rewarded, failure is punished" can lead agents to allocate all jammers to all links at full power, resulting in severe resource waste and electromagnetic exposure risks. In actual combat, jamming resources are scarce. It is necessary to conserve resources as much as possible while ensuring suppression effectiveness, and to prioritize high-priority links. This is a typical multi-objective optimization problem, which a single-objective reward cannot handle.

[0004] Second, there is a lack of multi-target conflict handling; there are natural conflicts between various tactical objectives in the interference resource allocation task: improving the suppression effect may require increasing power consumption and reducing resource efficiency; prioritizing high-priority links may sacrifice the suppression opportunities of low-priority links; pursuing critical power control may lead to suppression failure due to battlefield fluctuations; existing methods lack a systematic multi-target fusion framework, which cannot effectively balance these conflicts, resulting in the trained strategy performing well in some aspects but poor overall performance.

[0005] Third, the problem of sparse rewards is prominent. The effect of interfering with resource allocation often requires multiple decision steps to be realized. If the reward is based solely on whether the final suppression is successful, the reward signal will be sparse. The agent will have difficulty obtaining effective gradient information in the early stages of training, resulting in slow convergence or even failure to converge. It is necessary to design dense immediate reward components to provide feedback on the intermediate effects of each decision step.

[0006] Fourth, the expression of tactical constraints is insufficient. There are many tactical constraints in the actual allocation of jamming resources, such as fixed jammers should not frequently switch targets, mobile jammers need time to switch targets, and equipment needs to cool down and recover after failure. The existing reward function lacks a quantitative expression of these tactical constraints, which means that the trained strategy may perform well in terms of numerical indicators, but its effectiveness is greatly reduced in actual execution due to the violation of constraints.

[0007] Fifth, priority differences are not fully reflected; the tactical value of different communication links varies significantly, with high-priority links (such as command and control links) having a much higher suppression requirement than low-priority links. Existing methods either treat all links equally or differentiate them only through simple weighting coefficients, failing to establish a hierarchical reward mechanism that matches tactical value.

[0008] Sixth, there is a lack of forward-looking guidance. Existing reward functions usually only consider the immediate effect of the current decision step and cannot guide the agent to make decisions that are beneficial to long-term gains. For example, continuously suppressing a certain link can trigger its downgrade, thereby reducing the difficulty of future suppression. This kind of delayed gain cannot be reflected in the immediate reward. It is necessary to design reward components with forward-looking guidance capabilities. Summary of the Invention

[0009] To overcome the shortcomings of the prior art, this invention discloses a multi-objective fusion reward function design method for interference resource allocation.

[0010] To achieve the aforementioned objective, the present invention employs the following technical solution: A multi-objective fusion reward function design method for interference resource allocation includes the following steps: S1. Construct the total reward expression for the composite structure. ; in, For the first Each weighted reward component For the corresponding weight coefficients, satisfying ; For the first Each tactical constraint penalty item is directly accumulated as a negative value and does not participate in the weight allocation; The quantity of the weighted reward component, The number of penalty items; S2. Design six weighted reward components, each corresponding to one of the six key tactical objectives that interfere with resource allocation. (1) Overall suppression reward for high-priority links : Let the set of high-priority links be The quantity is Define the success rate of suppressing high-priority links. ,in A segmented reward system is used to determine the number of high-priority links that are successfully suppressed. ; in This represents the maximum reward value when all high-priority links are suppressed. This is the penalty value when most high-priority links are not suppressed. (2) Reward for successful suppression : With link interference ratio As a condition for successful single-link suppression, among which... This is the threshold value for the suppression factor of the link at the current rate and in the spread spectrum state. ; in, This is an indicator function that takes the value 1 if the condition is true, and 0 otherwise. Indicates link Normal operation; To be based on link priority The set reward value; (3) Supplementary rewards for medium-priority links : Independent of Additional rewards are provided for medium-priority links to prevent resources from being entirely allocated to high-priority links. ; in For medium priority link set, Additional reward value for each successfully suppressed medium-priority link; (4) Resource efficiency reward : The total resource consumption is obtained by weighting the combined interference power and bandwidth consumption: ; in, The average power ratio of the allocated jammers. The average bandwidth ratio of the allocated jammers; and Let be the weighting coefficient, satisfying Since power resources have a more critical impact on the interference effect, ; Resource efficiency rewards are designed using a segmented approach based on critical points: ; in The critical consumption threshold is set to 0.8. This is the maximum reward value with zero consumption, and its value is 100. This is the penalty coefficient for exceeding the limit; (5) Allocate reasonable rewards : Assume the number of jammers allocated is The number of allocated links (after deduplication) is Define a one-to-one ratio ; The reward function is designed as a step function: ; Typical values , , , ; (6) Optimize the reward for suppressing the margin : For each successfully suppressed link Define the dry signal ratio margin Single-link margin reward function Defined as a piecewise function: ; The meanings and typical values ​​of each parameter are as follows: Optimal margin, take dB; Maximum tolerance margin, take dB; Maximum reward value, taken as... ; : Over-limit penalty coefficient, taken as ; The total suppression margin optimization reward is for all successfully suppressed links. sum: ; S3. Introduce tactical constraint penalties, which are added directly to the total reward as negative values. These penalties are used to constrain actions by agents that violate actual combat rules, including penalties for switching fixed jammers. Motor vehicle safety penalties and Highest priority link exemption and penalty ; S4. Design-oriented guidance reward items: Forward-looking guidance rewards are used to guide agents to make decisions that benefit long-term gains, compensating for the short-sightedness of immediate rewards. These include link degradation rewards. Continuous suppression trend reward ; S5. Establish a tiered reward system that matches tactical value. Based on the tactical priority of interference resource allocation, the weights of each component are set as follows: ; S6. Calculate the total reward using the step-by-step reward calculation process: The multi-objective fusion reward function design method for interference resource allocation, as described in step S2... Use a step-by-step assignment method: .

[0011] The multi-objective fusion reward function design method for interference resource allocation, in step S3, fixes the jammer switching penalty. The penalty is imposed when: the target assigned in the previous decision step is still functioning normally in the current step, but the jammer switches targets, and this switch is not caused by its own malfunction. ; in: : A collection of fixed jammers; jammer At any moment The target link number to be allocated; jammer At any moment Operating status (1 for normal, 0 for fault); The current working status of the target link assigned in the previous time step; : Switch penalty value, retrieve value ; The aforementioned motor safety penalty and For mobile jamming aircraft, the jamming distance and deviation angle should be kept within a safe and reasonable range to avoid excessive proximity or deviation. The distance safety penalty is defined as follows: ; Angle safety penalty is defined as: ; Typical parameter settings are: Safe distance range: km, km; Angle safety range: , ; Penalty coefficient: , , , ; The penalty coefficient is set to a small value to avoid sacrificing the suppression effect in pursuit of excessive mobility and safety, and only mild penalties are imposed on behaviors that clearly exceed the safety range; Highest priority link exemption penalty When the highest priority link is not completely suppressed, differentiated penalties are imposed based on the amount of resources invested by the agent, reflecting the principle of "best effort, no liability": ; in: : Success rate of suppressing the highest priority link; The proportion of power that an agent allocates to the highest priority links is defined as the ratio of the sum of the interference power actually allocated to these links to the maximum available power. Base penalty value, range of values .

[0012] The multi-objective fusion reward function design method for interference resource allocation, in step S4, includes the link degradation reward... When an enemy link is downgraded in priority due to continuous suppression, a fixed reward is given for each downgraded link. ; in For link At any moment priority, This indicates that the priority has been downgraded. The value is the downgrade reward value. ; The aforementioned continuous suppression trend reward A reward will be given when the interference-to-signal ratio of the link shows a continuous improvement trend, even if the suppression threshold has not yet been reached. ; in The trend reward value has a set value. .

[0013] The specific calculation process of step S6 in the multi-objective fusion reward function design method for interference resource allocation is as follows: Step 1: Obtain the current battlefield status, including the working status, priority, rate, spread spectrum status, and transmit power of each link; the target allocation, actual transmit power, and actual bandwidth used by each jammer; and the interference-to-signal ratio of each link. Step 2: Calculate each weighted reward component: Based on the comparison between the interference-to-signal ratio and the suppression coefficient threshold, determine whether the suppression of each link is successful; calculate the suppression ratio of high-priority links, and obtain... Calculate and sum the rewards for successful suppression of each link to obtain the result. and ; Calculate the average power ratio and bandwidth ratio of the allocated jammers to obtain ; Calculate the one-to-one allocation ratio to obtain ; Calculate the interference-to-signal ratio margin of the successfully suppressed link, and obtain ; Step 3: Calculate tactical constraint penalties: Check whether the fixed jammer has performed unnecessary target switching, and calculate... Check if the location of the mobile jamming device exceeds the safe range, and calculate... and Check the suppression status of the highest priority link and calculate... ; Step 4: Calculate the forward guidance reward: Check if the link priority has been downgraded, and calculate... Examine the continuous trend of the dry-to-sound ratio and calculate... ; Step 5: Calculate the total weighted reward by summing the weighted amounts, then add the various penalties and forward rewards to obtain the final reward value. .

[0014] Due to the adoption of the above technical solution, the present invention has the following beneficial effects: The multi-objective fusion reward function design method for jamming resource allocation described in this invention, compared with existing technologies, achieves coordinated optimization of multiple objectives such as suppression success rate, priority guarantee, resource efficiency, allocation rationality, and suppression margin control through the organic combination of six types of weighted reward components and three types of tactical constraint penalty terms. The designed six weighted reward components enable the agent to obtain effective gradient feedback at each decision step, allowing it to quickly establish the correct decision direction in the early stages of training, and shortening the convergence time compared to sparse reward methods. The introduction of tactical constraint penalty terms ensures that the trained strategy naturally adheres to practical constraints—the stability of the fixed jammer target, the safety of the mobile jammer position, and the... The highest priority link operates on a best-effort basis, and the strategy can be directly applied to actual combat without additional post-processing. The forward-looking reward item teaches the agent to focus on long-term gains, manifested in actively and continuously suppressing high-value targets to trigger their downgrade. The reward function architecture of this invention supports flexible addition or subtraction of reward components and adjustment of weights, and can quickly customize and adapt reward functions according to different combat scenarios (such as focusing on resource conservation or focusing on comprehensive suppression), adapting to diverse task requirements. The multi-objective fusion reward function design concept of this invention is not only applicable to tasks involving interference with resource allocation, but can also be extended to other reinforcement learning decision-making tasks with multi-objective, multi-constraint, and delayed gain characteristics, and has high promotional value. Detailed Implementation

[0015] The present invention will be explained in detail through the following embodiments. The purpose of disclosing the present invention is to protect all technical improvements within the scope of the present invention.

[0016] The multi-objective fusion reward function design method for interference resource allocation described in this invention specifically includes the following steps: S1. Construct the total reward expression for the composite structure. ; in, For the first Each weighted reward component For the corresponding weight coefficients, satisfying ; For the first Each tactical constraint penalty item is directly accumulated as a negative value and does not participate in the weight allocation; The quantity of the weighted reward component, The number of penalty items; The weighted reward component is used to guide the agent to optimize core tactical objectives, and the weight coefficients reflect the relative importance of each objective; the penalty term is used to constrain the agent's illegal behavior, and direct accumulation ensures that the strength of the constraint signal is not diluted by the weights. S2. Design six weighted reward components, each corresponding to one of the six key tactical objectives that interfere with resource allocation. (1) Overall suppression reward for high-priority links This component is used to enhance the protection of the collective suppression effect of high-priority links, guiding the agent to focus on the overall suppression status of high-priority links, rather than just focusing on a single link. Let the set of high-priority links be The quantity is Define the success rate of suppressing high-priority links. ,in A segmented reward system is used to determine the number of high-priority links that are successfully suppressed. ; in This is the maximum reward value when all high-priority links are suppressed, and it is set to 400. This is the penalty value when most high-priority links are not suppressed, and its value is 300. The segmentation logic of the design is as follows: when all high-priority links are suppressed, the highest reward is given to encourage the agent to fully guarantee the critical task; when some are suppressed, a positive reward is given proportionally to maintain learning motivation; when more than half of the high-priority links are not suppressed, a severe penalty is given to force the agent to prioritize the allocation of resources to high-priority links.

[0017] (2) Reward for successful suppression This component is the core of the reward function, directly reflecting the primary objective of interference decision-making—the effective suppression of communication links; for each normally functioning link... To determine whether it has been successfully suppressed. With link interference ratio As a condition for successful single-link suppression, among which... This is the threshold value for the suppression factor of the link at the current rate and in the spread spectrum state. ; in, This is an indicator function that takes the value 1 if the condition is true, and 0 otherwise. Indicates link Normal operation; To be based on link priority The set reward value; the aforementioned Use a step-by-step assignment method: ; Typical values ​​are as follows , , ; The theoretical basis of this design is as follows: First, only links that are working normally are rewarded, and faulty links do not need to be suppressed, thus avoiding rewarding ineffective decisions; Second, a tiered reward value is adopted, with the reward for successfully suppressing a high-priority link being much higher than that for a low-priority link, clearly reflecting the priority difference; Third, each link is calculated and summed independently, which encourages the suppression of as many links as possible while avoiding excessive punishment for the failure to suppress a single link, thus maintaining the smoothness of the reward signal.

[0018] (3) Supplementary rewards for medium-priority links

[0019] To prevent the agent from concentrating all resources on high-priority links and completely ignoring medium-priority links, a separate supplementary reward is set for medium-priority links: ; in For medium priority link set, An additional reward of 100 is given for each successfully suppressed medium-priority link, independent of [other factors]. This further strengthens the tactical value of medium-priority links and achieves a more balanced allocation of resources. (4) Resource efficiency reward This component guides the agent to minimize the consumption of interference resources (power, bandwidth) while ensuring the suppression effect; The total resource consumption is obtained by weighting the combined interference power and bandwidth consumption: ; in, The average power ratio of the assigned jammers (the ratio of actual transmit power to maximum power). The average bandwidth ratio of the allocated jammers (the ratio of actual bandwidth used to maximum bandwidth). and Let be the weighting coefficient, satisfying Since power resources have a more critical impact on the interference effect, typically Values , ; Resource efficiency rewards are designed using a segmented approach based on critical points: ; in The critical consumption threshold is set to 0.8. This is the maximum reward value with zero consumption, and its value is 100. This is the penalty coefficient for exceeding the limit, with a value of 100; The theoretical basis of this design is as follows: First, a continuous function is used instead of a discrete threshold, so that even small changes in resource utilization can generate gradient signals, which facilitates the model to learn fine resource adjustment; Second, a critical point is set, and a linear positive reward is given when the value is below the critical point to encourage resource conservation, and a linear penalty is given when the value is above the critical point to warn against excessive consumption; Third, the power weight is greater than the bandwidth weight, which is in line with the physical law - insufficient power directly leads to suppression failure, while insufficient bandwidth can be compensated for by adjusting the interference method. (5) Allocate reasonable rewards This component encourages agents to achieve a one-to-one allocation of jammers and links, avoiding the waste of resources caused by multiple jammers suppressing the same link at the same time; Assume the number of jammers allocated is The number of allocated links (after deduplication) is Define a one-to-one ratio ; The reward function is designed as a step function: ; Typical values ​​are as follows , , , ; The design encourages agents to achieve one-to-one allocation as much as possible, because many-to-one usually means wasted resources; at the same time, considering that some high-interference-resistant links may indeed require multi-machine collaborative suppression, the reward function does not completely prohibit many-to-one, but guides agents to adopt the many-to-one strategy only when necessary through tiered rewards. (6) Optimize the reward for suppressing the margin This component encourages agents to keep the interference-to-signal ratio margin within a reasonable range to avoid power waste (excessive margin) or insufficient suppression (insufficient margin). For each successfully suppressed link Define the dry signal ratio margin Unit: dB; Ideally, the margin should be within a reasonable range, such as 0~3dB, to ensure stable suppression without causing excessive power waste; Link margin reward function Defined as a piecewise function: ; The meanings and typical values ​​of each parameter are as follows: Optimal margin, take dB; Maximum tolerance margin, take dB; Maximum reward value, taken as... ; : Over-limit penalty coefficient, taken as ; The total suppression margin optimization reward is for all successfully suppressed links. sum: ; This design enables the agent to learn to precisely control interference power: when the margin is within a reasonable range, a reward is given, with the highest reward at the optimal margin; when the margin is too large, a linear penalty is given to warn of wasted power; links that fail to be suppressed are not included in the calculation, and their penalty is already reflected in the lack of reward for successful suppression. S3. Introduce tactical constraint penalties, which are added directly to the total reward as negative values. These penalties are used to constrain actions by agents that violate actual combat rules, including penalties for switching fixed jammers. Motor vehicle safety penalties and Highest priority link exemption and penalty ; Fixed jammer switching penalty The penalty is imposed when: the target assigned in the previous decision step is still functioning normally in the current step, but the jammer switches targets, and this switch is not caused by its own malfunction. ; in: : A collection of fixed jammers; jammer At any moment The target link number to be allocated; jammer At any moment Operating status (1 for normal, 0 for fault); The current working status of the target link assigned in the previous time step; : Switch penalty value, retrieve value ; Fixed jammers are penalized for switching targets under non-fault conditions to meet the tactical requirements of "fixed deployment and continuous suppression". This design ensures the target stability of fixed jammers, avoids suppression gaps caused by frequent switching, and allows switching due to objective reasons such as link failure. The aforementioned motor safety penalty and For mobile jamming aircraft, the jamming distance and deviation angle should be kept within a safe and reasonable range to avoid getting too close or deviating too far. Distance safety penalty is defined as: ; Angle safety penalty is defined as: ; Typical parameter settings are: Safe distance range: km, km; Angle safety range: , ; Penalty coefficient: , , , ; The penalty coefficient is set to a small value to avoid sacrificing the suppression effect in pursuit of excessive mobility and safety, and only mild penalties are imposed on behaviors that clearly exceed the safety range; Highest priority link exemption penalty When the highest priority link is not completely suppressed, differentiated penalties are imposed based on the amount of resources invested by the agent, reflecting the principle of "best effort, no liability": ; in: : Success rate of suppressing the highest priority link; The proportion of power that an agent allocates to the highest priority links is defined as the ratio of the sum of the interference power actually allocated to these links to the maximum available power. Base penalty value, range of values ; The tactical logic of this design is: if the agent has already invested most of its resources ( If the highest priority link is not completely suppressed, the penalty is reduced or even waived; if resources are insufficient, a heavier penalty is imposed. This avoids over-punishing the agent for occasional failure to suppress the highest priority link, while also preventing it from giving up suppression altogether due to insufficient resources. S4. Design of forward-looking guidance incentive items Forward-looking guidance rewards are used to guide agents to make decisions that benefit long-term gains, compensating for the short-sightedness of immediate rewards. These include link degradation rewards. Continuous suppression trend reward ; The aforementioned link degradation reward When an enemy link is downgraded in priority due to continuous suppression, a fixed reward is given for each downgraded link: ; in For link At any moment priority, This indicates that the priority has been downgraded; The value is the downgrade reward value. The tactical logic behind this reward is that link degradation means reduced tactical importance and easier suppression in the future, which is beneficial to the interfering party. Therefore, a positive reward is given to encourage the agent to continue suppressing the link until it is degraded. The aforementioned continuous suppression trend reward A reward will be given when the interference-to-signal ratio of the link shows a continuous improvement trend, even if the suppression threshold has not yet been reached: ; in The trend reward value has a set value. This reward guides the agent to focus on the improving trend of the interference effect, rather than just whether the threshold is reached, which helps to provide more intensive feedback signals in the early stages of training. S5. Establish a tiered reward system that matches tactical value. Based on the tactical priority of interference resource allocation, the weights of each component are set as follows: ; The theoretical basis for setting the weights is as follows: The high-priority overall suppression reward weight is the highest (0.35), highlighting the absolute core position of the high-priority link; The reward for successful suppression has the second highest weight (0.30), covering the overall suppression effect of all links, taking into account both the overall situation and key points; The reward weight for medium-priority links is moderate (0.10) to ensure that they are not completely ignored; The resource efficiency reward weight is moderate (0.15), which achieves efficient resource utilization while ensuring the suppression effect; The allocation rationality and suppression margin optimization have relatively low weights (0.05 each), belonging to the fine-tuning target.

[0020] The penalty items and forward guidance reward items are directly accumulated and do not participate in the weighting distribution. Their numerical design has taken into account the balance with the magnitude of the weighted reward. S6. Calculate the total reward using the step-by-step reward calculation process. Step 1: Obtain the current battlefield status, including the working status, priority, rate, spread spectrum status, and transmit power of each link; the target allocation, actual transmit power, and actual bandwidth used by each jammer; and the interference-to-signal ratio of each link. Step 2: Calculate each weighted reward component: Based on the comparison between the interference-to-signal ratio and the suppression coefficient threshold, determine whether the suppression of each link is successful; calculate the suppression ratio of high-priority links, and obtain... Calculate and sum the rewards for successful suppression of each link to obtain the result. and ; Calculate the average power ratio and bandwidth ratio of the allocated jammers to obtain ; Calculate the one-to-one allocation ratio to obtain ; Calculate the interference-to-signal ratio margin of the successfully suppressed link, and obtain ; Step 3: Calculate tactical constraint penalties: Check whether the fixed jammer has performed unnecessary target switching, and calculate... Check if the location of the mobile jamming device exceeds the safe range, and calculate... and Check the suppression status of the highest priority link and calculate... ; Step 4: Calculate the forward guidance reward: Check if the link priority has been downgraded, and calculate... Examine the continuous trend of the dry-to-sound ratio and calculate... ; Step 5: Calculate the total weighted reward by summing the weighted amounts, then add the various penalties and forward rewards to obtain the final reward value. .

[0021] Example Taking a low-dimensional communication countermeasure scenario with three jammers (J1 and J2 fixed, J3 mobile) and three microwave links (L1 high priority, L2 medium priority, L3 high priority) as an example, the specific implementation process of the present invention is explained. Scenario assumption: 1. Jammer: J1: Fixed, maximum power dBW; J2: Fixed, maximum power dBW; J3: Maneuverability, Maximum Power dBW.

[0022] 2. Link: L1: High priority, non-spread spectrum, high rate kbps, suppression coefficient dB; L2: Medium priority, spread spectrum, rate kbps, suppression coefficient dB; L3: High priority, non-spread spectrum, high rate kbps, suppression coefficient dB.

[0023] 3. Reward function parameters: , , , , , , , , , , , , , , , dB dB , , km, km, , , , , ,

[0024] , , , , 4. Weighting coefficients: , , , , , .

[0025] Calculation Example 1: Normal Suppression Scenario 1. State and action of a certain decision step: J1 is allocated to L1, power dBW (power ratio) ), bandwidth matching; J2 allocates L2, power dBW (power ratio) ),bandwidth MHz (bandwidth ratio) ); J3 is allocated to L3, power dBW (power ratio) ), bandwidth matching.

[0026] 2. Suppression effect: L1: dB dB, suppression successful. dB; L2: dB dB, suppression failed; L3: dB dB, suppression successful. dB.

[0027] 3. Calculate the components: Both L1 and L3 were successful. , ; L1 successful L2 failure L3 successfully achieved ,total ; L2 failure .

[0028] Average power ratio Average bandwidth ratio , , , .

[0029] : , , , .

[0030] L1: L3: L2 failures are disregarded, total .

[0031] , , , , , .

[0032] 4. Total Rewards: ; .

[0033] Calculation Example 2: Resource Waste Penalty Scenario Assume the agent allocates all three links (J1, J2, and J3) at full power to L1 (high priority), and leaves the other links unallocated: Suppression effect: L1 suppression successful. dB dB; L2 and L3 were not suppressed.

[0034] 1. Calculate the components: : , ; L1 successful Other failures ,total ; L2 failure ; The power ratio is 1. Average bandwidth , , ; : , , , falling interval, ; L1 margin , ; The highest priority L3 was not suppressed, and the power allocation ratio was [not specified]. , ; 2. Total Rewards:

[0035] ; Compared to normal scenarios The reward for this scenario is significantly reduced, with the main deductions coming from excessive penalties for suppressing margins and incomplete priority suppression, effectively guiding agents to avoid resource waste and imbalance in allocation.

[0036] Calculation Example 3: Penalty Scenario for Violation Switching Assuming that in the previous decision step J1 assigned L1 and L1 was functioning correctly, in this step J1 actively switches to L2 (L2 is also functioning correctly): ; If the total of other reward components is The total reward then drops to This penalty guides the agent to maintain the target stability of the fixed jammer.

[0037] The parts of this invention not described in detail are prior art.

[0038] The embodiments selected herein for the purpose of disclosing the inventive objectives are currently considered suitable; however, it should be understood that the invention is intended to include all variations and modifications of the embodiments that fall within the scope of this concept and invention.

Claims

1. A multi-objective fusion reward function design method for interference resource allocation; characterized in that: Specifically, the following steps are included: S1. Construct the total reward expression for the composite structure. ； in, For the first Each weighted reward component For the corresponding weight coefficients, satisfying ; For the first Each tactical constraint penalty item is directly accumulated as a negative value and does not participate in the weight allocation; The quantity of the weighted reward component, The number of penalty items; S2. Design six weighted reward components, including overall suppression reward for high-priority links. Successful suppression reward Medium-priority link supplementary reward Resource efficiency rewards Reasonable distribution of rewards And suppression margin optimization rewards High-priority link overall suppression reward Segmented rewards are used; rewards are given for successful suppression. With link interference ratio As a condition for successful single-link suppression, among which... The suppression factor threshold for this link under the current rate and spread spectrum conditions; supplementary rewards for medium-priority links. Independent of A fixed additional reward is added to each successfully suppressed medium-priority link; resource efficiency reward. The design adopts a segmented approach based on critical points; the suppression margin is optimized for rewards. For all successfully suppressed links The sum for each successfully suppressed link Define the dry signal ratio margin Single-link margin reward function Defined as a piecewise function; S3. Introduce tactical constraint penalties, which are added directly to the total reward as negative values. These penalties are used to constrain actions by agents that violate actual combat rules, including penalties for switching fixed jammers. Motor vehicle safety penalties and Highest priority link exemption and penalty ; S4. Design of forward-looking guidance incentive items Forward-looking guidance rewards are used to guide agents to make decisions that benefit long-term gains, compensating for the short-sightedness of immediate rewards. These include link degradation rewards. Continuous suppression trend reward ; S5. Establish a tiered reward system that matches tactical value. Based on the tactical priority of interference resource allocation, the weights of each component are set as follows: ； S6. Calculate the total reward through the step-by-step reward calculation process.

2. The multi-objective fusion reward function design method for interference resource allocation according to claim 1, characterized in that: The high-priority link overall suppression reward mentioned in step S2 The specific steps for using segmented rewards are as follows: Let the set of high-priority links be... The quantity is Define the success rate of suppressing high-priority links. ,in A segmented reward system is used to determine the number of high-priority links that are successfully suppressed. ； in This represents the maximum reward value when all high-priority links are suppressed. This is the penalty value when most high-priority links are not suppressed.

3. The multi-objective fusion reward function design method for interference resource allocation according to claim 1, characterized in that: The successful suppression reward mentioned in step S2 The calculation formula is ; in, This is an indicator function that takes the value 1 if the condition is true, and 0 otherwise. Indicates link Normal operation; To be based on link priority The set reward value.

4. The multi-objective fusion reward function design method for interference resource allocation according to claim 1, characterized in that: The medium-priority link supplementary reward mentioned in step S2 The calculation formula is: ; in For medium priority link set, Additional reward value for each successfully suppressed medium-priority link.

5. The multi-objective fusion reward function design method for interference resource allocation according to claim 1, characterized in that: The resource efficiency reward mentioned in step S2 The design adopts a segmented approach based on critical points: ； in The critical consumption threshold is set to 0.

8. This is the maximum reward value with zero consumption, and its value is 100. This is the penalty coefficient for exceeding the limit; where, The total resource consumption is obtained by weighting the combined interference power and bandwidth consumption: ;in, The average power ratio of the allocated jammers. The average bandwidth ratio of the allocated jammers; and Let be the weighting coefficient, satisfying Since power resources have a more critical impact on the interference effect, .

6. The multi-objective fusion reward function design method for interference resource allocation according to claim 1, characterized in that: The allocation of reasonable rewards described in step S2 Designed in a stepped manner: ； Typical value , , , ; To define a one-to-one ratio: ;in, The number of jammers already allocated, The number of links assigned (after deduplication).

7. The multi-objective fusion reward function design method for interference resource allocation according to claim 1, characterized in that: The suppression margin optimization reward mentioned in step S2 The piecewise function is: ； The meanings and typical values ​​of each parameter are as follows: Optimal margin, take dB; Maximum tolerance margin, take dB; Maximum reward value, taken as... ; : Over-limit penalty coefficient, taken as ; The total suppression margin optimization reward is for all successfully suppressed links. sum: .

8. The multi-objective fusion reward function design method for interference resource allocation according to claim 1, characterized in that: In step S3, the jammer switching penalty is fixed. The penalty is imposed when: the target assigned in the previous decision step is still functioning normally in the current step, but the jammer switches targets, and this switch is not caused by its own malfunction. ； in: : A collection of fixed jammers; jammer At any moment The target link number to be allocated; jammer At any moment Operating status (1 for normal, 0 for fault); The current working status of the target link assigned in the previous time step; : Switch penalty value, retrieve value ; The aforementioned motor safety penalty and For mobile jamming aircraft, the jamming distance and deviation angle should be kept within a safe and reasonable range to avoid excessive proximity or deviation. The distance safety penalty is defined as follows: ； Angle safety penalty is defined as: ； Typical parameter settings are: Safe distance range: km, km; Angle safety range: , ; Penalty coefficient: , , , ; The penalty coefficient is set to a small value to avoid sacrificing the suppression effect in pursuit of excessive mobility and safety, and only mild penalties are imposed on behaviors that clearly exceed the safety range; Highest priority link exemption penalty When the highest priority link is not completely suppressed, differentiated penalties are imposed based on the amount of resources invested by the agent, reflecting the principle of "best effort, no liability": ； in: : Success rate of suppressing the highest priority link; The proportion of power that an agent allocates to the highest priority links is defined as the ratio of the sum of the interference power actually allocated to these links to the maximum available power. Base penalty value, range of values .

9. The multi-objective fusion reward function design method for interference resource allocation, wherein the link degradation reward in step S4... When an enemy link is downgraded in priority due to continuous suppression, a fixed reward is given for each downgraded link. ； in For link At any moment priority, This indicates that the priority has been downgraded. The value is the downgrade reward value. ; The aforementioned continuous suppression trend reward A reward will be given when the interference-to-signal ratio of the link shows a continuous improvement trend, even if the suppression threshold has not yet been reached. ； in The trend reward value, with a range of values. .

10. The multi-objective fusion reward function design method for interference resource allocation according to claim 1, characterized in that: The specific calculation process for step S6 is as follows: Step 1: Obtain the current battlefield status, including the working status, priority, rate, spread spectrum status, and transmit power of each link; the target allocation, actual transmit power, and actual bandwidth used by each jammer; and the interference-to-signal ratio of each link. Step 2: Calculate each weighted reward component: Based on the comparison between the interference-to-signal ratio and the suppression coefficient threshold, determine whether the suppression of each link is successful; Calculate the suppression ratio of high-priority links to obtain ; Calculate the successful suppression reward for each link and sum them up to obtain the result. and Calculate the average power ratio and bandwidth ratio of the allocated jammers to obtain... ; Calculate the one-to-one allocation ratio to obtain ; Calculate the interference-to-signal ratio margin of the successfully suppressed link, and obtain ; Step 3: Calculate tactical constraint penalties: Check whether the fixed jammer has performed unnecessary target switching, and calculate... Check if the location of the mobile jamming device exceeds the safe range, and calculate... and Check the suppression status of the highest priority link and calculate... ; Step 4: Calculate the forward guidance reward: Check if the link priority has been downgraded, and calculate... Examine the continuous trend of the dry-to-sound ratio and calculate... ; Step 5: Calculate the total weighted reward by summing the weighted amounts, then add the various penalties and forward rewards to obtain the final reward value. .

Images