Interference resource scheduling method, device and equipment for air-ground integrated network

Abstract

The invention relates to the technical field of communication interference, and provides an interference resource scheduling method, device and equipment for an air-ground integrated network, and the method comprises the steps: obtaining corresponding environment state parameters for a plurality of pieces of interference equipment, a plurality of pieces of interfered equipment and a plurality of communication frequency points corresponding to the interfered equipment, the multi-agent mixed near-end strategy allows a plurality of interference devices to perform information exchange and experience sharing, so that the multi-agent mixed near-end strategy has the characteristics of accelerating training and improving strategy updating, and the target interference strategy is obtained. In the interference strategy generation model, the interfered communication frequency point parameter and the interference power value can be trained at the same time, the training complexity can be reduced, the convergence speed of the model is accelerated, the efficiency of obtaining the target interference strategy is improved, and the efficiency of scheduling the interference device to interfere the interfered communication frequency point according to the target interference strategy is further improved.

Description

Technical Field

[0001] The present invention relates to the field of communication interference technology, and in particular to an interference resource scheduling method, device and equipment for an air-ground integrated network. Background Art

[0002] With the continuous evolution of electronic warfare technology, in the closed-loop electromagnetic spectrum combat process of perception, decision-making, action and evaluation, scientific decision-making is the key link in the effectiveness of electronic countermeasures. Its core lies in how to achieve efficient allocation and scheduling of resources. Specifically, it mainly involves the current optimization of interference resource parameters to achieve efficient allocation of interference resources.

[0003] Based on this, in the prior art, interference resource allocation is achieved through deep reinforcement learning (DRL) technology. Specifically, a single-agent reinforcement learning method is used, centralized control is adopted, and interference resource allocation is achieved by constructing a global state and action space, or a multi-agent reinforcement learning method is used, and interference resource allocation is achieved through independent modeling of each device and collaborative decision-making.

[0004] However, using existing technologies, since the single-agent reinforcement learning method has the problems of too high decision dimension and high communication overhead, and the multi-agent reinforcement learning method has the problem of high training complexity, when using existing technologies to implement interference resource allocation for multiple agents, there are problems of poor scalability and low training efficiency, which leads to reduced efficiency of interference resource allocation. Summary of the invention

[0005] Based on this, it is necessary to provide an interference resource scheduling method, device and equipment for an air-ground integrated network to address the above technical problems.

[0006] In a first aspect, an embodiment of the present invention provides an interference resource scheduling method for an air-ground integrated network, the method comprising:

[0007] Acquire environmental state parameters of a target area, wherein the target area includes a plurality of interfering devices, a plurality of interfered devices, and a plurality of communication frequencies corresponding to the interfered devices, and the environmental state parameters include at least: a historical execution action of the interfering device, a power state, and an interference-to-signal ratio of the interfered device, and the historical execution action includes: an interfered communication frequency parameter and an interference power value of the interfered communication frequency, and the interfered communication frequency parameter is used to indicate at least one interfered communication frequency that the interfering device interferes with among the plurality of communication frequencies;

[0008] Input the environmental state parameters into a trained interference strategy generation model to obtain a target interference strategy, wherein the interference strategy generation model is trained using a multi-agent hybrid proximal strategy, and the interference strategy generation model is composed of a strategy network and a value network, and the strategy network includes an input layer, multiple hidden layers, and an output layer, and the output layer includes a first output sublayer for outputting the interfered communication frequency point parameters and a second output sublayer for outputting the interference power value;

[0009] For the target area, according to the target interference strategy, the interference device is scheduled to interfere with the interfered communication frequency point by using the interference power value of the interfered communication frequency point.

[0010] In one embodiment, before inputting the environmental state parameters into the trained interference strategy generation model to obtain the target interference strategy, the method further includes:

[0011] Establishing a target problem function set for the interference resource scheduling problem of multiple interfering devices, multiple interfered devices, and multiple communication frequency points corresponding to the interfered devices in the target area;

[0012] For the target problem function set, a multi-agent hybrid proximal strategy is used to generate a model through an interference strategy to perform model training and solution.

[0013] In one embodiment, the target problem function set can be defined by the following expression:

[0014]

[0015] X nm (t)∈{0,1},n∈N,m∈M

[0016]

[0017] Where M represents the number of interfered communication frequencies corresponding to multiple interfered devices, N represents the number of interfering devices, K represents the number of interfered devices, and p nm (t) represents the interference power value of the nth interference device to the mth interfered communication frequency at time t, X nm (t) represents the interfered communication frequency parameter of the nth interference device to the mth interfered communication frequency at time t, G nk It represents the gain value of the nth interfering device to the kth interfered device, L nk represents the transmission loss of the nth interfering device to the kth interfered device, σ 2 Represents the noise power value, p k (t) represents the communication power value of the kth interfered device itself at time t, G k(t) represents the communication channel gain value of the kth interfered device at time t, L k Indicates the transmission loss of the interfered device during its own communication process, γ k Indicates the interference suppression coefficient set by the interference device on the interference power value of the interfered device, p max It represents the maximum interference power value emitted by the interference device, and H represents the maximum number of interference communication frequencies that a interference device can interfere at the same time.

[0018] In one embodiment, the method of using a multi-agent hybrid proximal strategy to generate a model through an interference strategy to perform model training and solving includes:

[0019] Initializing environmental state parameters of the target area;

[0020] According to the environmental state parameters, a multi-agent hybrid proximal strategy is used to train the initial interference strategy generation model based on a training loss function to obtain a trained interference strategy generation model.

[0021] In one embodiment, the method further comprises: training the initial interference strategy generation model based on the training loss function by using the multi-agent hybrid proximal strategy according to the environmental state parameters to obtain the trained interference strategy generation model;

[0022] For the initial interference strategy generation model, weighting is performed according to the value loss function, the strategy loss function, and the entropy regularization term of the value network to obtain the training loss function, wherein the strategy loss function is related to the interfered communication frequency parameters and the interference power value of the interfered communication frequency.

[0023] In one embodiment, the weighting of the value loss function, the strategy loss function, and the entropy regularization term of the value network to obtain the training loss function includes:

[0024] According to the formula Obtaining the training loss function;

[0025] Among them, L 1 (θ) represents the strategy loss function, represents the value loss function of the value network, X[π θ (S t )] represents the entropy regularization term, S t represents the environmental space parameters, θ and represents the network parameters of the interference strategy generation model, and a and b represent weighting factors.

[0026] In one embodiment, the initial interference strategy generation model is trained based on the training loss function using a multi-agent hybrid proximal strategy according to the environmental state parameters to obtain a trained interference strategy generation model, including:

[0027] The environmental state parameters are input into the initial interference strategy generation model, and an intermediate execution action is obtained through the strategy network included in the initial interference strategy generation model. The interference-signal ratios corresponding to multiple interference devices are determined according to the intermediate execution actions. The reward value of the intermediate execution action is calculated through the reward function set by the value network. The intermediate execution action, the reward value, and the interference-signal ratio are returned and input into the initial interference strategy generation model. Training is performed based on the training loss function until the initial interference strategy generation model converges to obtain a trained interference strategy generation model.

[0028] In one embodiment, the method further comprises:

[0029] The power status of the interfering device is periodically updated.

[0030] In a second aspect, an embodiment of the present invention provides an interference resource scheduling device for an air-ground integrated network, including:

[0031] An environmental state parameter acquisition module is used to acquire environmental state parameters of a target area, wherein the target area includes multiple interfering devices, multiple interfered devices, and multiple communication frequency points corresponding to the interfered devices, and the environmental state parameters at least include: historical execution actions of the interfering device, power status, and interference-to-signal ratio of the interfered device, and the historical execution actions include: interfered communication frequency point parameters and interference power values ​​of the interfered communication frequency points, and the interfered communication frequency point parameters are used to indicate at least one interfered communication frequency point that the interfering device interferes with among the multiple communication frequency points;

[0032] A target interference strategy acquisition module, used for inputting the environmental state parameters into a trained interference strategy generation model to obtain a target interference strategy, wherein the interference strategy generation model is obtained by training using a multi-agent hybrid proximal strategy, and the interference strategy generation model is composed of a strategy network and a value network, and the strategy network includes an input layer, multiple hidden layers, and an output layer, and the output layer includes a first output sublayer for outputting the interfered communication frequency point parameters and a second output sublayer for outputting the interference power value;

[0033] The scheduling module is used to schedule the interference device to interfere with the interfered communication frequency point by using the interference power value of the interfered communication frequency point according to the target interference strategy for the target area.

[0034] In a third aspect, an embodiment of the present invention provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the interference resource scheduling method for the air-ground integrated network described in the first aspect are implemented.

[0035] The technical solution provided by the embodiment of the present invention has the following advantages compared with the prior art:

[0036] An interference resource scheduling method, device and equipment for an air-ground integrated network provided by an embodiment of the present invention adopts this method to obtain corresponding environmental state parameters for multiple interfering devices, multiple interfered devices and multiple communication frequencies corresponding to the interfered devices contained in the target area, such as the historical execution action of the interfering device, that is, the interfered communication frequency parameter of at least one interfered communication frequency used to indicate that the interfering device interferes with multiple communication frequencies, and the interference power value of the interfered communication frequency, the power state of the interfering device at the current moment, and the interference-to-signal ratio of the interfered device. The obtained environmental state parameters are input into a The target interference strategy is obtained from the interference strategy generation model obtained by training the multi-agent hybrid proximal strategy. Since the multi-agent hybrid proximal strategy allows multiple interference devices to exchange information and share experience, it has the characteristics of accelerated training and improved strategy updating. In addition, the interference strategy generation model can simultaneously train the parameters of the interfered communication frequency point and the interference power value, which can reduce the training complexity and accelerate the convergence speed of the interference strategy generation model. Therefore, the efficiency of obtaining the target interference strategy is improved. Further, the efficiency of scheduling the interference device to interfere with the interfered communication frequency point through the interference power value according to the target interference strategy is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0038] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0039] Figure 1 A schematic diagram of a flow chart of an interference resource scheduling method for an air-ground integrated network provided by an embodiment of the present invention;

[0040] Figure 2 A schematic diagram of a scenario of an interference resource scheduling method for an air-ground integrated network provided by an embodiment of the present invention;

[0041] Figure 3 A schematic diagram of a network model of an interference strategy generation model provided by an embodiment of the present invention;

[0042] Figure 4 A schematic structural diagram of an interference resource scheduling device for an air-ground integrated network provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0043] In order to more clearly understand the above-mentioned objectives, features and advantages of the present invention, the scheme of the present invention will be further described below. It should be noted that the embodiments of the present invention and the features in the embodiments can be combined with each other without conflict.

[0044] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present invention, rather than all of the embodiments.

[0045] In the closed-loop electromagnetic spectrum combat process of perception, decision-making, action and evaluation, how to optimize the interference resource parameters to achieve efficient allocation of interference resources. In the existing technology, the single-agent reinforcement learning method is specifically used, centralized control is adopted, and interference resource allocation is achieved by constructing a global state and action space, or the multi-agent reinforcement learning method is used to achieve interference resource allocation through independent modeling of each device and collaborative decision-making.

[0046] However, when using existing technologies, the single-agent reinforcement learning method has the problems of too high decision-making dimension and high communication overhead, while the multi-agent reinforcement learning method has the problem of high training complexity. As a result, when using existing technologies to implement interference resource allocation, there are problems of poor scalability and low training efficiency.

[0047] Therefore, the present invention provides an interference resource scheduling method for an air-ground integrated network, by obtaining corresponding environmental state parameters, such as the historical execution action of the interfering device, that is, the interfered communication frequency parameter of at least one interfered communication frequency used to indicate that the interfering device interferes with multiple communication frequencies, and the interference power value of the interfered communication frequency, the current power state of the interfering device, and the interference-signal ratio of the interfered device for multiple interfering devices, and inputting the obtained environmental state parameters into an interference strategy generation model obtained by training using a multi-agent hybrid proximal strategy to obtain a target interference strategy. Since the multi-agent hybrid proximal strategy allows multiple interfering devices to exchange information and share experience, it has the characteristics of accelerating training and improving strategy updating. In addition, the interference strategy generation model can simultaneously train the interfered communication frequency parameters and the interference power value, which can reduce the training complexity and accelerate the convergence speed of the interference strategy generation model. Therefore, the efficiency of obtaining the target interference strategy is improved, and further, the efficiency of scheduling the interfering device to interfere with the interfered communication frequency through the interference power value according to the target interference strategy is improved.

[0048] In one embodiment, Figure 1 As shown, Figure 1 A flow chart of an interference resource scheduling method for an air-ground integrated network provided by an embodiment of the present invention specifically includes the following steps:

[0049] S10: Obtain environmental state parameters of the target area.

[0050] Among them, the target area includes multiple interfering devices, multiple interfered devices, and multiple communication frequency points corresponding to the interfered devices. The communication frequency point refers to the frequency point at which the interfered device can reselect a new communication frequency point for communication after the interfered device is interfered by the interfering device.

[0051] Exemplary, reference Figure 2 As shown, when interference resources are allocated for the target area 20, it is assumed that there are N interfering devices 201, then the set of the N interfering devices 201 is N = {1, 2, ..., N}, there are K interfered devices 202, then the set of the K interfered devices 202 is K = {1, 2, ..., K}, and there are M communication frequency points, then the set of the M communication frequency points is N = {1, 2, ..., M}. It should be noted that the channels of the various communication frequency points are orthogonal channels of equal bandwidth that do not interfere with each other and are independent of each other, but are not limited to this. The present invention is not specifically limited, and those skilled in the art can set it according to actual conditions.

[0052] The above-mentioned environmental status parameters include at least: historical execution actions and power status of the interfering device, wherein the historical execution actions include: interfered communication frequency parameters and interference power values ​​of the interfered communication frequencies. The interfered communication frequency parameters are used to indicate at least one interfered communication frequency among multiple communication frequencies that the interfering device interferes with, and the interference power value refers to the power value emitted when multiple interfering devices interfere with the interfered communication frequency.

[0053] Exemplarily, following the above embodiment, the interfered communication frequency parameter is set to X nm , X bn Indicates whether the nth interfering device among N interfering devices interferes with the mth interfered communication frequency among M communication frequencies. Specifically, assuming that X is set nm is 1, or X nm is 0, when X nm =1, it is determined that the nth interference device interferes with the mth communication frequency. nm A value of 0 determines that the nth interference device does not interfere with the mth communication frequency.

[0054] Or, when X nm =1 determines that the nth interference device does not interfere with the mth communication frequency point. nm The nth interference device is determined to interfere with the mth communication frequency point by 0. Based on this, the interference device can determine which communication frequency point among the M communication frequencies to interfere with through the interfered communication frequency point parameters, that is, one or more interfered communication frequencies to interfere with are determined among multiple communication frequencies, but it is not limited to this, the present invention is not specifically limited, and those skilled in the art can set it according to actual conditions.

[0055] The above power status refers to the current remaining power of the multiple interference devices.

[0056] Based on this, on the basis of the above embodiments, since the current power level of the interfering device can directly affect the effect of interfering with the interfered communication frequency, and avoid the phenomenon that the interfering device cannot achieve continuous interference with the interfered communication frequency due to insufficient power of the interfering device, based on this, in some embodiments of the present invention, the power status of the interfering device is periodically updated.

[0057] For example, for the battery of the interference device, the battery is periodically added to the interference device, or the interference device is charged. Based on this, one implementation method of periodically updating the power state of the interference device may be to determine the initial power state E of each interference device. max , after a preset time, obtain the power status of the interference device at the current moment, However, the present invention is not limited thereto, and persons skilled in the art may configure the configuration according to actual conditions.

[0058] The interference-to-signal ratio of the interfered device refers to the interfered device and is used to characterize the interference degree of the current interfered device. The interference-to-signal ratio can be calculated according to a corresponding function calculation formula.

[0059] Optionally, the function calculation formula of the interference-to-signal ratio can be defined by the following expression:

[0060]

[0061] Where M represents the number of interfered communication frequencies corresponding to multiple interfered devices, N represents the number of interfering devices, and p nm (t) represents the interference power value of the nth interference device to the mth interfered communication frequency at time t, X nm (t) represents the interfered communication frequency parameter of the nth interference device to the mth interfered communication frequency at time t, G nk It represents the gain value of the nth interfering device to the kth interfered device, L nk represents the transmission loss of the nth interfering device to the kth interfered device, σ 2 Represents the noise power value, P k (t) represents the communication power value of the kth interfered device itself at time t, G k (t) represents the communication channel gain value of the kth interfered device at time t, L k It represents the transmission loss in the communication process of the interfered device itself. It should be noted that the present invention does not specifically limit the communication channel gain value, gain value and noise power value, and those skilled in the art can determine them according to actual conditions.

[0062] Specifically, for multiple interfering devices, multiple interfered devices, and multiple communication frequencies corresponding to the interfered devices contained in the target area, corresponding environmental state parameters are obtained, such as the historical execution actions of the interfering device, that is, the interfered communication frequency parameters of at least one interfered communication frequency that indicates that the interfering device interferes with the multiple communication frequencies, as well as the interference power value of the interfered communication frequency, the current power state of the interfering device, and the interference-to-signal ratio of the interfered device.

[0063] S11: Input the environmental state parameters into the trained interference strategy generation model to obtain the target interference strategy.

[0064] Among them, the interference strategy generation model is trained using a multi-agent hybrid proximal strategy. The multi-agent hybrid proximal strategy refers to an extended form of the proximal policy optimization algorithm (PPO). The multi-agent hybrid proximal strategy allows multiple agents, such as multiple interference devices, to exchange information and share experience, thereby accelerating training and improving the efficiency of strategy updates, thereby improving the efficiency of obtaining the target interference strategy through the interference strategy generation model.

[0065] The above-mentioned interference strategy generation model is composed of a strategy network and a value network, wherein the strategy network can be an actor network, the value network can be a critic network, the actor network includes an input layer, multiple hidden layers and an output layer, and the output layer includes a first output sublayer for outputting the parameters of the interfered communication frequency point and a second output sublayer for outputting the interference power value.

[0066] Exemplary, reference Figure 3 As shown, for the actor network, it includes an input layer 30, multiple hidden layers 31 and an output layer 32. The output layer 32 includes a first output sublayer 321 for outputting interfered communication frequency parameters and a second output sublayer 322 for outputting interference power values, but is not limited to this. The present invention is not specifically limited, and those skilled in the art can determine it according to actual conditions.

[0067] S12: For the target area, according to the target interference strategy, the interference device is scheduled to interfere with the interfered communication frequency point by using the interference power value of the interfered communication frequency point.

[0068] Specifically, the environmental state parameters in the target area are input into the interference strategy generation model trained by using the multi-agent hybrid proximal strategy to obtain the target interference strategy output by the interference strategy generation model. For multiple interference devices contained in the target area, the multiple interference devices are scheduled according to the target interference strategy, so that the multiple interference devices transmit interference power of the interference power value to one or more interfered communication frequencies determined to be interfered among the multiple communication frequencies of the interference devices to interfere with the one or more interfered communication frequencies.

[0069] In this way, the interference resource scheduling method for the air-ground integrated network provided in this embodiment obtains corresponding environmental state parameters, such as the historical execution action of the interfering device, that is, the interfered communication frequency parameter of at least one interfered communication frequency that indicates that the interfering device interferes with multiple communication frequencies, and the interference power value of the interfered communication frequency, the current power state of the interfering device, and the interference-signal ratio of the interfered device, by targeting multiple interfering devices, multiple interfered devices, and multiple communication frequencies corresponding to the interfered devices contained in the target area. The obtained environmental state parameters are input into the interference strategy generation model obtained by training using the multi-agent hybrid proximal strategy to obtain the target interference strategy. Since the multi-agent hybrid proximal strategy allows multiple interfering devices to exchange information and share experience, it has the characteristics of accelerating training and improving strategy updating. In addition, the interference strategy generation model can simultaneously train the interfered communication frequency parameters and the interference power value, which can reduce the training complexity and accelerate the convergence speed of the interference strategy generation model. Therefore, the efficiency of obtaining the target interference strategy is improved, and further, the efficiency of scheduling the interfering device to interfere with the interfered communication frequency through the interference power value according to the target interference strategy is improved.

[0070] Optionally, based on the above embodiment, in some embodiments of the present invention, before executing S11, the process further includes:

[0071] S21: Establish a target problem function set for the interference resource scheduling problem of multiple interfering devices, multiple interfered devices, and multiple communication frequency points corresponding to the interfered devices in the target area.

[0072] The target problem function set refers to the mathematical model constructed to solve the target interference strategy.

[0073] Specifically, for the interference resource scheduling problem of multiple interfering devices, multiple interfered devices, and multiple communication frequency points corresponding to the interfered devices in the target area, a corresponding target problem function set is established.

[0074] Optionally, based on the above embodiment, in some embodiments of the present invention, the target problem function set can be defined by the following expression:

[0075]

[0076] X nm (t)∈{0,1},n∈N,m∈M

[0077]

[0078] Where M represents the number of interfered communication frequencies corresponding to multiple interfered devices, N represents the number of interfering devices, K represents the number of interfered devices, and p nm (t) represents the interference power value of the nth interference device to the mth interfered communication frequency at time t, X nm (t) represents the interfered communication frequency parameter of the nth interference device to the mth interfered communication frequency at time t, G nk It represents the gain value of the nth interfering device to the kth interfered device, L nk represents the transmission loss of the nth interfering device to the kth interfered device, σ 2 Represents the noise power value, p k (t) represents the communication power value of the kth interfered device itself at time t, G k (t) represents the communication channel gain value of the kth interfered device at time t, L k Indicates the transmission loss of the interfered device during its own communication process, γ k Indicates the interference suppression coefficient set by the interference device on the interference power value of the interfered device, p max It represents the maximum interference power value emitted by the interference device, and H represents the maximum number of interference communication frequencies that a interference device can interfere at the same time.

[0079] It should be noted that, following the above embodiment, since the power status of the interference device is updated periodically, based on this, the target problem function set is updated in real time, that is,

[0080] S22: For the target problem function set, a multi-agent hybrid proximal strategy is used to generate a model through an interference strategy for model training and solution.

[0081] Specifically, after constructing the target problem function set, the multi-agent hybrid proximal strategy is used to train and solve the target problem function set through the interference strategy generation model to obtain the result corresponding to the target problem function set, namely the target interference strategy.

[0082] Optionally, based on the above embodiment, in some embodiments of the present invention, an implementation manner of S22 may be:

[0083] S221: Initialize the environmental state parameters of the target area.

[0084] Specifically, the environmental status parameters of the target area, such as the historical execution actions of the interfering device, that is, the interfered communication frequency parameters used to indicate that the interfering device interferes with at least one interfered communication frequency among multiple communication frequencies, the interference power value of the interfered communication frequency, the current power status of the interfering device, and the interference-to-signal ratio of the interfered device are initialized.

[0085] Optionally, based on the above embodiments, in some embodiments of the present invention, an implementation method of S221 may be: from the historical environmental state parameters of multiple communication frequency points at multiple times for multiple interference devices in the target area, obtain the environmental state parameters at the most recent moment to initialize the environmental state parameters of the target area.

[0086] Optionally, based on the above embodiments, in some embodiments of the present invention, another implementation method of S221 may be: randomly obtaining environmental state parameters at any time from historical environmental state parameters at multiple times to initialize the environmental state parameters of the target area.

[0087] Optionally, based on the above embodiments, in some embodiments of the present invention, another implementation method of S221 may be: calculating the mean of historical environmental state parameters at multiple moments, and using the mean to initialize the environmental state parameters of the target area.

[0088] S222: According to the environmental state parameters, the initial interference strategy generation model is trained using a multi-agent hybrid proximal strategy based on a training loss function to obtain a trained interference strategy generation model.

[0089] The training loss function refers to a function used to set when training the interference strategy generation model.

[0090] Optionally, based on the above embodiment, in some embodiments of the present invention, a way to set the training loss function may be:

[0091] For the initial interference strategy generation model, the training loss function is obtained by weighting the value loss function, strategy loss function, and entropy regularization term of the value network.

[0092] The strategy loss function is related to the parameters of the interfered communication frequency point and the interference power value of the interfered communication frequency point. Exemplarily, the strategy loss function can be defined by the following expression:

[0093] L 1 (θ) = L 11 +L 12

[0094] Among them, L 11 represents the loss function for the interfered communication frequency parameters set during the training of the interference strategy generation model, L 12 Represents the loss function for the interference power value set during the training of the interference strategy generation model.

[0095] Optionally, based on the above embodiment, in some embodiments of the present invention, the training loss function may be defined by the following expression:

[0096]

[0097] Among them, L 1 (θ) represents the strategy loss function, represents the value loss function of the value network, X[π θ (S t )] represents the entropy regularization term, S t represents the environmental space parameters, θ and represents the network parameters of the interference strategy generation model, a and b represent weighting factors, and the present invention does not specifically limit the values ​​of a and b, and those skilled in the art can set them according to actual conditions.

[0098] In this way, the interference resource scheduling method for the air-ground integrated network provided by the present invention obtains a training loss function by weighting according to the value loss function of the critic network, the strategy loss function related to the interfered communication frequency parameters and the interference power value of the interfered communication frequency, and the entropy regularization term, which can ensure the coordinated optimization between the value network and the strategy network.

[0099] Specifically, after the environmental state parameters of the target area are initialized, the environmental state parameters after initialization are input into the initial interference strategy generation model. According to the environmental state parameters after initialization, a multi-agent hybrid proximal strategy is used, and the initial interference strategy generation model is trained based on the training loss function to obtain a trained interference strategy generation model.

[0100] In this way, the interference resource scheduling method of the air-ground integrated network provided by the present invention, after initializing the environmental state parameters of the target area, inputs the environmental state parameters after the initialization processing into the initial interference strategy generation model, and uses the multi-agent hybrid proximal strategy according to the environmental state parameters after the initialization processing, and trains the initial interference strategy generation model based on the training loss function, thereby obtaining a trained interference strategy generation model, which can use the multi-agent hybrid proximal strategy to allow multiple interference devices to exchange information and share experience, and has the characteristics of accelerating training and improving strategy updates, thereby improving the efficiency of obtaining the target interference strategy through the interference strategy generation model. Further, the efficiency of scheduling the interference device according to the target interference strategy to interfere with the interfered communication frequency through the interference power value is improved.

[0101] Optionally, based on the above embodiment, in some embodiments of the present invention, an implementation manner of S222 may be:

[0102] The environmental state parameters are input into the initial interference strategy generation model. The intermediate execution action is obtained through the policy network contained in the initial interference strategy generation model. The interference-signal ratios corresponding to multiple interference devices are determined according to the intermediate execution action. The reward value of the intermediate execution action is calculated through the reward function set by the value network. The intermediate execution action, reward value, and interference-signal ratio are returned and input into the initial interference strategy generation model. Training is performed based on the training loss function until the initial interference strategy generation model converges to obtain a trained interference strategy generation model.

[0103] Among them, the intermediate execution action refers to the execution action obtained by the strategy network during the training of the initial interference strategy generation model, and the execution action is the parameters of the interfered communication frequency point and the interference power value of the interfered communication frequency point.

[0104] Specifically, the environmental state parameters after initialization processing are input into the initial interference strategy generation model. The policy network included in the initial interference strategy generation model outputs intermediate execution actions for the environmental state parameters, and by setting a reward function for the value network, the value network calculates and scores the intermediate execution actions according to the reward function to obtain the corresponding reward value. When the intermediate execution action is obtained, the interference-to-signal ratio of the interfered device at the current moment is calculated using the calculation function for calculating the interference-to-signal ratio. When the initial interference strategy generation model is converged, the intermediate execution action, reward value and interference-to-signal ratio are re-feedback as global environmental observation variables and input into the untrained interference strategy generation model. Training is performed based on the training loss function, and the corresponding network parameters of the policy network and the value network are adjusted until the initial interference strategy generation model converges to obtain a trained interference strategy generation model.

[0105] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0106] In one embodiment, Figure 4 As shown, an interference resource scheduling device for an air-ground integrated network is provided, comprising: an environment state parameter acquisition module 10, a target interference strategy acquisition module 11, and a scheduling module 12.

[0107] Among them, the environmental state parameter acquisition module 10 is used to obtain the environmental state parameters of the target area, wherein the target area includes multiple interfering devices, multiple interfered devices, and multiple communication frequency points corresponding to the interfered devices, and the environmental state parameters at least include: historical execution actions of the interfering device, power status, and interference-to-signal ratio of the interfered device, and the historical execution actions include: interfered communication frequency point parameters and interference power values ​​of the interfered communication frequency points, and the interfered communication frequency point parameters are used to indicate at least one interfered communication frequency point that the interfering device interferes with among multiple communication frequency points;

[0108] A target interference strategy acquisition module 11 is used to input the environmental state parameters into a trained interference strategy generation model to obtain a target interference strategy, wherein the interference strategy generation model is obtained by training using a multi-agent hybrid proximal strategy, and the interference strategy generation model is composed of a strategy network and a value network, and the strategy network includes an input layer, multiple hidden layers, and an output layer, and the output layer includes a first output sublayer for outputting the interfered communication frequency point parameters and a second output sublayer for outputting the interference power value;

[0109] The scheduling module 12 is used to schedule the interference device to interfere with the interfered communication frequency point by using the interference power value of the interfered communication frequency point according to the target interference strategy for the target area.

[0110] In the above embodiment, for multiple interfering devices, multiple interfered devices, and multiple communication frequencies corresponding to the interfered devices contained in the target area, the corresponding environmental state parameters are obtained through the environmental state parameter acquisition module, such as the historical execution action of the interfering device, that is, the interfered communication frequency parameters of at least one interfered communication frequency used to indicate that the interfering device interferes with multiple communication frequencies, and the interference power value of the interfered communication frequency, the power state of the interfering device at the current moment, and the interference-to-signal ratio of the interfered device. The obtained environmental state parameters are input into the interference strategy generation model obtained by training using the multi-agent hybrid proximal strategy through the target interference strategy acquisition module to obtain the target interference strategy. Since the multi-agent hybrid proximal strategy allows multiple interfering devices to exchange information and share experience, it has the characteristics of accelerating training and improving strategy updating. In addition, the interference strategy generation model can be trained for the interfered communication frequency parameters and the interference power value at the same time, which can reduce the training complexity and accelerate the convergence speed of the interference strategy generation model. Therefore, the efficiency of obtaining the target interference strategy is improved. Further, the efficiency of using the scheduling module to schedule the interfering device to interfere with the interfered communication frequency through the interference power value according to the target interference strategy is improved.

[0111] For the specific definition of the interference resource scheduling device of the air-ground integrated network, please refer to the definition of the interference resource scheduling method of the air-ground integrated network mentioned above, which will not be repeated here. Each module in the above-mentioned server can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0112] An embodiment of the present invention provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the interference resource scheduling method for the air-ground integrated network provided in the embodiment of the present invention can be implemented. For example, when the processor executes the computer program, Figure 1 The technical solution of any of the method embodiments shown has similar implementation principles and technical effects, which will not be repeated here.

[0113] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided by the present invention can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static random access memory (SRAM) and dynamic random access memory (DRAM).

[0114] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0115] The above-mentioned embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A method for scheduling interference resources in an air-ground integrated network, characterized in that: include: Acquire environmental state parameters of a target area, wherein the target area includes a plurality of interfering devices, a plurality of interfered devices, and a plurality of communication frequencies corresponding to the interfered devices, and the environmental state parameters include at least: a historical execution action of the interfering device, a power state, and an interference-to-signal ratio of the interfered device, and the historical execution action includes: an interfered communication frequency parameter and an interference power value of the interfered communication frequency, and the interfered communication frequency parameter is used to indicate at least one interfered communication frequency that the interfering device interferes with among the plurality of communication frequencies; Input the environmental state parameters into a trained interference strategy generation model to obtain a target interference strategy, wherein the interference strategy generation model is trained using a multi-agent hybrid proximal strategy, and the interference strategy generation model is composed of a strategy network and a value network, and the strategy network includes an input layer, multiple hidden layers, and an output layer, and the output layer includes a first output sublayer for outputting the interfered communication frequency point parameters and a second output sublayer for outputting the interference power value; For the target area, according to the target interference strategy, the interference device is scheduled to interfere with the interfered communication frequency point by using the interference power value of the interfered communication frequency point.

2. The method according to claim 1, characterized in that Before inputting the environmental state parameters into the trained interference strategy generation model to obtain the target interference strategy, the method further includes: Establishing a target problem function set for the interference resource scheduling problem of multiple interfering devices, multiple interfered devices, and multiple communication frequency points corresponding to the interfered devices in the target area; For the target problem function set, a multi-agent hybrid proximal strategy is used to generate a model through an interference strategy to perform model training and solution.

3. The method according to claim 2, characterized in that The target problem function set can be defined by the following expression: X nm (t)∈{0,1},n∈N,m∈M Where M represents the number of interfered communication frequencies corresponding to multiple interfered devices, N represents the number of interfering devices, K represents the number of interfered devices, and p nm (t) represents the interference power value of the nth interference device to the mth interfered communication frequency at time t, X nm (t) represents the interfered communication frequency parameter of the nth interference device to the mth interfered communication frequency at time t, G nk It represents the gain value of the nth interfering device to the kth interfered device, L nk represents the transmission loss of the nth interfering device to the kth interfered device, σ 2 Represents the noise power value, p k (t) represents the communication power value of the kth interfered device itself at time t, G k (t) represents the communication channel gain value of the kth interfered device at time t, L k Indicates the transmission loss of the interfered device during its own communication process, γ k Indicates the interference suppression coefficient set by the interference device on the interference power value of the interfered device, p max It represents the maximum interference power value emitted by the interference device, and H represents the maximum number of interference communication frequencies that a interference device can interfere at the same time.

4. The method according to claim 2, characterized in that: The method of using a multi-agent hybrid proximal strategy to generate a model through an interference strategy to perform model training and solving includes: Initializing environmental state parameters of the target area; According to the environmental state parameters, a multi-agent hybrid proximal strategy is used to train the initial interference strategy generation model based on a training loss function to obtain a trained interference strategy generation model.

5. The method according to claim 4, characterized in that Before the initial interference strategy generation model is trained based on the training loss function using the multi-agent hybrid proximal strategy according to the environmental state parameters to obtain the trained interference strategy generation model, the method further includes: For the initial interference strategy generation model, weighting is performed according to the value loss function, the strategy loss function, and the entropy regularization term of the value network to obtain the training loss function, wherein the strategy loss function is related to the interfered communication frequency parameters and the interference power value of the interfered communication frequency.

6. The method according to claim 5, characterized in that The training loss function is obtained by weighting the value loss function, the strategy loss function, and the entropy regularization term of the value network, including: According to the formula Obtaining the training loss function; Among them, L 1 (θ) represents the strategy loss function, represents the value loss function of the value network, X[π θ (S t )] represents the entropy regularization term, S t represents the environmental space parameters, θ and represents the network parameters of the interference strategy generation model, and a and b represent weighting factors.

7. The method according to claim 2, characterized in that The method of training the initial interference strategy generation model based on the training loss function using the multi-agent hybrid proximal strategy according to the environmental state parameters to obtain the trained interference strategy generation model includes: The environmental state parameters are input into the initial interference strategy generation model, and an intermediate execution action is obtained through the strategy network included in the initial interference strategy generation model. The interference-signal ratios corresponding to multiple interference devices are determined according to the intermediate execution actions. The reward value of the intermediate execution action is calculated through the reward function set by the value network. The intermediate execution action, the reward value, and the interference-signal ratio are returned and input into the initial interference strategy generation model. Training is performed based on the training loss function until the initial interference strategy generation model converges to obtain a trained interference strategy generation model.

8. The method according to claim 1, characterized in that The method further comprises: The power status of the interfering device is periodically updated.

9. An interference resource scheduling device for an air-ground integrated network, characterized in that: include: An environmental state parameter acquisition module is used to acquire environmental state parameters of a target area, wherein the target area includes multiple interfering devices, multiple interfered devices, and multiple communication frequency points corresponding to the interfered devices, and the environmental state parameters at least include: historical execution actions of the interfering device, power status, and interference-to-signal ratio of the interfered device, and the historical execution actions include: interfered communication frequency point parameters and interference power values ​​of the interfered communication frequency points, and the interfered communication frequency point parameters are used to indicate at least one interfered communication frequency point that the interfering device interferes with among the multiple communication frequency points; A target interference strategy acquisition module, used for inputting the environmental state parameters into a trained interference strategy generation model to obtain a target interference strategy, wherein the interference strategy generation model is obtained by training using a multi-agent hybrid proximal strategy, and the interference strategy generation model is composed of a strategy network and a value network, and the strategy network includes an input layer, multiple hidden layers, and an output layer, and the output layer includes a first output sublayer for outputting the interfered communication frequency point parameters and a second output sublayer for outputting the interference power value; The scheduling module is used to schedule the interference device to interfere with the interfered communication frequency point by using the interference power value of the interfered communication frequency point according to the target interference strategy for the target area.

10. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the interference resource scheduling method for the air-ground integrated network described in any one of claims 1 to 8 are implemented.

Images