An interference resource scheduling method, device and equipment of an air-ground integrated network

Abstract

The present application relates to the technical field of communication interference, and provides an interference resource scheduling method, device and equipment for an air-ground integrated network, wherein corresponding environmental state parameters are obtained by aiming at a plurality of interference devices, a plurality of interfered devices and a plurality of communication frequency points corresponding to the interfered devices contained in a target area, the environmental state parameters are input into an interference strategy generation model trained by using a multi-agent hybrid near-end strategy, and a target interference strategy is obtained. Since the multi-agent hybrid near-end strategy allows multiple interference devices to exchange information and share experience, it has the characteristics of accelerating training and improving strategy updating, and the interference strategy generation model can simultaneously train the interfered communication frequency point parameters and the interference power value, thereby reducing the training complexity, accelerating the convergence speed of the model, improving the efficiency of obtaining the target interference strategy, and further improving the efficiency of scheduling the interference devices to interfere with the interfered communication frequency points according to the target interference strategy.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of communication interference, and in particular to an interference resource scheduling method, device and equipment for an air-ground integrated network. BACKGROUND

[0002] With the continuous evolution of electronic warfare technology, in the closed-loop electromagnetic spectrum combat process of perception, decision-making, action and evaluation, scientific decision-making as a key link for electronic countermeasure effectiveness plays a key role, and the core is how to realize efficient allocation and scheduling of resources. Specifically, it mainly involves optimization of current interference resource parameters to realize efficient allocation of interference resources.

[0003] Based on this, in the prior art, interference resource allocation is realized by deep reinforcement learning (Deep Reinforcement Learning, DRL) technology. Specifically, a single-agent reinforcement learning method is used, centralized control is adopted, global state and action space are constructed to realize interference resource allocation, or a multi-agent reinforcement learning method is used, each device is independently modeled and cooperative decision-making is realized to realize interference resource allocation.

[0004] However, using the prior art, since the single-agent reinforcement learning method has the problems of high decision-making dimension and high communication overhead, and the multi-agent reinforcement learning method has the problem of high training complexity, when the prior art is used to realize interference resource allocation for multiple agents, there are problems of poor scalability and low training efficiency, which reduces the efficiency of interference resource allocation. SUMMARY

[0005] Based on this, it is necessary to provide an interference resource scheduling method, device and equipment for an air-ground integrated network in view of the above technical problems.

[0006] In a first aspect, an interference resource scheduling method for an air-ground integrated network is provided, and the method comprises:

[0007] obtaining an environment state parameter of a target area, wherein the target area includes a plurality of interference devices, a plurality of interfered devices, and a plurality of communication frequency points corresponding to the interfered devices, and the environment state parameter at least includes a historical execution action of the interference device, a power state, and a jamming-to-signal ratio value of the interfered device, the historical execution action includes an interfered communication frequency point parameter and an interference power value of the interfered communication frequency point, and the interfered communication frequency point parameter is used to indicate at least one interfered communication frequency point in which the interference device interferes in the plurality of communication frequency points;

[0008] input the environment state parameter into the trained interference strategy generation model to obtain a target interference strategy, wherein the interference strategy generation model is obtained by training using a multi-agent hybrid near-end strategy, the interference strategy generation model is composed of a policy network and a value network, the policy network includes an input layer, a plurality of hidden layers, and an output layer, the output layer includes a first output sublayer for outputting the interfered communication frequency point parameter and a second output sublayer for outputting the interference power value;

[0009] According to the target interference strategy, the interference device is scheduled to interfere with the interfered communication frequency point through the interference power value of the interfered communication frequency point.

[0010] In one embodiment, before the environment state parameter is input into the trained interference strategy generation model to obtain the target interference strategy, the method further comprises:

[0011] For the interference resource scheduling problem of a plurality of interference devices, a plurality of interfered devices, and a plurality of communication frequency points corresponding to the interfered devices in the target area, a target problem function set is established.

[0012] For the target problem function set, a model training solution is performed by using a multi-agent hybrid near-end strategy through an interference strategy generation model.

[0013] In one embodiment, the target problem function set can be defined by the following expression:

[0014]

[0015] X nm (t)∈{0,1},n∈N,m∈M

[0016]

[0017] wherein M represents the number of interfered communication frequency points corresponding to a plurality of interfered devices, N represents the number of interference devices, K represents the number of interfered devices, p nm (t) represents the interference power value of the nth interference device to the mth interfered communication frequency point at time t, X nm (t) represents the interfered communication frequency point parameter of the nth interference device to the mth interfered communication frequency point at time t, G nk represents the gain value of the nth interference device to the kth interfered device, L nk represents the transmission loss of the nth interference device to the kth interfered device, σ 2 represents the noise power value, p k (t) represents the communication power value of the kth interfered device to itself at time t, G k(t) represents the communication channel gain value of the kth interfered device at time t, L k represents the transmission loss in the communication process of the interfered device itself, γ k represents the interference suppression coefficient set by the interference device for the interference power value of the interfered device, p max represents the maximum interference power value transmitted by the interference device, H represents the maximum number of interference communication frequency points that can be interfered by one interference device at the same time.

[0018] In one embodiment, the model training solution by using the multi-agent hybrid proximal policy through the interference policy generation model includes:

[0019] Initialize the environment state parameters of the target area;

[0020] According to the environment state parameters, the initial interference policy generation model is trained based on the training loss function by using the multi-agent hybrid proximal policy to obtain the trained interference policy generation model.

[0021] In one embodiment, before the environment state parameters are used to train the initial interference policy generation model based on the training loss function by using the multi-agent hybrid proximal policy to obtain the trained interference policy generation model, it further includes:

[0022] For the initial interference policy generation model, the value loss function of the value network, the policy loss function, and the entropy regularization term are weighted to obtain the training loss function, wherein the policy loss function is related to the interfered communication frequency point parameters and the interference power value of the interfered communication frequency point.

[0023] In one embodiment, the value loss function of the value network, the policy loss function, and the entropy regularization term are weighted to obtain the training loss function, including:

[0024] According to the formula the training loss function is obtained;

[0025] wherein L 1 (θ) represents the policy loss function, represents the value loss function of the value network, X[π θ (S t )] represents the entropy regularization term, S t represents the environment space parameter, θ and represents the network parameter of the interference policy generation model, and a and b represent weighting factors.

[0026] In one embodiment, the training of the initial interference strategy generation model based on the training loss function according to the environment state parameter by using the multi-agent hybrid near-end strategy to obtain the trained interference strategy generation model comprises:

[0027] The environment state parameter is input into the initial interference strategy generation model, and an intermediate execution action is obtained through the policy network included in the initial interference strategy generation model. The jam-to-signal ratio values corresponding to the plurality of interference devices are determined according to the intermediate execution action. The reward value of the intermediate execution action is calculated through the reward function set by the value network. The intermediate execution action, the reward value, and the jam-to-signal ratio values are returned and input into the initial interference strategy generation model. The initial interference strategy generation model is trained based on the training loss function until the initial interference strategy generation model converges, and the trained interference strategy generation model is obtained.

[0028] In one embodiment, the method further comprises:

[0029] Periodically updating the power state of the interference device.

[0030] In a second aspect, an embodiment of the present application provides an interference resource scheduling device for an air-ground integrated network, comprising:

[0031] An environment state parameter acquisition module is configured to acquire an environment state parameter of a target region, wherein the target region includes a plurality of interference devices, a plurality of interfered devices, and a plurality of communication frequency points corresponding to the interfered devices. The environment state parameter at least includes a historical execution action of the interference device, a power state, and a jam-to-signal ratio value of the interfered device. The historical execution action includes an interfered communication frequency point parameter and an interference power value of the interfered communication frequency point. The interfered communication frequency point parameter is used to indicate at least one interfered communication frequency point in which the interference device performs interference among the plurality of communication frequency points.

[0032] A target interference strategy acquisition module is configured to input the environment state parameter into the trained interference strategy generation model to acquire a target interference strategy. The interference strategy generation model is obtained by training using the multi-agent hybrid near-end strategy. The interference strategy generation model is composed of a policy network and a value network. The policy network includes an input layer, a plurality of hidden layers, and an output layer. The output layer includes a first output sublayer for outputting the interfered communication frequency point parameter and a second output sublayer for outputting the interference power value.

[0033] A scheduling module is configured to, for the target region, schedule the interference device to interfere with the interfered communication frequency point through the interference power value of the interfered communication frequency point according to the target interference strategy.

[0034] In a third aspect, an embodiment of the present invention provides an electronic device comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it implements the steps of the interference resource scheduling method for the air-ground integrated network described in the first aspect.

[0035] The technical solution provided by the embodiment of the present invention has the following advantages compared with the existing technology:

[0036] An interference resource scheduling method, apparatus and equipment for an air-ground integrated network provided by an embodiment of the present invention adopts this method to obtain corresponding environmental state parameters by targeting multiple interfering devices, multiple interfered devices and multiple communication frequencies corresponding to the interfered devices contained in the target area, such as the historical execution action of the interfering device, that is, the interfered communication frequency parameter of at least one interfered communication frequency point used to indicate that the interfering device interferes with multiple communication frequencies, as well as the interference power value of the interfered communication frequency point, the current power state of the interfering device, and the interference-to-signal ratio of the interfered device. The obtained environmental state parameters are input into the multi-intelligence system. The target interference strategy is obtained from the interference strategy generation model obtained by training the multi-agent hybrid proximal strategy. Since the multi-agent hybrid proximal strategy allows multiple interference devices to exchange information and share experience, it has the characteristics of accelerated training and improved strategy updating. In addition, the interference strategy generation model can simultaneously train the parameters of the interfered communication frequency point and the interference power value, which can reduce the training complexity and accelerate the convergence speed of the interference strategy generation model. Therefore, the efficiency of obtaining the target interference strategy is improved. Further, the efficiency of scheduling the interference device to interfere with the interfered communication frequency point through the interference power value according to the target interference strategy is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0038] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0039] Figure 1 A flow chart of an interference resource scheduling method for an air-ground integrated network provided by an embodiment of the present invention;

[0040] Figure 2 A schematic diagram of a scenario of an interference resource scheduling method for an air-ground integrated network provided by an embodiment of the present invention;

[0041] Figure 3 A network model schematic diagram of an interference strategy generation model provided for an embodiment of the present application;

[0042] Figure 4 A structural schematic diagram of an interference resource scheduling device of a space-air integrated network provided for an embodiment of the present application. DETAILED DESCRIPTION

[0043] In order to more clearly understand the above-mentioned purposes, features and advantages of the present application, the schemes of the present application will be further described below. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0044] In the following description, many specific details are set forth in order to provide a thorough understanding of the present application, but the present application can also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some of the embodiments of the present application, not all the embodiments.

[0045] In the closed-loop electromagnetic spectrum combat process of perception, decision, action and evaluation, how to realize the optimization of interference resource parameters to realize the efficient allocation of interference resources. In the prior art, a single-agent reinforcement learning method is specifically used, centralized control is adopted, the global state and action space are constructed to realize interference resource allocation, or a multi-agent reinforcement learning method is used, each device is independently modeled and cooperatively decides to realize interference resource allocation.

[0046] However, using the prior art, due to the problems of high decision dimension and large communication overhead of the single-agent reinforcement learning method, and the problem of high training complexity of the multi-agent reinforcement learning method, when the prior art is used to realize interference resource allocation, the problems of poor expansibility and low training efficiency exist.

[0047] Therefore, the present application provides an interference resource scheduling method of an air-ground integrated network. By obtaining corresponding environmental state parameters of a plurality of interfering devices, a plurality of interfered devices, and a plurality of communication frequency points corresponding to the interfered devices in a target area, such as the historical execution actions of the interfering devices, i.e., the interfered communication frequency point parameters of at least one interfered communication frequency point for indicating the interference of the interfering devices in the plurality of communication frequency points, and the interference power value of the interfered communication frequency point, the power state of the interfering device at the current time, and the interference-to-signal ratio value of the interfered device, the obtained environmental state parameters are input into an interference strategy generation model trained by using a multi-agent hybrid near-end strategy to obtain a target interference strategy. Since the multi-agent hybrid near-end strategy allows information exchange and experience sharing among the plurality of interfering devices, it has the characteristics of accelerating training and improving strategy updating. Moreover, the interference strategy generation model can simultaneously train the interfered communication frequency point parameters and the interference power value, which can reduce the training complexity and accelerate the convergence speed of the interference strategy generation model. Therefore, the efficiency of obtaining the target interference strategy is improved, and the efficiency of scheduling the interfering devices to interfere with the interfered communication frequency point by the interference power value according to the target interference strategy is further improved.

[0048] In one embodiment, as shown in Figure 1 , Figure 1 A flowchart of an interference resource scheduling method of an air-ground integrated network according to an embodiment of the present application is shown. The method includes the following steps:

[0049] S10: Obtain the environmental state parameters of the target area.

[0050] The target area includes a plurality of interfering devices, a plurality of interfered devices, and a plurality of communication frequency points corresponding to the interfered devices. The communication frequency point refers to the frequency point that the interfered device can select a new communication frequency point for communication after being interfered by the interfered device.

[0051] For example, as shown in Figure 2 , when interference resource allocation is performed for the target area 20, N interfering devices 201 are set. The set of N interfering devices 201 is N={1, 2, …, N}. K interfered devices 202 are set. The set of K interfered devices 202 is K={1, 2, …, K}. M communication frequency points are set. The set of M communication frequency points is N={1, 2, …, M}. It should be noted that the channels of the communication frequency points are mutually independent orthogonal channels with the same bandwidth, but are not limited thereto. The present application does not specifically limit this, and those skilled in the art can set it according to the actual situation.

[0052] The environmental state parameter at least includes: a historical execution action of the interference device, and a power state, wherein the historical execution action includes: a disturbed communication frequency point parameter and an interference power value of the disturbed communication frequency point, the disturbed communication frequency point parameter is used to indicate at least one disturbed communication frequency point in a plurality of communication frequency points interfered by the interference device, and the interference power value refers to a power value emitted by the plurality of interference devices when interfering with the disturbed communication frequency point.

[0053] For example, in the above embodiment, the disturbed communication frequency point parameter is set as X nm , X bn indicates whether the nth interference device in the N interference devices interferes with the mth disturbed communication frequency point in the M communication frequency points, specifically, assuming that X nm is set as 1 or X nm is set as 0, when X nm is 1, it is determined that the nth interference device interferes with the mth communication frequency point, and when X nm is 0, it is determined that the nth interference device does not interfere with the mth communication frequency point.

[0054] Or, when X nm is 1, it is determined that the nth interference device does not interfere with the mth communication frequency point, and when X nm is 0, it is determined that the nth interference device interferes with the mth communication frequency point. Based on this, the interference device can determine which one of the M communication frequency points is interfered by the disturbed communication frequency point parameter, that is, one or more disturbed communication frequency points in the plurality of communication frequency points are determined to be interfered, but not limited to this, the present application does not specifically limit it, and those skilled in the art can set it according to the actual situation.

[0055] The power state refers to the current remaining power of the plurality of interference devices.

[0056] Based on the above embodiment, since the current power of the interference device can directly affect the interference effect on the disturbed communication frequency point, and the phenomenon that the interference device cannot realize sustained interference when interfering with the disturbed communication frequency point due to insufficient power of the interference device is avoided, based on this, in some embodiments of the present application, the power state of the interference device is periodically updated.

[0057] For example, for the battery of the interference device, the battery of the interference device is periodically increased or the interference device is charged. Based on this, one implementation of periodically updating the power state of the interference device can be to determine the initial power state E max of each interference device, and after a predetermined time, the power state of the interference device at the current time is obtained, but not limited to this, the present application does not specifically limit it, and those skilled in the art can set it according to the actual situation.

[0058] The interference to signal ratio of the above-mentioned interfered device refers to, for the interfered device, a value used to represent the degree of interference of the current interfered device, and the interference to signal ratio can be calculated according to a corresponding function calculation formula.

[0059] Optionally, the function calculation formula of the interference to signal ratio can be defined by the following expression:

[0060]

[0061] Wherein, M represents the number of the corresponding interfered communication frequency points of the plurality of interfered devices, N represents the number of the interference devices, p nm (t) represents the interference power value of the nth interference device to the mth interfered communication frequency point at the t time, X nm (t) represents the interfered communication frequency point parameter of the nth interference device to the mth interfered communication frequency point at the t time, G nk represents the gain value of the nth interference device to the kth interfered device, L nk represents the transmission loss of the nth interference device to the kth interfered device, σ 2 represents the noise power value, P k (t) represents the communication power value of the kth interfered device itself at the t time, G k (t) represents the communication channel gain value of the kth interfered device at the t time, L k represents the transmission loss in the communication process of the interfered device itself, and it should be noted that the communication channel gain value, the gain value and the noise power value are not specifically limited in the present application, and can be determined according to the actual situation by the person skilled in the art.

[0062] Specifically, for the plurality of interference devices, the plurality of interfered devices and the plurality of communication frequency points corresponding to the interfered devices contained in the target area, the corresponding environmental state parameters are obtained, such as the historical execution actions of the interference devices, that is, the interfered communication frequency point parameters of at least one interfered communication frequency point for indicating the interference of the interference devices in the plurality of communication frequency points, and the interference power value of the interfered communication frequency point, the power state of the interference device at the current time, and the interference to signal ratio of the interfered device.

[0063] S11: input the environmental state parameters into the trained interference strategy generation model to obtain a target interference strategy.

[0064] Among them, the interference strategy generation model is obtained by training using a multi-agent hybrid proximal strategy. The multi-agent hybrid proximal strategy refers to an extended form of the Proximal Policy Optimization (PPO) algorithm. The multi-agent hybrid proximal strategy allows multiple agents, such as multiple interference devices, to exchange information and share experience, thereby accelerating training and improving the efficiency of strategy updates, thereby improving the efficiency of obtaining the target interference strategy through the interference strategy generation model.

[0065] The above-mentioned interference strategy generation model consists of a strategy network and a value network, wherein the strategy network can be an actor network, the value network can be a critic network, the actor network includes an input layer, multiple hidden layers and an output layer, and the output layer includes a first output sublayer for outputting the parameters of the interfered communication frequency point and a second output sublayer for outputting the interference power value.

[0066] Exemplary, reference Figure 3 As shown, the actor network includes an input layer 30, multiple hidden layers 31 and an output layer 32. The output layer 32 includes a first output sublayer 321 for outputting the interfered communication frequency parameters and a second output sublayer 322 for outputting the interference power value, but is not limited to this. The present invention is not specifically limited, and those skilled in the art can determine it according to actual conditions.

[0067] S12: For the target area, according to the target interference strategy, the interference device is scheduled to interfere with the interfered communication frequency point using the interference power value of the interfered communication frequency point.

[0068] Specifically, the acquired environmental state parameters within the target area are input into an interference strategy generation model trained using a multi-agent hybrid proximal strategy, resulting in a target interference strategy output by the interference strategy generation model. For multiple interfering devices within the target area, the multiple interfering devices are scheduled according to the target interference strategy so that the multiple interfering devices transmit interference power equal to the interference power value to interfere with one or more interfered communication frequencies determined to be interfered with by the multiple communication frequencies of the interfering devices.

[0069] Thus, the interference resource scheduling method of the air-ground integrated network provided by the embodiment can obtain corresponding environmental state parameters, such as a historical execution action of the interference device, that is, a disturbed communication frequency point parameter of at least one disturbed communication frequency point in which the interference device performs interference among the multiple communication frequency points, and an interference power value of the disturbed communication frequency point, a power state of the interference device at the current time, and a jamming-to-signal ratio value of the interfered device, by taking the multiple interference devices, the multiple interfered devices, and the multiple communication frequency points corresponding to the interfered devices in the target area as objects. The obtained environmental state parameters are input into the interference strategy generation model trained by using the multi-agent hybrid near-end strategy to obtain a target interference strategy. Since the multi-agent hybrid near-end strategy allows information exchange and experience sharing among the multiple interference devices, the multi-agent hybrid near-end strategy has the characteristics of accelerating training and improving strategy updating. Moreover, the interference strategy generation model can simultaneously train the disturbed communication frequency point parameter and the interference power value, thereby reducing the training complexity and accelerating the convergence speed of the interference strategy generation model. Therefore, the efficiency of obtaining the target interference strategy is improved, and the efficiency of scheduling the interference device to interfere with the disturbed communication frequency point by using the interference power value according to the target interference strategy is further improved.

[0070] Optionally, before S11, the method further includes:

[0071] S21: establishing a target problem function set for the interference resource scheduling problem of the multiple interference devices, the multiple interfered devices, and the multiple communication frequency points corresponding to the interfered devices in the target area.

[0072] The target problem function set refers to a mathematical model constructed for solving the target interference strategy.

[0073] Specifically, the target problem function set is established for the interference resource scheduling problem of the multiple interference devices, the multiple interfered devices, and the multiple communication frequency points corresponding to the interfered devices in the target area.

[0074] Optionally, the target problem function set can be defined by the following expression based on the above embodiment.

[0075]

[0076] X nm (t)∈{0,1},n∈N,m∈M

[0077]

[0078] Among them, M represents the number of interfered communication frequencies corresponding to multiple interfered devices, N represents the number of interfering devices, K represents the number of interfered devices, and p nm (t) represents the interference power value of the nth interference device to the mth interfered communication frequency at time t, X nm (t) represents the interfered communication frequency parameter of the nth interfering device to the mth interfered communication frequency at time t, G nk It represents the gain value of the nth interfering device to the kth interfered device, L nk represents the transmission loss from the nth interfering device to the kth interfered device, σ 2 Represents the noise power value, p k (t) represents the communication power value of the kth interfered device itself at time t, G k (t) represents the communication channel gain value of the kth interfered device at time t, L k Indicates the transmission loss of the interfered device during its own communication process, γ k Indicates the interference suppression coefficient set by the interference device on the interference power value of the interfered device, p max It represents the maximum interference power value emitted by the interference device, and H represents the maximum number of interference communication frequencies that a interference device can interfere with at the same time.

[0079] It should be noted that, following the above embodiment, since the power status of the interfering device is updated periodically, based on this, the target problem function set is updated in real time, that is,

[0080] S22: For the target problem function set, a multi-agent hybrid proximal strategy is used to generate a model through interference strategy to perform model training and solution.

[0081] Specifically, after constructing the target problem function set, the multi-agent hybrid proximal strategy is used to generate a model through the interference strategy to train and solve the target problem function set, and obtain the result corresponding to the target problem function set, namely the target interference strategy.

[0082] Optionally, based on the above embodiment, in some embodiments of the present invention, an implementation of S22 may be:

[0083] S221: Initialize the environmental state parameters of the target area.

[0084] Specifically, the environmental status parameters of the target area, such as the historical execution actions of the interfering device, that is, the interfered communication frequency parameters of at least one interfered communication frequency that the interfering device interferes with among multiple communication frequencies, as well as the interference power value of the interfered communication frequency, the current power status of the interfering device, and the interference-to-signal ratio of the interfered device are initialized.

[0085] Optionally, based on the above embodiments, in some embodiments of the present invention, an implementation method of S221 may be: from the historical environmental state parameters of multiple communication frequency points of multiple interference devices in the target area at multiple times, obtain the environmental state parameters at the most recent moment to initialize the environmental state parameters of the target area.

[0086] Optionally, based on the above embodiments, in some embodiments of the present invention, another implementation method of S221 may be: randomly obtaining the environmental state parameters at any moment from the historical environmental state parameters at multiple moments to initialize the environmental state parameters of the target area.

[0087] Optionally, based on the above embodiments, in some embodiments of the present invention, another implementation method of S221 may be: calculating the mean of historical environmental state parameters at multiple moments, and using the mean to initialize the environmental state parameters of the target area.

[0088] S222: According to the environmental state parameters, the multi-agent hybrid proximal strategy is used to train the initial interference strategy generation model based on the training loss function to obtain a trained interference strategy generation model.

[0089] The training loss function refers to the function set when training the interference strategy generation model.

[0090] Optionally, based on the above embodiment, in some embodiments of the present invention, a way to set the training loss function may be:

[0091] For the initial interference strategy generation model, the training loss function is obtained by weighting the value loss function, strategy loss function, and entropy regularization term of the value network.

[0092] The strategy loss function is related to the parameters of the interfered communication frequency point and the interference power value of the interfered communication frequency point. For example, the strategy loss function can be defined by the following expression:

[0093] L 1 (θ)=L 11 +L 12

[0094] Among them, L 11 It represents the loss function for the interfered communication frequency parameters set in the process of training the interference strategy generation model, L 12 Represents the loss function for the interference power value set during the training process of the interference strategy generation model.

[0095] Optionally, based on the above embodiments, in some embodiments of the present application, the training loss function can be defined by the following expression:

[0096]

[0097] wherein L 1 (θ) represents the policy loss function, represents the value loss function of the value network, X[π θ (S t ) represents the entropy regularization term, S t represents the environment space parameter, θ and represents the network parameter of the interference policy generation model, a and b represent weighting factors, and the present application does not specifically limit the values of a and b, and the person skilled in the art can set them according to the actual situation.

[0098] In this way, the interference resource scheduling method of the air-ground integrated network provided by the present application can ensure the collaborative optimization between the value network and the policy network by weighting the value loss function of the critic network, the policy loss function related to the parameters of the interfered communication frequency and the interference power value of the interfered communication frequency, and the entropy regularization term to obtain the training loss function.

[0099] Specifically, after the initialization processing of the environment state parameter of the target area, the initialized environment state parameter is input into the initial interference policy generation model, and the initialized environment state parameter is used to utilize the multi-agent hybrid near-end policy and train the initial interference policy generation model based on the training loss function, so as to obtain the trained interference policy generation model.

[0100] In this way, the interference resource scheduling method of the air-ground integrated network provided by the present application can utilize the multi-agent hybrid near-end policy to allow multiple interference devices to exchange information and share experience, has the characteristics of accelerating training and improving policy updating, so as to improve the efficiency of obtaining the target interference policy through the interference policy generation model. Further, the efficiency of scheduling the interference devices to interfere with the interfered communication frequency through the interference power value according to the target interference policy is improved.

[0101] Optionally, based on the above embodiments, in some embodiments of the present application, one implementation of S222 can be:

[0102] The environmental state parameter is input to the initial interference strategy generation model, an intermediate execution action is obtained through a policy network included in the initial interference strategy generation model, a jam-to-signal ratio value corresponding to the multiple interference devices is determined according to the intermediate execution action, a reward value of the intermediate execution action is calculated through a reward function set by the value network, and the intermediate execution action, the reward value, and the jam-to-signal ratio value are returned and input to the initial interference strategy generation model. Training is performed based on a training loss function until the initial interference strategy generation model converges, and a trained interference strategy generation model is obtained.

[0103] The intermediate execution action refers to an execution action obtained through the policy network in the process of training the initial interference strategy generation model. The execution action is a jam-to-signal ratio value corresponding to the environmental state parameter and the interference power value of the jammed communication frequency point.

[0104] Specifically, the environmental state parameter after the initialization processing is input to the initial interference strategy generation model, the policy network included in the initial interference strategy generation model outputs an intermediate execution action for the environmental state parameter, and a reward function is set for the value network, so that the value network calculates and scores the intermediate execution action according to the reward function to obtain a corresponding reward value. When the intermediate execution action is obtained, a jam-to-signal ratio value of the jammed device at the current time is calculated by using a calculation function for calculating the jam-to-signal ratio value. In the case where the initial interference strategy generation model converges, the intermediate execution action, the reward value, and the jam-to-signal ratio value are fed back as global environment observation variables and input to the untrained interference strategy generation model. Training is performed based on a training loss function, the network parameters of the policy network and the value network are adjusted, and the trained interference strategy generation model is obtained until the initial interference strategy generation model converges.

[0105] It should be understood that, although Figure 1 the steps in the flowchart of the method are shown in sequence according to the arrows, these steps are not necessarily executed in sequence according to the arrows. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other sequences. Moreover, Figure 1 at least part of the steps in the method can include multiple sub-steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution sequence of the sub-steps or stages is not necessarily sequential, but can be executed in rotation or alternation with at least part of other steps or sub-steps or stages of other steps.

[0106] In one embodiment, as shown in Figure 4 , an interference resource scheduling device for a space-ground integrated network is provided, which includes an environmental state parameter acquisition module 10, a target interference strategy acquisition module 11, and a scheduling module 12.

[0107] The environment state parameter acquisition module 10 is configured to acquire environment state parameters of a target area, wherein the target area includes a plurality of interfering devices, a plurality of interfered devices, and a plurality of communication frequency points corresponding to the interfered devices, and the environment state parameters at least include historical execution actions of the interfering devices, a power state of the interfering devices, and a jamming-to-signal ratio of the interfered devices. The historical execution actions include an interfered communication frequency point parameter and an interference power value of the interfered communication frequency point, and the interfered communication frequency point parameter is used to indicate at least one interfered communication frequency point in which the interfering devices interfere with the plurality of communication frequency points.

[0108] The target interference strategy acquisition module 11 is configured to input the environment state parameters into a trained interference strategy generation model to acquire a target interference strategy. The interference strategy generation model is obtained by training a multi-agent hybrid near policy, and the interference strategy generation model is composed of a policy network and a value network. The policy network includes an input layer, a plurality of hidden layers, and an output layer. The output layer includes a first output sub-layer used to output the interfered communication frequency point parameter and a second output sub-layer used to output the interference power value of the interfered communication frequency point.

[0109] The scheduling module 12 is configured to schedule the interfering devices to interfere with the interfered communication frequency points by the interference power values of the interfered communication frequency points according to the target interference strategy for the target area.

[0110] In the above embodiment, for the plurality of interfering devices, the plurality of interfered devices, and the plurality of communication frequency points corresponding to the interfered devices in the target area, the corresponding environment state parameters are acquired by the environment state parameter acquisition module, such as the historical execution actions of the interfering devices, i.e., the interfered communication frequency point parameter used to indicate at least one interfered communication frequency point in which the interfering devices interfere with the plurality of communication frequency points, the interference power value of the interfered communication frequency point, the power state of the interfering devices at the current time, and the jamming-to-signal ratio of the interfered devices. The environment state parameters acquired are input into the interference strategy generation model trained by the multi-agent hybrid near policy by the target interference strategy acquisition module to acquire the target interference strategy. Since the multi-agent hybrid near policy allows information exchange and experience sharing among the plurality of interfering devices, the multi-agent hybrid near policy has the characteristics of accelerating training and improving strategy updating. Moreover, the interference strategy generation model can simultaneously train the interfered communication frequency point parameter and the interference power value, which can reduce the training complexity and accelerate the convergence speed of the interference strategy generation model. Therefore, the efficiency of acquiring the target interference strategy is improved, and further, the efficiency of scheduling the interfering devices to interfere with the interfered communication frequency points by the interference power values according to the target interference strategy by the scheduling module is improved.

[0111] The specific limitation of the interference resource scheduling apparatus of the air-ground integrated network can refer to the limitation of the interference resource scheduling method of the air-ground integrated network, which will not be repeated here. Each module in the server can be realized by software, hardware and their combination. The above-mentioned modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory in the computer device in the form of software, so that the processor calls and executes the operation corresponding to each module.

[0112] The embodiment of the application provides an electronic device, comprising: a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the interference resource scheduling method of the air-ground integrated network provided by the embodiment of the application can be realized, for example, when the processor executes the computer program, the interference resource scheduling method of the air-ground integrated network provided by the embodiment of the application can be realized. Figure 1 The technical solutions of any of the method embodiments have similar implementation principles and technical effects, which will not be repeated here.

[0113] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by a computer program instructing related hardware, and the computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments. In the embodiments of the present application, any reference to memory, database or other medium can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (Read-Only Memory, ROM), magnetic tape, floppy disk, flash memory or optical memory. Volatile memory can include random access memory (Random Access Memory, RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static random access memory (Static Random Access Memory, SRAM) and dynamic random access memory (Dynamic Random Access Memory, DRAM).

[0114] Each technical feature of the above embodiments can be combined arbitrarily. In order to make the description simple, all possible combinations of each technical feature in the above embodiments are not described, but as long as the combination of these technical features does not exist, it should be considered as the scope of the present application.

[0115] The above embodiments only express several implementation manners of the present application, and the description is relatively specific and detailed, but should not be understood as a limitation on the patent scope of the present application. It should be noted that, for ordinary skilled persons in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application should be subject to the appended claims.

Claims

1. A method for interference resource scheduling of an air-ground integrated network, characterized in that, The method comprises: acquiring an environment state parameter of a target area, wherein the target area comprises a plurality of interference devices, a plurality of interfered devices, and a plurality of communication frequency points corresponding to the interfered devices, and the environment state parameter at least comprises a historical execution action of the interference device, a power state, and an interference-to-signal ratio value of the interfered device, wherein the historical execution action comprises an interfered communication frequency point parameter and an interference power value of the interfered communication frequency point, and the interfered communication frequency point parameter is used to indicate at least one interfered communication frequency point in which the interference device interferes in the plurality of communication frequency points; inputting the environment state parameter into a trained interference strategy generation model to acquire a target interference strategy, wherein the interference strategy generation model is obtained by training a multi-agent hybrid near-end strategy, and the interference strategy generation model comprises a policy network and a value network, the policy network comprises an input layer, a plurality of hidden layers, and an output layer, and the output layer comprises a first output sublayer used to output the interfered communication frequency point parameter and a second output sublayer used to output the interference power value of the interfered communication frequency point; for the target area, the interference device is scheduled to interfere in the interfered communication frequency point through the interference power value of the interfered communication frequency point according to the target interference strategy.

2. The method of claim 1, wherein, Before the environment state parameter is input into the trained interference strategy generation model to acquire the target interference strategy, the method further comprises: establishing a target problem function set for an interference resource scheduling problem of the plurality of interference devices, the plurality of interfered devices, and the plurality of communication frequency points corresponding to the interfered devices in the target area; solving the model training through the interference strategy generation model by using the multi-agent hybrid near-end strategy for the target problem function set.

3. The method of claim 2, wherein, The target problem function set can be defined by the following expression: X nm (t) e {0,1}, n e N, m e M Wherein, M represents the number of the corresponding interfered communication frequency points of the plurality of interfered devices, N represents the number of the interference devices, K represents the number of the interfered devices, p nm (t) represents the interference power value of the nth interference device to the mth interfered communication frequency point at t time, X nm (t) represents the interfered communication frequency point parameter of the nth interference device to the mth interfered communication frequency point at t time, G nk represents the gain value of the nth interference device to the kth interfered device, L nk represents the transmission loss of the nth interference device to the kth interfered device, σ 2 represents the noise power value, p k (t) represents the communication power value of the kth interfered device to itself at t time, G k (t) represents the communication channel gain value of the kth interfered device at t time, L k represents the transmission loss in the communication process of the interfered device itself, γ k represents the interference suppression coefficient set by the interference power value of the interference device to the interfered device, p max represents the maximum interference power value transmitted by the interference device, H represents the maximum number of interference communication frequency points that can be interfered by one interference device at the same time.

4. The method of claim 2, wherein, The model training through the interference strategy generation model by using the multi-agent hybrid near-end strategy comprises: initializing the environment state parameter of the target area; training an initial interference strategy generation model based on a training loss function by using the multi-agent hybrid near-end strategy according to the environment state parameter to obtain the trained interference strategy generation model.

5. The method of claim 4, wherein, Before the initial interference strategy generation model is trained based on the training loss function by using the multi-agent hybrid near-end strategy according to the environment state parameter to obtain the trained interference strategy generation model, the method further comprises: weighting a value loss function of the value network, a policy loss function, and an entropy regularization term to obtain the training loss function for the initial interference strategy generation model, wherein the policy loss function is related to the interfered communication frequency point parameter and the interference power value of the interfered communication frequency point.

6. The method of claim 5, wherein, The weighting of the value loss function of the value network, the policy loss function, and the entropy regularization term to obtain the training loss function comprises: According to the formula obtaining the training loss function; where L 1 (θ) denotes the policy loss function, denotes the value loss function of the value network, X[π θ (S t ) denotes the entropy regularization term, S t denotes the environment space parameter, θ and denotes the network parameter of the interference policy generation model, a and b denote the weighting factors.

7. The method of claim 2, wherein, The training of the initial interference strategy generation model based on the training loss function by using the multi-agent hybrid near-end strategy according to the environment state parameter to obtain the trained interference strategy generation model comprises: The environment state parameter is input to the initial interference strategy generation model, an intermediate execution action is obtained through the policy network included in the initial interference strategy generation model, a jam-to-signal ratio value corresponding to each of the interference devices is determined according to the intermediate execution action, a reward value of the intermediate execution action is calculated through a reward function set by the value network, the intermediate execution action, the reward value, and the jam-to-signal ratio value are returned and input to the initial interference strategy generation model, training is performed based on the training loss function until the initial interference strategy generation model converges, and a trained interference strategy generation model is obtained.

8. The method of claim 1, wherein, The method further includes: Periodically updating the power state of the interference device.

9. An interference resource scheduling apparatus of a space-air integrated network, characterized in that, Comprise: An environment state parameter acquisition module configured to acquire environment state parameters of a target region, wherein the target region includes a plurality of interference devices, a plurality of interfered devices, and a plurality of communication frequency points corresponding to the interfered devices, and the environment state parameters at least include historical execution actions of the interference devices, power states, jam-to-signal ratio values of the interfered devices, and the historical execution actions include interfered communication frequency point parameters and interference power values of the interfered communication frequency points, the interfered communication frequency point parameters are used to indicate at least one interfered communication frequency point in which the interference devices perform interference among the plurality of communication frequency points; A target interference strategy acquisition module configured to input the environment state parameters to a trained interference strategy generation model to acquire a target interference strategy, wherein the interference strategy generation model is obtained by training a multi-agent hybrid near-end policy, the interference strategy generation model is composed of a policy network and a value network, the policy network includes an input layer, a plurality of hidden layers, and an output layer, and the output layer includes a first output sublayer used to output the interfered communication frequency point parameters and a second output sublayer used to output the interference power values; A scheduling module configured to, for the target region, schedule the interference devices to interfere with the interfered communication frequency points through the interference power values of the interfered communication frequency points according to the target interference strategy.

10. An electronic device comprising a memory and a processor, the memory storing a computer program, characterized in that, The processor executes the computer program to implement the steps of the interference resource scheduling method of the air-ground integrated network in any one of claims 1 to 8. The processor executes the computer program to implement the steps of the interference resource scheduling method of the air-ground integrated network in any one of claims 1 to 8.

Images