Dynamic beam forming method for spectrum sharing of air network and ground network
Through deep reinforcement learning algorithm training strategy network model, the problem of high beamforming complexity of ground base stations is solved, and rapid and accurate beamforming is achieved in the spectrum sharing environment between air network and ground network, reducing interference to aircraft and improving efficiency.
Patent Information
- Application Number
- CN202410951785.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-16
- Publication Date
- 2024-11-12
AI Technical Summary
The prior art has high complexity when determining the beamforming vector, which makes it difficult to effectively reduce the interference of ground base stations to aircraft, and requires real-time global channel state information, large data volume and low efficiency.
Deep reinforcement learning algorithm is adopted to obtain the optimal beamforming vector through initialization policy network model training, which reduces the complexity of beamforming vectors of ground base stations, improves efficiency, avoids manual participation errors, and reduces communication overhead.
It realizes that the beamforming vectors can be quickly and accurately obtained in the spectrum sharing environment between the air network and the ground network, reduces the interference of the ground base station to the aircraft, and improves the accuracy and efficiency of beamforming.
Smart Images

Figure CN118944724A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of communication technology, and in particular to a dynamic beamforming method for spectrum sharing between airborne networks and ground networks. Background Art
[0002] With the development of wireless communication technology, Air to Ground (ATG) broadband communication network technology has been widely used in the aviation field to provide aviation Internet services. In the same communication environment, the spectrum used by the ground base station corresponding to the ground network and the dedicated base station corresponding to the air network may be the same, which will cause a large number of ground base stations to cause serious interference to the air equipment in the aircraft during flight.
[0003] In the related technology, an iterative optimization algorithm is mainly used to obtain the optimal beamforming vector of each ground base station. Then each ground base station sends a signal to the ground device based on its corresponding optimal beamforming vector to reduce the interference of the ground base station to the aerial device.
[0004] However, in the related art, there is a problem of high complexity when determining the beamforming vector. Summary of the invention
[0005] Based on this, it is necessary to provide a dynamic beamforming method for spectrum sharing between air networks and ground networks to address the above technical problems, which can reduce the complexity of determining beamforming vectors.
[0006] In a first aspect, an embodiment of the present application provides a dynamic beamforming method for spectrum sharing between an air network and a ground network, the method comprising:
[0007] S1. Input the state of at least one ground base station in multiple time slots within the target range into the initialization strategy network model, obtain the actions of each ground base station in each time slot, and execute the actions of each ground base station in each time slot to obtain the reward and cost of each ground base station in each time slot; the target range is the area covered by the air network and the ground network;
[0008] S2. Determine the reward advantage function of each time slot according to the state of each ground base station in each time slot, the state of the next time slot of each time slot, the reward of each time slot and the reward advantage function of each next time slot; and determine the target of the reward value of each time slot according to the reward advantage function of each time slot, the reward value network model of each time slot and the state of each time slot;
[0009] S3, determining the cost advantage function of each time slot according to the state of each ground base station in each time slot, the state of the next time slot of each time slot, the cost of each time slot and the cost advantage function of each next time slot; and determining the cost value target of each time slot according to the cost advantage function of each time slot, the cost value network of each time slot and the state of each time slot;
[0010] S4. Obtain multiple training data sets according to the state, action, reward advantage function, cost advantage function, reward value target and cost value target of each ground base station in each time slot, and train the initialization strategy network model corresponding to each ground base station according to each training data set until the training convergence condition is reached, and obtain the optimal strategy network model of each ground base station;
[0011] S5. Input the state of each ground base station in the current time slot into each optimal strategy network model to obtain the optimal action of each ground base station in the current time slot; the optimal action is obtained with the goal of maximizing the expected value of the cumulative discount reward and the expected value of the cumulative discount cost to be less than or equal to the first preset threshold;
[0012] S6. Determine the beamforming vector of each ground base station in the current time slot according to the optimal action of each ground base station in the current time slot.
[0013] In one embodiment, the action a of the ground base station n at time slot t′ n (t′) is expressed as:
[0014]
[0015] Among them, 1,...,,..., respectively represent the numbers of each ground cell within the target range; 1,...,,..., respectively represent the numbers of each aerial device within the target range; 1,...,,..., respectively represent the numbers of each ground device within the target range; represents the ratio of the total transmission power of the ground base station n to the maximum transmission power at time slot t′; q n,k (t′) represents the ratio of the transmission power allocated by the ground base station n to the ground device k in the cell n at time slot t′ to the total transmission power of the ground base station n; α n,j,i (t′) represents the relevant parameters of the channel vector from the ground base station n to the ground device i in the corresponding cell j at time slot t′; μ n,l (t′) represents the relevant parameters of the channel vector of the ground base station n for the air device l at time slot t′; η n (t′) represents the noise related parameters of ground base station n at time slot t′.
[0016] In one embodiment, the reward r of the ground base station n at time slot t′ is n (t′) is expressed as:
[0017]
[0018] Among them, R n,k (t′) represents the rate of ground equipment k in cell n where ground base station n is located at time slot t′; represents the rate loss caused by the interference of ground base station n at time slot t′; P j,i (t′) represents the achievable rate of the ground device i in the cell j when the ground base station n does not exist minus the achievable rate of the ground device i in the cell j when the ground base station n exists at the time slot t′; represents the set of interfered ground devices of ground base station n at time slot t′;
[0019] represents the expected signal power received by ground device i in cell j at time slot t′; β n,j,i (t′) represents the interference power received by ground equipment i in cell j from ground base station n at time slot t′; β j,i (t′) represents the interference power plus noise power received by ground equipment i in cell j at time slot t′.
[0020] In one embodiment, the cost c of the ground base station n in time slot t′ n (t′) is expressed as:
[0021] c n (t′) = c n,1 (t′),..., n,l (t′),..., n,L (t′)]
[0022]
[0023] Among them, c n,l (t′) represents the cost of ground base station n with respect to aerial device l at time slot t′; ρ l (t′) represents the interference power received by the air device l from all ground base stations at time slot t′; I max Indicates the maximum receiving interference power threshold that the air device l can tolerate.
[0024] In one embodiment, the step of determining the reward advantage function of each time slot in S2 according to the state of each ground base station in each time slot, the state of the next time slot of each time slot, the reward of each time slot and the reward advantage function of each next time slot includes:
[0025] S21. For any time slot, the state of each ground base station in the time slot and the state of the next time slot of the time slot are input into the corresponding reward value network model to obtain the reward value of the time slot and the reward value of the next time slot; the reward value network model is obtained by training the data corresponding to the time slot in each training data set;
[0026] The state s of ground base station n at time slot t′ n (t′) is:
[0027]
[0028] represents the local information of ground base station n at time slot t′; Indicates the information of the ground base station that interferes with each ground device in the cell n where the ground base station n is located at time slot t′; in,pri (t′) represents the information of the interferer ground base station n of each aerial device at time slot t′; Indicates the information of each interfered ground device in cell n where ground base station n is located at time slot t′; Information indicating the interference caused by ground base station n to each aerial device at time slot t′;
[0029] in,
[0030] represents the compressed version of the downlink channel estimated by the ground base station n to the ground device k in the cell n where it is located at time slot t′; p n,k (t′-1) represents the transmission power of the ground base station n to the corresponding ground device k at the previous time slot (t′-1); R n,k (t′-1) represents the rate of ground equipment k in cell n where ground base station n is located at the previous time slot (t′-1); represents the expected signal power received by the ground device k in the cell n where the ground device is located at the previous time slot (t′-1); β n,k (t′-1) represents the interference power plus noise power received by the ground device k in the cell n where the ground device n is located at the previous time slot (t′-1); represents the compressed version of the interference channel estimated by the ground base station n to the air device l at time slot t′; ρ n, (t′-1) The interference power received by space device l from ground base station n in the previous time slot (t′-1);
[0031] Indicates from Information of interfering ground base stations; represents the set of interfering ground base stations of ground device k in cell n where ground base station n is located at the previous time slot (t′-1); It is a guarantee The threshold value of
[0032]
[0033] Indicates that at time slot t′, The information of interfering ground base stations in represents the set of interfering ground base stations of device l in space at the previous time slot (t′-1); It is a guarantee The threshold value of
[0034]
[0035] represents the set of interfering ground base stations of ground base station n at the previous time slot (t′-1), It is a guarantee The threshold value of
[0036] ρ l (t′-1) represents the interference power received by space device l from all ground base stations in the previous time slot (t′-1);
[0037] S22, obtaining a reward advantage function of a time slot according to the reward value of the time slot, the reward value of the next time slot, and the reward advantage function of the next time slot;
[0038] The reward advantage function in the mth training data in the training dataset for:
[0039]
[0040] Wherein, m represents the index of each training data in the training data set corresponding to time slot t′; represents the reward advantage function in the (m+1)th training data in the training data set; δ m represents the temporal difference error in the mth training data in the training data set; r m Represents the reward for the (m+1)th training data in the training dataset; Represents the current reward value network model; Represents the reward value of the mth training data in the training data set; represents the reward value of the (m+1)th training data in the training data set; BM represents the number of data in the training data set; γ is the discount factor; λ is the generalized advantage estimation parameter;
[0041] S22, obtaining a reward advantage function of a time slot according to the reward value of the time slot, the reward value of the next time slot, and the reward advantage function of the next time slot;
[0042] The reward advantage function in the mth training data in the training dataset for:
[0043]
[0044] Wherein, m represents the index of each training data in the training data set corresponding to time slot t′; represents the reward advantage function in the (m+1)th training data in the training data set; δ m represents the temporal difference error in the mth training data in the training data set; r m Represents the reward for the (m+1)th training data in the training dataset; Represents the current reward value network model; Represents the reward value of the mth training data in the training data set; represents the reward value of the (m+1)th training data in the training data set; B M represents the number of data in the training data set; γ is the discount factor; λ is the generalized advantage estimation parameter.
[0045] In one embodiment, the step of determining the target of the reward value of each time slot in S2 according to the reward advantage function of each time slot, the reward value network model of each time slot and the state of each time slot includes:
[0046] S23. For any time slot, the reward value of the time slot is obtained according to the state of the time slot and the initialized reward value network model;
[0047] S24, taking the sum of the reward value of the time slot and the reward advantage function of the time slot as the target of the reward value of the time slot;
[0048] The target of the reward value in the mth training data for:
[0049]
[0050] In one embodiment, the step of determining the cost advantage function of each time slot in S3 according to the state of each ground base station in each time slot, the state of the next time slot of each time slot, the cost of each time slot and the cost advantage function of each next time slot includes:
[0051] S31. For any time slot, the state of each ground base station in the time slot and the state of the next time slot of the time slot are input into the corresponding cost value network model to obtain the cost value of the time slot and the cost value of the next time slot; the cost value network model is obtained by training the data corresponding to the time slot in the training data set;
[0052] S32, according to the cost value of the time slot, the cost value of the next time slot and the cost advantage function of the next time slot, obtain the cost advantage function of the time slot; the cost advantage function in the mth training data for:
[0053]
[0054] in, represents the cost advantage function of the (m+1)th training data in the training data set; represents the temporal difference error in the mth training data in the training data set; c m Represents the cost in the mth training data; Represents the current cost-value network model; Represents the cost value in the mth training data; represents the cost value in the (m+1)th training data; γ represents the discount factor; λ represents the generalized advantage estimation parameter.
[0055] In one embodiment, the step of determining the target of the cost value of each time slot in S3 according to the cost advantage function of each time slot, the cost value network of each time slot and the state of each time slot includes:
[0056] S33, for any time slot, according to the state of the time slot and the initialization cost value network model, obtain the cost value of the time slot;
[0057] S34, taking the sum of the cost value of the time slot and the cost advantage function of the time slot as the target of the cost value of the time slot;
[0058] The target of the cost value in the mth training data for:
[0059]
[0060] In one embodiment, the step of training the initialization strategy network model corresponding to each ground base station according to each training data set in S4 until the training convergence condition is reached to obtain the optimal strategy network model of each ground base station includes:
[0061] S41, updating the cost parameter vector according to the average cost of the ground base station in each time slot and the strategy network learning rate to obtain an updated cost parameter vector;
[0062] The updated cost parameter vector includes:
[0063]
[0064] Among them, α v represents the learning rate of the cost parameter vector; v represents the cost parameter vector; represents the average cost in the training dataset, b represents the threshold vector of discounted cumulative cost; ν max represents the maximum value of the cost parameter vector; Indicates that Clipped to the interval [0,v max ];
[0065] S42, obtain the strategy network model trained by the previous training data set: Among them, let the initialization strategy network model also be
[0066] S43. For any ground base station, multiple sets of training are required; in each set of training, multiple sets of sub-data sets are obtained according to a training data set containing training data of multiple time slots corresponding to the ground base station; each set of sub-data sets includes training data of multiple time slots;
[0067] S44, training the initialization strategy network model of the ground base station according to each sub-data set in the current group to obtain the latest strategy network parameters;
[0068] S45, updating the current policy network parameters to the latest policy network parameters to obtain the current policy network model;
[0069] S46, using the current policy network model as the initialization policy network model, and executing the steps in S44-S45 in sequence according to each sub-data set in other groups, until the latest current policy network model meets the training convergence condition or reaches the preset maximum number of training times, and using the latest current policy network model as the optimal policy network model of the ground base station;
[0070] The training convergence conditions include:
[0071]
[0072] in, Represents the KL divergence between the current policy network model and the initialization policy network model; s m represents the state vector in the mth training data in the training data set; ε represents the second preset threshold.
[0073] In one embodiment, the step of training the initialization strategy network model of the ground base station according to each sub-data set in the current group to obtain the latest strategy network parameters in S44 includes:
[0074] S441, determining the update parameter value of the current policy network model according to the state, action, reward advantage function, cost advantage function, parameter vector of the updated cost in each sub-data set and the number of sub-data sets in the current group;
[0075] Update parameter values are expressed as:
[0076]
[0077] Among them, B S Indicates the data length of the sub-dataset; Represents the current policy network parameters; Represents the loss function of the current policy network model; express about The gradient of Represents the current policy network model; Represents the policy network model trained by the previous training data set; Represents the reward advantage function in the bth data of each sub-data set in any group; represents the cost advantage function of the bth data in each sub-data set in any group; a b represents the action in the bth data of each sub-data set in any group; s b Indicates the state of the bth data in each sub-data set in any group;
[0078] S442, obtaining the latest policy network parameters according to the current policy network parameters, policy network learning rate and update parameter values of the initialized policy network model;
[0079] Among them, the latest strategy network parameters are:
[0080] In a second aspect, an embodiment of the present application provides a dynamic beamforming device for spectrum sharing between an air network and a ground network, the device comprising:
[0081] The first acquisition module is used to input the state of at least one ground base station in a target range in multiple time slots into the initialization strategy network model, obtain the action of each ground base station in each time slot, and execute the action of each ground base station in each time slot to obtain the reward and cost of each ground base station in each time slot; the target range is the area covered by the air network and the ground network;
[0082] The first determination module is used to determine the reward advantage function of each time slot according to the state of each ground base station in each time slot, the state of the next time slot of each time slot, the reward of each time slot and the reward advantage function of each next time slot; and determine the target of the reward value of each time slot according to the reward advantage function of each time slot, the reward value network model of each time slot and the state of each time slot;
[0083] The second determination module is used to determine the cost advantage function of each time slot according to the state of each ground base station in each time slot, the state of the next time slot of each time slot, the cost of each time slot and the cost advantage function of each next time slot; and determine the cost value target of each time slot according to the cost advantage function of each time slot, the cost value network of each time slot and the state of each time slot;
[0084] A training module is used to obtain multiple training data sets according to the state, action, reward advantage function, cost advantage function, reward value target and cost value target of each ground base station in each time slot, and train the initialization strategy network model corresponding to each ground base station according to each training data set until the training convergence condition is reached, thereby obtaining the optimal strategy network model of each ground base station;
[0085] The second acquisition module is used to input the state of each ground base station in the current time slot into each optimal strategy network model to obtain the optimal action of each ground base station in the current time slot; the optimal action is obtained with the goal of maximizing the expected value of the cumulative discount reward and the expected value of the cumulative discount cost to be less than or equal to the first preset threshold;
[0086] The solution module is used to determine the beamforming vector of each ground base station in the current time slot according to the optimal action of each ground base station in the current time slot.
[0087] In a third aspect, an embodiment of the present application further provides a communication device, comprising a transceiver, a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the method of any embodiment in the first aspect are implemented.
[0088] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the method of any embodiment in the first aspect are implemented.
[0089] In a fifth aspect, an embodiment of the present application further provides a computer program product, the computer program product comprising a computer program, which, when executed by a processor, implements the steps of the method of any embodiment in the first aspect.
[0090] The dynamic beamforming method for spectrum sharing between airborne networks and ground networks provided in an embodiment of the present application includes: inputting the state of at least one ground base station in a target range in multiple time slots into an initialization strategy network model, obtaining the action of each ground base station in each time slot, and executing the action of each ground base station in each time slot to obtain the reward and cost of each ground base station in each time slot, the target range is the area jointly covered by the airborne network and the ground network, and determining the reward advantage function of each time slot according to the state of each ground base station in each time slot, the state of the next time slot of each time slot, the reward of each time slot and the reward advantage function of each next time slot; and determining the target of the reward value of each time slot according to the reward advantage function of each time slot, the reward value network model of each time slot and the state of each time slot, and determining the reward cost of each time slot according to the state of each ground base station in each time slot, the state of the next time slot of each time slot, the cost of each time slot and the cost advantage function of each next time slot. The invention relates to a cost advantage function of each time slot; and according to the cost advantage function of each time slot, the cost value network of each time slot and the state of each time slot, the cost value target of each time slot is determined; according to the state, action, reward advantage function, cost advantage function, reward value target and cost value target of each ground base station in each time slot, a plurality of training data sets are obtained; and according to each training data set, the initialization strategy network model corresponding to each ground base station is trained until the training convergence condition is reached, and the optimal strategy network model of each ground base station is obtained; the state of each ground base station in the current time slot is input into each optimal strategy network model, and the optimal action of each ground base station in the current time slot is obtained; the optimal action is obtained with the goal of maximizing the expected value of the cumulative discounted reward and the expected value of the cumulative discounted cost less than or equal to the first preset threshold value; according to the optimal action of each ground base station in the current time slot, the beamforming vector of each ground base station in the current time slot is determined. The above method can be used to obtain the beamforming vectors of each ground base station in the current time slot within the target range jointly covered by the air network and the ground network through a set of deep reinforcement learning algorithms. Compared with the real-time global channel state information required when using an iterative optimization algorithm in the traditional technology to obtain the beamforming vectors of each ground base station within the target range, the required amount of data is less, thereby reducing the complexity of obtaining the beamforming vectors of each ground base station and improving the speed and efficiency of obtaining the beamforming vectors of each ground base station; at the same time, the method does not require manual participation and can avoid manual participation errors, thereby improving the accuracy of the beamforming vectors of each ground base station obtained; in addition, the above method avoids the process of adjusting the penalty coefficient through multiple trial and error, thereby reducing the communication overhead required for obtaining the beamforming vectors of each ground base station and reducing the cost; furthermore, the above method obtains the optimal action of each ground base station with the goal of maximizing the expected value of the cumulative discounted reward and the expected value of the cumulative discounted cost less than or equal to the first preset threshold value when each ground base station violates the interference power constraint a little, and on this basis, the accuracy of the beamforming vectors of each ground base station finally obtained is higher. BRIEF DESCRIPTION OF THE DRAWINGS
[0091] Figure 1 FIG. 1 is an application environment diagram of a dynamic beamforming method for spectrum sharing between an airborne network and a ground network in one embodiment;
[0092] Figure 2 A schematic diagram of a flow chart of a dynamic beamforming method for spectrum sharing between an airborne network and a ground network in one embodiment;
[0093] Figure 3 It is a flowchart of a dynamic beamforming method for spectrum sharing between an air network and a ground network in another embodiment;
[0094] Figure 4 It is a flowchart of a dynamic beamforming method for spectrum sharing between an air network and a ground network in another embodiment;
[0095] Figure 5 It is a flowchart of a dynamic beamforming method for spectrum sharing between an air network and a ground network in another embodiment;
[0096] Figure 6 It is a flowchart of a dynamic beamforming method for spectrum sharing between an air network and a ground network in another embodiment;
[0097] Figure 7 It is a flowchart of a dynamic beamforming method for spectrum sharing between an air network and a ground network in another embodiment;
[0098] Figure 8 It is a flowchart of a dynamic beamforming method for spectrum sharing between an air network and a ground network in another embodiment;
[0099] Fig. 9 It is a structural block diagram of a dynamic beamforming device for spectrum sharing between airborne networks and ground networks in one embodiment;
[0100] Fig.10 FIG. 4 is a diagram showing the internal structure of a communication device in one embodiment. DETAILED DESCRIPTION
[0101] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0102] In the field of communications, ground-to-air broadband communication network communication technology is widely used in the aviation field to provide aviation Internet services. In the same communication environment, the spectrum used by the ground base station corresponding to the ground network and the dedicated base station corresponding to the air network may be the same, which will cause a large number of ground base stations to cause serious interference to the air equipment when the aircraft is flying. In the related art, an iterative optimization algorithm is mainly used to obtain the optimal beamforming vector of each ground base station, and then each ground base station sends a signal to the ground equipment based on its corresponding optimal beamforming vector to reduce the interference of the ground base station to the air equipment. However, in the related art, there is a problem of high complexity when determining the beamforming vector. Based on this, an embodiment of the present application provides a dynamic beamforming method for spectrum sharing between air networks and ground networks, which can reduce the complexity of determining beamforming vectors.
[0103] The dynamic beamforming method for spectrum sharing between airborne networks and ground networks provided in the embodiment of the present application can be applied to a communication device, which can be any ground base station within the target range, or other ground base stations outside the target range. In the embodiment of the present application, the airborne network within the target range is the primary network, and the ground network is the secondary network, such as Figure 1 The figure shows a schematic diagram of the aerial network and the ground network within the target range, wherein the embodiment of the present application is described by taking the L single-antenna aerial devices (i.e., aerial devices) and N cells located on an aircraft within the target range as an example. A ground base station is set in each cell, and the ground base station in the cell transmits signals to the K single-antenna ground devices (i.e., ground devices) in the cell. In the embodiment of the present application, each ground base station has a uniform rectangular array composed of M antennas. The index set of all aerial devices within the target range is {1,...,,...,}, and the index set of base stations of all ground networks is
[0104] {1,...,,...,}, the index set of all ground users is {1,...,,...,}×{1,...,,...,}. For simplicity, Figure 1 In the figure, an aircraft is taken as an aerial device, and includes 3 aerial devices, 3 ground base stations, and each ground base station is located in a cell including two ground devices.
[0105] Among them, different ground base stations within the target range are set at intervals, and the interval distances can be equal or unequal; the communication coverage ranges of different ground base stations can be non-overlapping or overlapping. Among them, the above-mentioned ground base stations can be but not limited to macro base stations, micro base stations, urban base stations, etc. The specific process of the dynamic beamforming method for spectrum sharing between air networks and ground networks will be specifically introduced in the following embodiments, and the specific process of the dynamic beamforming method for spectrum sharing between air networks and ground networks will be introduced with the execution subject as a communication device.
[0106] like Figure 2 As shown, it is a flow chart of a dynamic beamforming method for spectrum sharing between an airborne network and a ground network provided in an embodiment of the present application. The method may include the following steps:
[0107] S1. Input the state of at least one ground base station in multiple time slots within the target range into the initialization strategy network model, obtain the actions of each ground base station in each time slot, and execute the actions of each ground base station in each time slot to obtain the reward and cost of each ground base station in each time slot. The target range is the area covered by the air network and the ground network.
[0108] Specifically, the communication equipment can obtain the status of each ground base station in the target range in multiple consecutive time slots, and then for each ground base station, input the status of the ground base station in multiple consecutive time slots into the initialization strategy network model to obtain the action of the ground base station in each time slot, and further, execute the action of each ground base station in each time slot to obtain the reward and cost of each ground base station in each time slot.
[0109] In the embodiment of the present application, the target range may be an area covered by both the aerial network and the ground network, and the target range may be an urban network coverage area, a rural network coverage area, a mountain network coverage area, a special network coverage area, etc. At the same time, any of the above time slots is any time slot before the current time slot.
[0110] Optionally, the above-mentioned initialization strategy network model can be a combination of at least one of a fully connected neural network model, a recurrent neural network model, a convolutional neural network model, a long short-term memory neural network model, a deep residual network model, etc.
[0111] Among them, the communication equipment can pre-train an algorithm model, and then input the actions of each ground base station in each time slot within the target range into the algorithm model, and the algorithm model outputs the rewards and costs of each ground base station in each time slot.
[0112] In practical applications, the communication device can control each ground base station within the target range to perform its own actions in each time slot to obtain the rewards and costs corresponding to each ground base station in each time slot. It should be noted here that for any ground base station n within the target range at any time slot t', the action a of the ground base station n in the time slot t' is executed. n (t′) is actually the action a performed by the ground base station n at the time slot t′ n (t′), correspondingly, the state of the ground base station n can be expressed as n (t′+1)|s n (t′),a n (t′)) from the state s at time slot t′ n (t') changes to the state s of the next time slot (t'+1) of time slot t' n (t′+1), at this time, the ground base station n can obtain the reward r of this time slot t′ n (t′) and cost c n (t′).
[0113] Among them, P(s n (t′+1)|s n (t′),a n (t′))∈T,a n (t′)∈A,r n (t′)∈R,c n (t′)∈C,s n (t′)∈S,s n (t′+1)∈S.
[0114] In one embodiment, the action a of any ground base station n within the target range at any time slot t′ n (t′) is expressed as:
[0115]
[0116] Among them, 1,...,,..., respectively represent the numbers of each ground cell within the target range; 1,...,,..., respectively represent the numbers of each aerial device within the target range; 1,...,,..., respectively represent the numbers of each ground device within the target range; represents the ratio of the total transmission power of ground base station n to the maximum transmission power at time slot t′, q n,k (t′) represents the ratio of the transmission power allocated by ground base station n to ground device k in cell n at time slot t′ to the total transmission power of ground base station n, q n,k ∈(0,1], α n,j,irepresents the relevant parameters of the channel vector from the ground base station n to the ground device i in the corresponding cell j at time slot t′; μ n,l represents the relevant parameters of the channel vector of the ground base station n for the air device l at time slot t′; η n represents the noise related parameters of ground base station n at time slot t′;
[0117] Normalized beamforming vector It is expressed as:
[0118]
[0119] Where, (n,j,i) represents the ground base station n for the ground device i (i=1,...,K, and i≠k) in the cell j (j=1,...,N); (n,l) represents the ground base station n for the aerial device l, L represents the number of aerial devices within the target range; h n,j,i (t′) represents the channel vector from the ground base station n to the ground device i in the corresponding cell j at time slot t′; α n,j,i g represents the relevant parameters of the channel vector from the ground base station n to the ground device i in the corresponding cell j at time slot t′; n,l (t′) represents the channel vector of the ground base station n for the air device l at time slot t′; μ n,l represents the relevant parameters of the channel vector of the ground base station n for the air device l in the time slot t′; η n represents the noise related parameters of ground base station n at time slot t′; I represents the unit matrix. In practical applications,
[0120] In the embodiment of the present application, it can be calculated that And because So the constraint It must be true.
[0121] In one embodiment, the reward r of any ground base station n within the target range at any time slot t′ is n (t′) is expressed as:
[0122]
[0123] Among them, R n,k (t′) represents the rate of ground equipment k in cell n where ground base station n is located at time slot t′; Indicates that in time slot t ′ The rate loss caused by the interference of ground base station n; P j,i(t′) represents the achievable rate of the ground device i in the cell j when the ground base station n does not exist minus the achievable rate of the ground device i in the cell j when the ground base station n exists at the time slot t′; represents the set of interfered ground devices of ground base station n at time slot t′; represents the expected signal power received by ground device i in cell j at time slot t′; β n,j,i (t′) represents the interference power received by ground equipment i in cell j from ground base station n at time slot t′; β j,i (t′) represents the interference power plus noise power received by ground equipment i in cell j at time slot t′.
[0124] At the same time, the cost c of any ground base station n within the target range at any time slot t′ n (t′) is expressed as:
[0125] c n (t′) = c n,1 (t′),..., n,l (t′),..., n,L (t′)]
[0126]
[0127] Among them, c n,l (t′) represents the cost of ground base station n with respect to aerial device l at time slot t′; ρ l (t′) represents the interference power received by the air device l from all ground base stations at time slot t′, ρ n,l (t′) represents the interference power received by space device l from ground base station n at time slot t′, I max It represents the maximum receiving interference power threshold that device l can tolerate in time slot t′.
[0128] In the embodiment of the present application, if 0≤ρ l (t′)≤I max , then c n,l (t′) may be less than or equal to the second preset threshold, otherwise, c n,l (t′) may be greater than a second preset threshold value. The second preset threshold value may be a small positive number, such as 0.01, 0.005, 0.008, etc., but in the embodiment of the present application, the second preset threshold value may be equal to 0.
[0129] S2. Determine the reward advantage function of each time slot based on the state of each ground base station in each time slot, the state of the next time slot of each time slot, the reward of each time slot and the reward advantage function of each next time slot; and determine the target of the reward value of each time slot based on the reward advantage function of each time slot, the reward value network model of each time slot and the state of each time slot.
[0130] In practical applications, the communication device can pre-train a reward advantage function acquisition model, and then for any time slot, the state of each ground base station in the time slot, the state of the next time slot of the time slot, the reward of the time slot and the reward advantage function of the next time slot of the time slot are input into the reward advantage function acquisition model, and the reward advantage function acquisition model outputs the reward advantage function of each ground base station in the time slot.
[0131] Optionally, the above-mentioned reward advantage function acquisition model can be implemented by at least one of a convolutional neural network model, a feedforward neural network model, a recurrent neural network model, a Hopfield neural network model and a deep learning network model.
[0132] In addition, for any ground base station in any time slot, the communication device can perform arithmetic operations on the state of the ground base station in the time slot, the state of the ground base station in the next time slot of the time slot, the reward of the ground base station in the time slot, and the reward advantage function of the ground base station in the next time slot of the time slot to obtain the reward advantage function of the ground base station in the time slot. Optionally, the above-mentioned arithmetic operation can include at least one of addition operation, subtraction operation, multiplication operation, division operation, weighted sum operation, etc.
[0133] Furthermore, the communication device can process the reward advantage function of the ground base station in the time slot and the state of the ground base station in the time slot according to the reward value network model of the ground base station in the time slot to obtain the target reward value of the ground base station in the time slot.
[0134] Alternatively, the communication device can input the reward value network model of the ground base station in the time slot, the reward advantage function of the ground base station in the time slot and the state of the ground base station in the time slot into the algorithm model based on a pre-trained algorithm model, and the algorithm model outputs the target reward value of the ground base station in the time slot.
[0135] S3. Determine the cost advantage function of each time slot based on the state of each ground base station in each time slot, the state of the next time slot of each time slot, the cost of each time slot and the cost advantage function of each next time slot; and determine the cost value target of each time slot based on the cost advantage function of each time slot, the cost value network of each time slot and the state of each time slot.
[0136] Specifically, the communication device can pre-train a cost advantage function acquisition model, and then for any time slot, the state of each ground base station in the time slot, the state of the next time slot of the time slot, the cost of the time slot and the cost advantage function of the next time slot of the time slot are input into the cost advantage function acquisition model, and the cost advantage function acquisition model outputs the cost advantage function of each ground base station in the time slot.
[0137] Optionally, the cost advantage function acquisition model can be implemented by at least one of a convolutional neural network model, a feedforward neural network model, a recurrent neural network model, a Hopfield neural network model, and a deep learning network model. It should be noted that the cost advantage function acquisition model and the reward advantage function acquisition model may have the same or different composition structures. If the cost advantage function acquisition model and the reward advantage function acquisition model have the same composition structures, the model parameters may be different.
[0138] In addition, for any ground base station in any time slot, the communication device can perform arithmetic operations on the state of the ground base station in the time slot, the state of the ground base station in the next time slot of the time slot, the cost of the ground base station in the time slot and the cost advantage function of the ground base station in the next time slot of the time slot to obtain the cost advantage function of the ground base station in the time slot.
[0139] S4. Obtain multiple training data sets according to the state, action, reward advantage function, cost advantage function, reward value target and cost value target of each ground base station in each time slot, and train the initialization strategy network model corresponding to each ground base station according to each training data set until the training convergence condition is reached, thereby obtaining the optimal strategy network model of each ground base station.
[0140] In the embodiment of the present application, the process of obtaining the state, action, reward and cost of each ground base station in each time slot can be understood as the process of constructing a constrained Markov decision process. On this basis, the optimal strategy network model of each ground base station is obtained through the state, action, reward and cost of each ground base station in each time slot, and the optimal action of each ground base station in the current time slot is determined to maximize the expected value of the cumulative discount reward and the expected value of the cumulative discount cost less than or equal to the first preset threshold. Among them, the expected value of the above-mentioned cumulative discount reward is
[0141] Based on the states, actions, reward advantage functions, cost advantage functions, reward value targets and cost value targets of each ground base station in multiple consecutive time slots obtained in the previous steps, the time period corresponding to the multiple time slots can be divided into multiple sub-time periods, and then the states, actions, reward advantage functions, cost advantage functions, reward value targets and cost value targets of each ground base station in each time slot in each sub-time period form a training data set to obtain multiple training data sets.
[0142] In the embodiment of the present application, the optimal strategy network models of different ground base stations are different. In practical applications, for the initialization strategy network model of any ground base station, the communication device can train the initialization strategy network model corresponding to the ground base station according to each training data set respectively until the training convergence condition is reached, and obtain multiple trained strategy network models, and then test the multiple trained strategy network models, and select the trained strategy network model with the most accurate output result from the multiple trained strategy network models to determine it as the optimal strategy network model of the ground base station.
[0143] In addition, the communication equipment can also train the initialization strategy network model corresponding to the ground base station through each training data set, that is, first train the initialization strategy network model through a training data set, and after reaching the training convergence condition, train the trained strategy network model obtained from the previous training through the next training data set, and so on, until the initialization strategy network model corresponding to each ground base station is trained through all training data sets and reaches the training convergence condition, and the currently obtained trained strategy network model is determined as the optimal strategy network model of the ground base station.
[0144] In an embodiment of the present application, the step in S4 above can be referred to as a step of using a deep reinforcement learning algorithm to solve a constrained Markov process to obtain an optimal strategy network model for each ground base station.
[0145] S5. Input the state of each ground base station in the current time slot into each optimal strategy network model to obtain the optimal action of each ground base station in the current time slot. The optimal action is obtained with the goal of maximizing the expected value of the cumulative discount reward and the expected value of the cumulative discount cost to be less than or equal to the first preset threshold.
[0146] In practical applications, for any ground base station, the communication device can obtain the state of the ground base station in the current time slot, and input the state of the ground base station in the current time slot into the optimal strategy network model of the ground base station. The optimal strategy network model outputs the optimal action of the ground base station in the current time slot.
[0147] Optionally, the first preset threshold may be user-defined or determined based on historical experience, which is not limited in the embodiments of the present application.
[0148] S6. Determine the beamforming vector of each ground base station in the current time slot according to the optimal action of each ground base station in the current time slot.
[0149] It should be noted here that, within the target range, when the spectrum used by the ground base station corresponding to the ground network and the dedicated base station corresponding to the air network is the same, in order to reduce the interference of a large number of ground base stations to the air equipment on the aircraft when the aircraft is flying, the ground base stations can exchange information with each other and perform collaborative beamforming. Naturally, the ground network and the air network can be applied by obtaining the beamforming vector of each ground base station in the current time slot. In an embodiment of the present application, the above-mentioned ground network and air network can be referred to as a ground-to-air network or a cognitive ground-to-air network. In the ground-to-air network, it is necessary to protect each air equipment to ensure that the interference of each ground network to each air equipment is less than the safety threshold.
[0150] Based on the optimal actions of each ground base station in the current time slot obtained in the previous steps, the communication device can perform arithmetic operations on the optimal actions of each ground base station in the current time slot to obtain the beamforming vectors of each ground base station in the current time slot.
[0151] In addition, the communication device can also find the action that matches the optimal action of each ground base station in the current time slot in the corresponding relationship between the actions of different ground base stations in different time slots and the beamforming vectors of different ground base stations in different time slots, and determine the beamforming vector corresponding to the action that matches in the comparison relationship as the beamforming vector of each ground base station in the current time slot. In practical applications, the signal received by the ground device k in the cell n where the ground base station n is located in time slot t′
[0152] Among them, x n,k represents the information that ground equipment k in cell n wants to obtain at time slot t′, z n,k (t′) indicates that the ground equipment k in cell n obeys the The received additive white Gaussian noise is, where
[0153] The technical solution in the embodiment of the present application inputs the state of at least one ground base station in a target range in multiple time slots into the initialization strategy network model, obtains the action of each ground base station in each time slot, and executes the action of each ground base station in each time slot to obtain the reward and cost of each ground base station in each time slot. The target range is the area covered by the aerial network and the ground network. The reward advantage function of each time slot is determined according to the state of each ground base station in each time slot, the state of the next time slot of each time slot, the reward of each time slot and the reward advantage function of each next time slot; and the reward value target of each time slot is determined according to the reward advantage function of each time slot, the reward value network model of each time slot and the state of each time slot. The cost advantage function of each time slot is determined according to the state of each ground base station in each time slot, the state of the next time slot of each time slot, the cost of each time slot and the cost advantage function of each next time slot; and based on According to the cost advantage function of each time slot, the cost value network of each time slot and the state of each time slot, the cost value target of each time slot is determined, and multiple training data sets are obtained according to the state, action, reward advantage function, cost advantage function, reward value target and cost value target of each ground base station in each time slot, and the initialization strategy network model corresponding to each ground base station is trained according to each training data set until the training convergence condition is reached, and the optimal strategy network model of each ground base station is obtained, and the state of each ground base station in the current time slot is input into each optimal strategy network model to obtain the optimal action of each ground base station in the current time slot; the optimal action is obtained with the goal of maximizing the expected value of the cumulative discounted reward and the expected value of the cumulative discounted cost less than or equal to the first preset threshold value, and the beamforming vector of each ground base station in the current time slot is determined according to the optimal action of each ground base station in the current time slot. The above method can obtain the beamforming vectors of each ground base station in the current time slot within the target range jointly covered by the air network and the ground network through a set of deep reinforcement learning algorithms. Compared with the traditional technology that uses an iterative optimization algorithm to obtain the beamforming vectors of each ground base station within the target range, which requires real-time global channel state information, the required amount of data is less, thereby reducing the complexity of obtaining the beamforming vectors of each ground base station and improving the speed and efficiency of obtaining the beamforming vectors of each ground base station; at the same time, the method does not require manual participation and can avoid manual participation errors, thereby improving the accuracy of the beamforming vectors of each ground base station obtained; in addition, the above method avoids the process of adjusting the penalty coefficient through multiple trial and error, thereby reducing the communication overhead required for obtaining the beamforming vectors of each ground base station and reducing the cost; furthermore, the above method obtains the optimal action of each ground base station with the goal of maximizing the expected value of the cumulative discounted reward and the expected value of the cumulative discounted cost less than or equal to the first preset threshold value when each ground base station violates the interference power constraint slightly, and on this basis, the accuracy of the beamforming vectors of each ground base station finally obtained is higher.
[0154] The following is a description of the process of determining the reward advantage function of each time slot based on the state of each ground base station in each time slot, the state of the next time slot of each time slot, the reward of each time slot, and the reward advantage function of each next time slot. Figure 3 As shown, the steps in S2 above can be implemented in the following ways:
[0155] S21. For any time slot, the state of each ground base station in the time slot and the state of the next time slot of the time slot are input into the corresponding reward value network model to obtain the reward value of the time slot and the reward value of the next time slot; the reward value network model is obtained by training the data corresponding to the time slot in each training data set.
[0156] Among them, the state s of the ground base station n in time slot t′ n (t′) is:
[0157]
[0158] in, represents the local information of ground base station n at time slot t′; Indicates that in time slot t ′ The information of the ground base stations that interfere with the ground equipment in the cell n where the ground base station n is located; in,pri (t′) represents the information of the interferer ground base station n of each aerial device at time slot t′; Indicates the information of each interfered ground device in cell n where ground base station n is located at time slot t′; Information indicating the interference caused by ground base station n to each aerial device at time slot t′;
[0159] in,
[0160] represents the compressed version of the downlink channel estimated by the ground base station n to the ground device k in the cell n where it is located at time slot t′; p n,k (t′-1) represents the transmission power of the ground base station n to the corresponding ground device k at the previous time slot (t′-1); R n,k (t′-1) represents the rate of ground equipment k in cell n where ground base station n is located at the previous time slot (t′-1); represents the expected signal power received by the ground device k in the cell n where the ground device is located at the previous time slot (t′-1), β n,k (t′-1) represents the interference power plus noise power received by ground device k in cell n where ground device n is located at the previous time slot (t′-1), represents the compressed version of the interference channel estimated by the ground base station n to the air device l at time slot t′; ρ n, (t′-1) The interference power received by space device l from ground base station n in the previous time slot (t′-1);
[0161] Indicates from Information of interfering ground base stations; represents the set of interfering ground base stations of ground device k in cell n where ground base station n is located at the previous time slot (t′-1); It is a guarantee The threshold value of
[0162]
[0163] Indicates that at time slot t′, The information of interfering ground base stations in represents the set of interfering ground base stations of device l in space at the previous time slot (t′-1); It is a guarantee The threshold value of
[0164]
[0165] represents the set of interfering ground base stations of ground base station n at the previous time slot (t′-1), It is a guarantee The threshold value of ρ l (t′-1) represents the interference power received by the space device l from all ground base stations in the previous time slot (t′-1).
[0166] In practical applications, for any ground base station in any time slot, the communication device can input the state of the ground base station in the time slot and the state of the ground base station in the next time slot into the corresponding reward value network model, and the reward value network model outputs the reward value of the ground base station in the time slot and the reward value of the next time slot.
[0167] Optionally, the above-mentioned reward value network model can be implemented by at least one of a convolutional neural network model, a fully connected neural network model, a residual neural network model, a recurrent neural network model, etc.
[0168] S22. Obtain a reward advantage function for a time slot according to the reward value of the time slot, the reward value of the next time slot, and the reward advantage function of the next time slot.
[0169] Specifically, the communication device can perform arithmetic operations on the reward value of the ground base station in the time slot, the reward value of the ground base station in the next time slot of the time slot, and the reward advantage function of the ground base station in the next time slot of the time slot to obtain the reward advantage function of the ground base station in the time slot.
[0170] In this embodiment of the present application, the reward advantage function in the mth training data in the training data set It can be expressed as:
[0171]
[0172] Wherein, m represents the index of each training data in the training data set corresponding to time slot t′; represents the reward advantage function in the (m+1)th training data in the training data set; δ m represents the temporal difference error in the mth training data in the training data set; r m Represents the reward for the (m+1)th training data in the training dataset; Represents the current reward value network model; Represents the reward value of the mth training data in the training data set; represents the reward value of the (m+1)th training data in the training data set; B M represents the number of data in the training data set; γ is the discount factor; λ is the generalized advantage estimation parameter. Further, in one embodiment, Figure 4 As shown, the step of determining the target of the reward value of each time slot in the above S2 according to the reward advantage function of each time slot, the reward value network model of each time slot and the state of each time slot may include:
[0173] S23. For any time slot, the reward value of the time slot is obtained according to the state of the time slot and the initialized reward value network model.
[0174] In an embodiment of the present application, for any ground base station in any time slot, the communication device can input the state of the ground base station in the time slot into the initialization reward value network model of the ground base station to obtain the reward value of the ground base station in the time slot.
[0175] S24. The sum of the reward value of the time slot and the reward advantage function of the time slot is used as the target of the reward value of the time slot.
[0176] Among them, the target of the reward value in the mth training data is for:
[0177] At the same time, in an embodiment of the present application, for any time slot of any ground base station, the communication device can use the sum of the reward value of the ground base station for that time slot and the reward advantage function of the ground base station for that time slot as the target of the reward value of the ground base station for that time slot.
[0178] The technical solution in the embodiment of the present application, for any time slot, inputs the state of each ground base station in the time slot and the state of the next time slot of the time slot into the corresponding reward value network model, obtains the reward value of the time slot and the reward value of the next time slot, and obtains the reward advantage function of the time slot based on the reward value of the time slot, the reward value of the next time slot and the reward advantage function of the next time slot; the above method can obtain the reward advantage function of each ground base station in each time slot, so as to provide basic information for further training the optimal strategy network model of each ground base station; at the same time, the method does not require manual participation, can reduce manual participation errors, and thus can improve the accuracy of the reward advantage function of each ground base station in each time slot.
[0179] In one embodiment, if Figure 5 As shown, the step of determining the cost advantage function of each time slot in the above S3 according to the state of each ground base station in each time slot, the state of the next time slot of each time slot, the cost of each time slot and the cost advantage function of each next time slot can be implemented in the following way:
[0180] S31. For any time slot, the state of each ground base station in the time slot and the state of the next time slot of the time slot are input into the corresponding cost value network model to obtain the cost value of the time slot and the cost value of the next time slot. The cost value network model is obtained by training the data corresponding to the time slot in the training data set.
[0181] In an embodiment of the present application, for any ground base station in any time slot, the communication device can input the state of the ground base station in the time slot and the state of the next time slot of the time slot into the cost-value network model corresponding to the ground base station to obtain the cost value of the ground base station in the time slot and the cost value of the ground base station in the next time slot.
[0182] S32. Obtain a cost advantage function of a time slot according to the cost value of the time slot, the cost value of the next time slot, and the cost advantage function of the next time slot.
[0183] Optionally, for any ground base station in any time slot, the communication device can perform arithmetic operations on the cost value of the ground base station in the time slot, the cost value of the ground base station in the next time slot, and the cost advantage function of the ground base station in the next time slot to obtain the cost advantage function of the ground base station in the time slot.
[0184] Optionally, the above-mentioned arithmetic operation processing can be implemented by at least one of weighted sum operation, addition operation, multiplication operation, division operation, etc.
[0185] In the embodiment of the present application, the cost advantage function in the mth training data It can be expressed as:
[0186]
[0187] in, represents the cost advantage function of the (m+1)th training data in the training data set; represents the temporal difference error in the mth training data in the training data set; c m Represents the cost in the mth training data; Represents the current cost-value network model; Represents the cost value in the mth training data; represents the cost value in the (m+1)th training data; γ represents the discount factor; λ represents the generalized advantage estimation parameter.
[0188] Further, in one embodiment, Figure 6 As shown, the step of determining the target of the cost value of each time slot in the above S3 according to the cost advantage function of each time slot, the cost value network of each time slot and the state of each time slot may include:
[0189] S33. For any time slot, the cost value of the time slot is obtained according to the state of the time slot and the initialized cost value network model.
[0190] In an embodiment of the present application, for any ground base station in any time slot, the communication device can input the state of the ground base station in the time slot into the initialization cost value network model of the ground base station to obtain the cost value of the ground base station in the time slot.
[0191] S34. The sum of the cost value of the time slot and the cost advantage function of the time slot is taken as the target of the cost value of the time slot.
[0192] Among them, the target of the cost value in the mth training data for:
[0193] At the same time, in an embodiment of the present application, for any time slot of any ground base station, the communication device can use the sum of the cost value of the time slot of the ground base station and the cost advantage function of the time slot of the ground base station as the target of the cost value of the time slot of the ground base station.
[0194] The technical solution in the embodiment of the present application, for any time slot, obtains the cost value of the time slot according to the state of the time slot and the initialized cost value network model, and takes the sum of the cost value of the time slot and the cost advantage function of the time slot as the target of the cost value of the time slot; the above method can obtain the target of the cost value of each ground base station in each time slot, so as to provide basic information for further training the optimal strategy network model of each ground base station; at the same time, the method does not require manual participation, can reduce manual participation errors, and thus can improve the accuracy of the target of the cost value of each ground base station in each time slot.
[0195] The following is a description of the process of training the initialization strategy network model corresponding to each ground base station according to each training data set until the training convergence condition is reached to obtain the optimal strategy network model for each ground base station. Figure 7 As shown, the steps in S4 above can be implemented in the following ways:
[0196] S41. Update the cost parameter vector according to the average cost of the ground base station in each time slot and the strategy network learning rate to obtain an updated cost parameter vector.
[0197] Specifically, the communication device can perform arithmetic operations on the average value of the cost of the ground base station in each time slot, the policy network learning rate and the cost parameter vector to obtain an updated cost parameter vector. Optionally, the arithmetic operation can be implemented by at least one of addition, subtraction, multiplication, etc.
[0198] In the embodiment of the present application, the updated cost parameter vector can be expressed as:
[0199]
[0200] Among them, α v represents the learning rate of the cost parameter vector; v represents the cost parameter vector; represents the average cost in the training dataset, b represents the threshold vector of discounted cumulative cost; v max represents the maximum value of the cost parameter vector; Indicates that Clipped to the interval [0,v max ].
[0201] In one embodiment, if Figure 8 As shown, the step of training the initialization strategy network model of the ground base station according to each sub-data set in the current group and obtaining the latest strategy network parameters in the above S44 may include:
[0202] S441. Determine the update parameter value of the current policy network model according to the state, action, reward advantage function, cost advantage function, parameter vector of the updated cost in each sub-data set and the number of sub-data sets in the current group.
[0203] The communication device may perform arithmetic operations on the states, actions, reward advantage functions, cost advantage functions, parameter vectors of updated costs, and the number of sub-data sets in the current group to obtain updated parameter values of the current policy network model. Optionally, the arithmetic operations may be implemented by at least one of addition, subtraction, derivation, weighted summation, and division.
[0204] In addition, the communication device can also pre-train an update parameter model, and input the states, actions, reward advantage functions, cost advantage functions, parameter vectors of updated costs in each sub-data set in the current group, and the number of sub-data sets in the current group into the update parameter model, which outputs the update parameter values of the current policy network model.
[0205] However, in the embodiment of the present application, the above update parameter value is expressed as:
[0206]
[0207] Among them, B S Indicates the data length of the sub-dataset; Represents the current policy network parameters; represents the loss function of the current policy network model; τ represents a fixed temperature parameter; express about The gradient of Represents the current policy network model; Represents the policy network model trained by the previous training data set; Represents the reward advantage function in the bth data of each sub-data set in any group; represents the cost advantage function of the bth data in each sub-data set in any group; a b represents the action in the bth data of each sub-data set in any group; s b Represents the state of the bth data in each sub-data set in any group.
[0208] S442, according to the current policy network parameters, policy network learning rate and update parameter value of the initialized policy network model, obtain the latest policy network parameters. Among them, the latest policy network parameters are:
[0209] In practical applications, the communication device can perform arithmetic operations on the current policy network parameters, policy network learning rate and update parameter values of the initialized policy network model to obtain the latest policy network parameters. Optionally, the arithmetic operations can be implemented by at least one of addition, subtraction, derivation, weighted summation, division, etc.
[0210] In the embodiment of the present application, the process of obtaining the latest strategic network parameters can be expressed by the formula:
[0211] S42, obtain the strategy network model trained by the previous training data set: Among them, let the initialization strategy network model also be
[0212] S43. For any ground base station, multiple sets of training are required; in each set of training, multiple sets of sub-data sets are obtained according to the training data set containing training data of multiple time slots corresponding to the ground base station; each set of sub-data sets includes training data of multiple time slots.
[0213] In an embodiment of the present application, at least one of the multiple training data sets includes training data of the same ground base station in multiple time slots, and the training data includes the state, action, reward advantage function, cost advantage function, reward value target and cost value target of the ground base station in multiple time slots.
[0214] In practical applications, for any ground base station, at least one training data set including the training data of the ground base station in multiple time slots can be selected from multiple training data sets, and then each acquired training data set can be grouped to obtain multiple groups of sub-data sets corresponding to each training data set of the ground base station.
[0215] It should be noted that each selected training data set includes the training data of the ground base station in multiple time slots. At the same time, each selected training data set corresponds to multiple groups of sub-data sets. In addition, a group of training is performed by dividing multiple groups of sub-data sets from any selected training data set.
[0216] Among them, according to the identifier of the ground base station carried in the training data of each training data set, a training data set including the training data of the ground base station in multiple time slots can be selected from multiple training data sets; and according to the identifier of the ground base station carried in each training data set, a training data set including the training data of the ground base station in multiple time slots can be selected from multiple training data sets.
[0217] Meanwhile, the training data set may be grouped in the training data set according to the order in which the training data are placed, specifically, the training data of a plurality of consecutive time slots are divided into one group, and the number of training data in each divided sub-data set is the same.
[0218] In addition, the training data set may also be grouped by selecting the training data that are separated by multiple time slots in the training data set and grouping them into one group to obtain multiple groups of sub-data sets.
[0219] S44. According to each sub-data set in the current group, the initialization strategy network model of the ground base station is trained to obtain the latest strategy network parameters.
[0220] Specifically, for each sub-data set in the current group, the communication device can train the initialization strategy network model of the ground base station through each sub-data set until the training convergence condition is reached, and determine the strategy network parameters in the current initialization strategy network model as the latest strategy network parameters.
[0221] S45. Update the current policy network parameters to the latest policy network parameters to obtain the current policy network model.
[0222] It should be noted here that the current strategy network model obtained by updating the current strategy network parameters of the initialized strategy network model to the latest strategy network parameters can be understood as the current strategy network model obtained by training the initialized strategy network model of the ground base station through each sub-data set in the current group.
[0223] S46. Use the current strategy network model as the initialization strategy network model, and execute the steps in S44-S45 in sequence according to the sub-data sets in other groups until the latest current strategy network model meets the training convergence conditions or reaches the preset maximum number of training times, and use the latest current strategy network model as the optimal strategy network model of the ground base station.
[0224] Among them, the training convergence conditions include: Represents the KL divergence between the current policy network model and the initialization policy network model; s m represents the state vector in the mth training data in the training data set; Λ represents the second preset threshold.
[0225] Furthermore, the current strategy network model can continue to be used as the initialization strategy network model, and the steps in S44-S45 above can be executed in sequence according to the sub-data sets in other groups until the latest current strategy network model meets the training convergence conditions or reaches the preset maximum number of training times, and the latest current strategy network model is used as the optimal strategy network model of the ground base station.
[0226] In the embodiment of the present application, the number of other groups can be determined by the number of selected training data sets, which is equal to the number of selected training data sets minus 1. If the number of other groups is greater than or equal to 2, each sub-data set in any other group can include each sub-data set obtained by grouping any other selected training data set.
[0227] In practical applications, the steps in S44-S45 can be executed in sequence according to each sub-data set in each other group, that is, for each sub-data set in any other group, the steps in S44-S45 can be executed in sequence, and then for each sub-data set in the next other group, the steps in S44-S45 can be executed in sequence until all the sub-data sets in all other groups are trained and the latest current strategy network model meets the training convergence conditions or reaches the preset maximum number of training times, and the latest current strategy network model is used as the optimal strategy network model of the ground base station.
[0228] The technical solution in the embodiment of the present application can perform multiple groups of training on the initialization strategy network model corresponding to each ground base station according to each training data set to obtain the optimal strategy network model of each ground base station. This method does not require manual participation and can reduce manual participation errors, thereby improving the accuracy of the determined optimal strategy network model, so as to prepare for further obtaining the optimal action of each ground base station in the current time slot; at the same time, this method performs multiple groups of training on the initialization strategy network model corresponding to each ground base station, which can make the optimal strategy network model finally obtained higher than that of a single group of training.
[0229] In one embodiment, the present application also provides a dynamic beamforming method for spectrum sharing between an air network and a ground network, the method comprising the following process:
[0230] (1) Inputting the state of at least one ground base station in multiple time slots within the target range into the initialization strategy network model, obtaining the actions of each ground base station in each time slot, and executing the actions of each ground base station in each time slot to obtain the reward and cost of each ground base station in each time slot; the target range is the area covered by both the aerial network and the ground network;
[0231] Action a of ground base station n at time slot t′ n (t′) is expressed as:
[0232]
[0233] Among them, 1,...,,..., respectively represent the numbers of each ground cell within the target range; 1,...,,..., respectively represent the numbers of each aerial device within the target range; 1,...,,..., respectively represent the numbers of each ground device within the target range; represents the ratio of the total transmission power of the ground base station n to the maximum transmission power at time slot t′; q n,k (t′) represents the ratio of the transmission power allocated by the ground base station n to the ground device k in the cell n at time slot t′ to the total transmission power of the ground base station n; α n,j,i represents the relevant parameters of the channel vector from the ground base station n to the ground device i in the corresponding cell j at time slot t′; μ n,l represents the relevant parameters of the channel vector of the ground base station n for the air device l at time slot t′; η n represents the noise related parameters of ground base station n at time slot t′;
[0234] (2) For any time slot, the state of each ground base station in the time slot and the state of the next time slot of the time slot are input into the corresponding reward value network model to obtain the reward value of the time slot and the reward value of the next time slot; the reward value network model is obtained by training the data corresponding to the time slot in each training data set;
[0235] Ground base station n in time slot t ′ Status n (t′) is:
[0236]
[0237] represents the local information of ground base station n at time slot t′; Indicates that in time slot t ′ The information of the ground base stations that interfere with the ground equipment in the cell n where the ground base station n is located; in,pri (t′) represents the information of the interferer ground base station n of each aerial device at time slot t′; Indicates the information of each interfered ground device in cell n where ground base station n is located at time slot t′; Information indicating the interference caused by ground base station n to each aerial device at time slot t′;
[0238] in,
[0239] represents the compressed version of the downlink channel estimated by the ground base station n to the ground device k in the cell n where it is located at time slot t′; p n,k (t′-1) represents the transmission power of the ground base station n to the corresponding ground device k at the previous time slot (t′-1); R n,k (t′-1) represents the rate of ground equipment k in cell n where ground base station n is located at the previous time slot (t′-1); represents the expected signal power received by the ground device k in the cell n where the ground device is located at the previous time slot (t′-1); β n,k (t′-1) represents the interference power plus noise power received by the ground device k in the cell n where the ground device n is located at the previous time slot (t′-1); represents the compressed version of the interference channel estimated by the ground base station n to the air device l at time slot t′; ρ n, (t′-1) The interference power received by space device l from ground base station n in the previous time slot (t′-1);
[0240] Indicates from Information of interfering ground base stations; represents the set of interfering ground base stations of ground device k in cell n where ground base station n is located at the previous time slot (t′-1); It is a guarantee The threshold value of
[0241]
[0242] Indicates that at time slot t′, The information of interfering ground base stations in represents the set of interfering ground base stations of device l in space at the previous time slot (t′-1); It is a guarantee The threshold value of
[0243]
[0244] represents the set of interfering ground base stations of ground base station n at the previous time slot (t′-1), It is a guarantee The threshold value of ρ l (t′-1) represents the interference power received by space device l from all ground base stations in the previous time slot (t′-1);
[0245] (3) obtaining a reward advantage function of a time slot according to the reward value of the time slot, the reward value of the next time slot, and the reward advantage function of the next time slot;
[0246] The reward advantage function in the mth training data in the training dataset for:
[0247]
[0248] Wherein, m represents the index of each training data in the training data set corresponding to time slot t′; represents the reward advantage function in the (m+1)th training data in the training data set; δ m represents the temporal difference error in the mth training data in the training data set; r m Represents the reward for the (m+1)th training data in the training dataset; Represents the current reward value network model; Represents the reward value of the mth training data in the training data set; represents the reward value of the (m+1)th training data in the training data set; B M represents the number of data in the training data set; γ is the discount factor; λ is the generalized advantage estimation parameter;
[0249] (4) For any time slot, the reward value of the time slot is obtained according to the state of the time slot and the initialization reward value network model;
[0250] (5) The sum of the reward value of the time slot and the reward advantage function of the time slot is taken as the target of the reward value of the time slot;
[0251] The target of the reward value in the mth training data for:
[0252]
[0253] (6) For any time slot, the state of each ground base station in the time slot and the state of the next time slot of the time slot are input into the corresponding cost value network model to obtain the cost value of the time slot and the cost value of the next time slot; the cost value network model is obtained by training the data corresponding to the time slot in the training data set;
[0254] (7) According to the cost value of the time slot, the cost value of the next time slot and the cost advantage function of the next time slot, the cost advantage function of the time slot is obtained; the cost advantage function in the mth training data for:
[0255]
[0256] in, represents the cost advantage function of the (m+1)th training data in the training data set; represents the temporal difference error in the mth training data in the training data set; c m Represents the cost in the mth training data; Represents the current cost-value network model; Represents the cost value in the mth training data; represents the cost value in the (m+1)th training data; γ represents the discount factor; λ represents the generalized advantage estimation parameter;
[0257] (8) For any time slot, the cost value of the time slot is obtained according to the state of the time slot and the initialization cost value network model;
[0258] (9) The sum of the cost value of the time slot and the cost advantage function of the time slot is taken as the target of the cost value of the time slot;
[0259] The target of the cost value in the mth training data for:
[0260]
[0261] (10) updating the cost parameter vector according to the average cost of the ground base station in each time slot and the strategy network learning rate to obtain an updated cost parameter vector;
[0262] The updated cost parameter vector includes:
[0263]
[0264] Among them, α v represents the learning rate of the cost parameter vector; v represents the cost parameter vector; represents the average cost in the training dataset, b represents the threshold vector of discounted cumulative cost; v max represents the maximum value of the cost parameter vector; Indicates that Clipped to the interval [0,v max ];
[0265] (11) Get the strategy network model trained by the previous training data set: Among them, let the initialization strategy network model also be
[0266] (12) For any ground base station, multiple sets of training are required; in each set of training, multiple sets of sub-data sets are obtained according to a training data set containing training data of multiple time slots corresponding to the ground base station; each set of sub-data sets includes training data of multiple time slots;
[0267] (13) determining the updated parameter values of the current policy network model according to the states, actions, reward advantage functions, cost advantage functions, parameter vectors of the updated costs in each sub-data set and the number of sub-data sets in the current group;
[0268] Update parameter values are expressed as:
[0269]
[0270] Among them, B S Indicates the data length of the sub-dataset; Represents the current policy network parameters; Represents the loss function of the current policy network model; express about The gradient of Represents the current policy network model; Represents the policy network model trained by the previous training data set; Represents the reward advantage function in the bth data of each sub-data set in any group; represents the cost advantage function of the bth data in each sub-data set in any group; a b represents the action in the bth data of each sub-data set in any group; s b Indicates the state of the bth data in each sub-data set in any group;
[0271] (14) obtaining the latest policy network parameters according to the current policy network parameters, policy network learning rate and update parameter values of the initialized policy network model;
[0272] Among them, the latest strategy network parameters are:
[0273] (15) Update the current policy network parameters to the latest policy network parameters to obtain the current policy network model;
[0274] (16) Using the current policy network model as the initialization policy network model, and executing the above steps S44-S45 in sequence according to each sub-data set in other groups, until the latest current policy network model meets the training convergence condition or reaches the preset maximum number of training times, and using the latest current policy network model as the optimal policy network model of the ground base station;
[0275] The training convergence conditions include:
[0276]
[0277] in, Represents the KL divergence between the current policy network model and the initialization policy network model; s m represents the state vector in the mth training data in the training data set; ε represents the second preset threshold;
[0278] (17) inputting the state of each ground base station in the current time slot into each optimal strategy network model to obtain the optimal action of each ground base station in the current time slot; the optimal action is obtained with the goal of maximizing the expected value of the cumulative discounted reward and the expected value of the cumulative discounted cost to be less than or equal to a first preset threshold;
[0279] (18) Determine the beamforming vector of each ground base station in the current time slot according to the optimal action of each ground base station in the current time slot.
[0280] The execution process of the above (1) to (18) can be specifically referred to the description of the above embodiment. The implementation principles and technical effects are similar and will not be repeated here.
[0281] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.
[0282] Based on the same inventive concept, the embodiment of the present application also provides a dynamic beamforming device for sharing spectrum between airborne networks and ground networks for implementing the dynamic beamforming method for sharing spectrum between airborne networks and ground networks involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the embodiments of one or more dynamic beamforming devices for sharing spectrum between airborne networks and ground networks provided below can be referred to the limitations of the dynamic beamforming method for sharing spectrum between airborne networks and ground networks above, and will not be repeated here.
[0283] In one embodiment, Fig. 9 This is a schematic diagram of the structure of a dynamic beamforming device for sharing spectrum between airborne networks and ground networks in one embodiment of the present application. The dynamic beamforming device for sharing spectrum between airborne networks and ground networks provided in the embodiment of the present application can be applied to communication equipment. Fig. 9 As shown, the dynamic beamforming device for spectrum sharing between air networks and ground networks in the embodiment of the present application may include: a first acquisition module 11, a first determination module 12, a second determination module 13, a training module 14, a second acquisition module 15 and a solution module 16, wherein:
[0284] The first acquisition module 11 is used to input the state of at least one ground base station in a target range in multiple time slots into the initialization strategy network model, obtain the action of each ground base station in each time slot, and execute the action of each ground base station in each time slot to obtain the reward and cost of each ground base station in each time slot; the target range is the area covered by the air network and the ground network;
[0285] The first determination module 12 is used to determine the reward advantage function of each time slot according to the state of each ground base station in each time slot, the state of the next time slot of each time slot, the reward of each time slot and the reward advantage function of each next time slot; and determine the target of the reward value of each time slot according to the reward advantage function of each time slot, the reward value network model of each time slot and the state of each time slot;
[0286] The second determination module 13 is used to determine the cost advantage function of each time slot according to the state of each ground base station in each time slot, the state of the next time slot of each time slot, the cost of each time slot and the cost advantage function of each next time slot; and determine the target of the cost value of each time slot according to the cost advantage function of each time slot, the cost value network of each time slot and the state of each time slot;
[0287] The training module 14 is used to obtain multiple training data sets according to the state, action, reward advantage function, cost advantage function, reward value target and cost value target of each ground base station in each time slot, and train the initialization strategy network model corresponding to each ground base station according to each training data set until the training convergence condition is reached, so as to obtain the optimal strategy network model of each ground base station;
[0288] The second acquisition module 15 is used to input the state of each ground base station in the current time slot into each optimal strategy network model to obtain the optimal action of each ground base station in the current time slot; the optimal action is obtained with the goal of maximizing the expected value of the cumulative discount reward and the expected value of the cumulative discount cost to be less than or equal to the first preset threshold;
[0289] A solution module 16, configured to determine a beamforming vector of each ground base station in a current time slot according to an optimal action of each ground base station in a current time slot;
[0290] Among them, the action a of the ground base station n in time slot t′ n (t′) is expressed as:
[0291]
[0292] Among them, 1,...,,..., respectively represent the numbers of each ground cell within the target range; 1,...,,..., respectively represent the numbers of each aerial device within the target range; 1,...,,..., respectively represent the numbers of each ground device within the target range; represents the ratio of the total transmission power of the ground base station n to the maximum transmission power at time slot t′; q n,k (t′) represents the ratio of the transmission power allocated by the ground base station n to the ground device k in the cell n at time slot t′ to the total transmission power of the ground base station n; α n,j,i represents the relevant parameters of the channel vector from the ground base station n to the ground device i in the corresponding cell j at time slot t′; μ n,l represents the relevant parameters of the channel vector of the ground base station n for the air device l at time slot t′; η n represents the noise related parameters of ground base station n at time slot t′;
[0293] The reward r of ground base station n at time slot t′ is n (t′) is expressed as:
[0294]
[0295] Among them, R n,k (t′) represents the rate of ground equipment k in cell n where ground base station n is located at time slot t′; Indicates that in time slot t ′ The rate loss caused by the interference of ground base station n; P j,i (t′) represents the achievable rate of the ground device i in the cell j when the ground base station n does not exist minus the achievable rate of the ground device i in the cell j when the ground base station n exists at the time slot t′; represents the set of interfered ground devices of ground base station n at time slot t′; represents the expected signal power received by ground device i in cell j at time slot t′; β n,j,i (t′) represents the interference power received by ground equipment i in cell j from ground base station n at time slot t′; β j,i (t′) represents the interference power plus noise power received by ground equipment i in cell j at time slot t′;
[0296] The cost c of ground base station n in time slot t′ n (t′) is expressed as:
[0297] c n (t′) = c n,1 (t′),..., n,l (t′),..., n,L (t′)]
[0298]
[0299] Among them, c n,l(t′) represents the cost of ground base station n with respect to aerial device l at time slot t′; ρ l (t′) represents the interference power received by the air device l from all ground base stations at time slot t′; I max Indicates the maximum receiving interference power threshold that the air device l can tolerate.
[0300] The dynamic beamforming device for spectrum sharing between airborne networks and terrestrial networks provided in the embodiment of the present application can be used to execute the technical solution in the above-mentioned dynamic beamforming method embodiment for spectrum sharing between airborne networks and terrestrial networks of the present application. Its implementation principle and technical effect are similar and will not be repeated here.
[0301] In one embodiment, the first determining module 12 includes: a first obtaining unit and a second obtaining unit, wherein:
[0302] The first acquisition unit is used for inputting the state of each ground base station in the time slot and the state of the next time slot of the time slot into the corresponding reward value network model for any time slot, so as to obtain the reward value of the time slot and the reward value of the next time slot; the reward value network model is obtained by training the data corresponding to the time slot in each training data set;
[0303] The state s of ground base station n at time slot t′ n (t′) is:
[0304]
[0305] represents the local information of ground base station n at time slot t′; Indicates that in time slot t ′ The information of the ground base stations that interfere with the ground equipment in the cell n where the ground base station n is located; in,pri (t′) represents the information of the interferer ground base station n of each aerial device at time slot t′; Indicates the information of each interfered ground device in cell n where ground base station n is located at time slot t′; Information indicating the interference caused by ground base station n to each aerial device at time slot t′;
[0306] in,
[0307] represents the compressed version of the downlink channel estimated by the ground base station n to the ground device k in the cell n where it is located at time slot t′; p n,k (t′-1) represents the transmission power of the ground base station n to the corresponding ground device k at the previous time slot (t′-1); R n,k(t′-1) represents the rate of ground equipment k in cell n where ground base station n is located at the previous time slot (t′-1); represents the expected signal power received by the ground device k in the cell n where the ground device is located at the previous time slot (t′-1); β n,k (t′-1) represents the interference power plus noise power received by the ground device k in the cell n where the ground device n is located at the previous time slot (t′-1); represents the compressed version of the interference channel estimated by the ground base station n to the air device l at time slot t′; ρ n, (t′-1) The interference power received by space device l from ground base station n in the previous time slot (t′-1);
[0308] Indicates from Information of interfering ground base stations; represents the set of interfering ground base stations of ground device k in cell n where ground base station n is located at the previous time slot (t′-1); It is a guarantee The threshold value of
[0309]
[0310] Indicates that at time slot t′, The information of interfering ground base stations in represents the set of interfering ground base stations of device l in space at the previous time slot (t′-1); It is a guarantee The threshold value of
[0311]
[0312] represents the set of interfering ground base stations of ground base station n at the previous time slot (t′-1), It is a guarantee The threshold value of
[0313] ρ l (t′-1) represents the interference power received by space device l from all ground base stations in the previous time slot (t′-1);
[0314] A second acquisition unit, configured to obtain a reward advantage function of a time slot according to the reward value of the time slot, the reward value of the next time slot, and the reward advantage function of the next time slot;
[0315] The reward advantage function in the mth training data in the training dataset for:
[0316]
[0317] Wherein, m represents the index of each training data in the training data set corresponding to time slot t′; represents the reward advantage function in the (m+1)th training data in the training data set; δ m represents the temporal difference error in the mth training data in the training data set; r m Represents the reward for the (m+1)th training data in the training dataset; Represents the current reward value network model; Represents the reward value of the mth training data in the training data set; represents the reward value of the (m+1)th training data in the training data set; B M represents the number of data in the training data set; γ is the discount factor; λ is the generalized advantage estimation parameter.
[0318] The dynamic beamforming device for spectrum sharing between airborne networks and terrestrial networks provided in the embodiment of the present application can be used to execute the technical solution in the above-mentioned dynamic beamforming method embodiment for spectrum sharing between airborne networks and terrestrial networks of the present application. Its implementation principle and technical effect are similar and will not be repeated here.
[0319] In one embodiment, the first determining module includes: a third obtaining unit and a fourth obtaining unit, wherein:
[0320] A third acquisition unit is used to obtain the reward value of any time slot according to the state of the time slot and the initialization reward value network model;
[0321] a fourth acquisition unit, configured to take the sum of the reward value of the time slot and the reward advantage function of the time slot as the target of the reward value of the time slot;
[0322] The target of the reward value in the mth training data for:
[0323]
[0324] The dynamic beamforming device for spectrum sharing between airborne networks and terrestrial networks provided in the embodiment of the present application can be used to execute the technical solution in the above-mentioned dynamic beamforming method embodiment for spectrum sharing between airborne networks and terrestrial networks of the present application. Its implementation principle and technical effect are similar and will not be repeated here.
[0325] In one embodiment, the second determining module includes:
[0326] A fifth acquisition unit is used for inputting the state of each ground base station in the time slot and the state of the next time slot of the time slot into the corresponding cost value network model for any time slot, so as to obtain the cost value of the time slot and the cost value of the next time slot; the cost value network model is obtained by training the data corresponding to the time slot in the training data set;
[0327] The sixth acquisition unit is used to obtain the cost advantage function of the time slot according to the cost value of the time slot, the cost value of the next time slot and the cost advantage function of the next time slot; the cost advantage function in the mth training data for:
[0328]
[0329] in, represents the cost advantage function of the (m+1)th training data in the training data set; represents the temporal difference error in the mth training data in the training data set; c m Represents the cost in the mth training data; Represents the current cost-value network model; Represents the cost value in the mth training data; represents the cost value in the (m+1)th training data; γ represents the discount factor; λ represents the generalized advantage estimation parameter.
[0330] The dynamic beamforming device for spectrum sharing between airborne networks and terrestrial networks provided in the embodiment of the present application can be used to execute the technical solution in the above-mentioned dynamic beamforming method embodiment for spectrum sharing between airborne networks and terrestrial networks of the present application. Its implementation principle and technical effect are similar and will not be repeated here.
[0331] In one embodiment, the second determination module includes: a seventh acquisition unit and a determination unit, wherein:
[0332] A seventh acquisition unit, configured to obtain, for any time slot, a cost value of the time slot according to a state of the time slot and an initialized cost value network model;
[0333] a determination unit, configured to take the sum of the cost value of the time slot and the cost advantage function of the time slot as a target of the cost value of the time slot;
[0334] The target of the cost value in the mth training data for:
[0335]
[0336] The dynamic beamforming device for spectrum sharing between airborne networks and terrestrial networks provided in the embodiment of the present application can be used to execute the technical solution in the above-mentioned dynamic beamforming method embodiment for spectrum sharing between airborne networks and terrestrial networks of the present application. Its implementation principle and technical effect are similar and will not be repeated here.
[0337] In one embodiment, the training module includes: a first updating unit, an eighth acquiring unit, a ninth acquiring unit, a training unit, a second updating unit and a cyclic training unit, wherein:
[0338] A first updating unit is used to update the cost parameter vector according to the average cost of the ground base station in each time slot and the policy network learning rate to obtain an updated cost parameter vector;
[0339] The updated cost parameter vector includes:
[0340]
[0341] Among them, α v represents the learning rate of the cost parameter vector; v represents the cost parameter vector; represents the average cost in the training dataset, b represents the threshold vector of discounted cumulative cost; v max represents the maximum value of the cost parameter vector; Indicates that Clipped to the interval [0,v max ];
[0342] The eighth acquisition unit is used to obtain the strategy network model obtained by training the previous training data set: Among them, let the initialization strategy network model also be
[0343] A ninth acquisition unit is used for performing multiple sets of training for any ground base station; in each set of training, multiple sets of sub-data sets are acquired according to a training data set containing training data of multiple time slots corresponding to the ground base station; each set of sub-data sets includes training data of multiple time slots;
[0344] A training unit, used to train the initialization strategy network model of the ground base station according to each sub-data set in the current group to obtain the latest strategy network parameters;
[0345] A second updating unit is used to update the current policy network parameters to the latest policy network parameters to obtain a current policy network model;
[0346] A cyclic training unit, used to use the current policy network model as the initialization policy network model, and execute the steps in S44-S45 in sequence according to each sub-data set in other groups, until the latest current policy network model meets the training convergence condition or reaches a preset maximum number of training times, and use the latest current policy network model as the optimal policy network model of the ground base station;
[0347] The training convergence conditions include:
[0348]
[0349] in, Represents the KL divergence between the current policy network model and the initialization policy network model; s m represents the state vector in the mth training data in the training data set; ε represents the second preset threshold.
[0350] The dynamic beamforming device for spectrum sharing between airborne networks and terrestrial networks provided in the embodiment of the present application can be used to execute the technical solution in the above-mentioned dynamic beamforming method embodiment for spectrum sharing between airborne networks and terrestrial networks of the present application. Its implementation principle and technical effect are similar and will not be repeated here.
[0351] In one embodiment, the training unit includes: a determination subunit and an acquisition subunit, wherein:
[0352] A determination subunit, used to determine the update parameter value of the current policy network model according to the state, action, reward advantage function, cost advantage function, parameter vector of the updated cost in each sub-data set and the number of sub-data sets in the current group;
[0353] Update parameter values are expressed as:
[0354]
[0355] Among them, B s Indicates the data length of the sub-dataset; Represents the current policy network parameters; Represents the loss function of the current policy network model; express about The gradient of Represents the current policy network model; Represents the policy network model trained by the previous training data set; Represents the reward advantage function in the bth data of each sub-data set in any group; represents the cost advantage function of the bth data in each sub-data set in any group; a brepresents the action in the bth data of each sub-data set in any group; s b Indicates the state of the bth data in each sub-data set in any group;
[0356] An acquisition subunit is used to acquire the latest policy network parameters according to the current policy network parameters, the policy network learning rate and the update parameter value of the initialized policy network model;
[0357] Among them, the latest strategy network parameters are:
[0358] The dynamic beamforming device for spectrum sharing between airborne networks and terrestrial networks provided in the embodiment of the present application can be used to execute the technical solution in the above-mentioned dynamic beamforming method embodiment for spectrum sharing between airborne networks and terrestrial networks of the present application. Its implementation principle and technical effect are similar and will not be repeated here.
[0359] For the specific definition of the dynamic beamforming device for spectrum sharing between airborne networks and terrestrial networks, please refer to the definition of the dynamic beamforming method for spectrum sharing between airborne networks and terrestrial networks mentioned above, which will not be repeated here. Each module in the above-mentioned dynamic beamforming device for spectrum sharing between airborne networks and terrestrial networks can be implemented in whole or in part through software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the communication device in the form of hardware, or can be stored in the memory in the communication device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.
[0360] In one embodiment, a communication device is provided. Fig.10 . Fig.10 It is a structural diagram of the communication device provided in an embodiment of the present application. Fig.10 The communication device shown includes: a transceiver, at least one processor, a memory, at least one network interface and a user interface. The various components in the communication device are coupled together through a bus system. It is understood that the bus system is used to achieve connection and communication between these components. In addition to the data bus, the bus system also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system is not shown in FIG. Fig.10 In the embodiment of the present application, various buses are all labeled as bus systems. In addition, in the embodiment of the present application, the transceiver can be multiple components, that is, including a transmitter and a receiver, and is a unit for communicating with various other devices on a transmission medium.
[0361] The user interface may include a display, a keyboard, or a pointing device (eg, a mouse, a trackball, a touch pad, or a touch screen, etc.).
[0362] It can be understood that the memory in the embodiments of the present application can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct RAM bus random access memory (DRRAM). The memory of the system and method described in the embodiments of the present application is intended to include but is not limited to these and any other suitable types of memory.
[0363] In some embodiments, the memory stores the following elements, executable modules or data structures, or other subsets, or other extended sets: operating system and application programs.
[0364] The operating system includes various system programs, such as a framework layer, a core library layer, a driver layer, etc., which are used to implement various basic services and process hardware-based tasks. The application includes various application programs, such as a media player, a browser, etc., which are used to implement various application services. The program for implementing the method of the embodiment of the present application can be included in the application.
[0365] In the embodiment of the present application, by calling a program or instruction stored in the memory, specifically, it can be a program or instruction stored in the application.
[0366] Part or all of the methods disclosed in the above embodiments of the present application can be applied to a processor, or implemented by a processor, or implemented by a processor in cooperation with other components (such as a transceiver). The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor or an instruction in the form of software. The above processor may be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiments of the present application can be directly embodied as a hardware decoding processor to be executed, or a combination of hardware and software modules in the decoding processor to be executed. The software module may be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.
[0367] It is to be understood that the embodiments described in the embodiments of the present application can be implemented by hardware, software, firmware, middleware, microcode or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more application specific integrated circuits (Application Specific Integrated Circuits, ASIC), digital signal processors (Digital Signal Processing, DSP), digital signal processing devices (DSPDevice, DSPD), programmable logic devices (Programmable Logic Device, PLD), field programmable gate arrays (Field-Programmable Gate Array, FPGA), general processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described in the present application or a combination thereof.
[0368] For software implementation, the technology in the embodiments of the present application can be implemented by a module (e.g., a process, a function, etc.) that performs the functions described in the embodiments of the present application. The software code can be stored in a memory and executed by a processor. The memory can be implemented in the processor or outside the processor.
[0369] Those skilled in the art will understand that Fig.10 The structure shown in the figure is only a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the communication device to which the scheme of the present application is applied. The specific communication device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0370] In one embodiment, a communication device is further provided, including a transceiver, a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the steps of the method in any of the above embodiments are implemented.
[0371] In one embodiment, a computer-readable storage medium is further provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method in any of the above embodiments are implemented.
[0372] In one embodiment, a computer program product is also provided, including a computer program, which implements the steps of the method in any of the above embodiments when executed by a processor.
[0373] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).
[0374] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0375] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.
Claims
1. A dynamic beamforming method for spectrum sharing between airborne networks and ground networks, characterized in that: The method comprises: S1. Input the state of at least one ground base station in a target range in multiple time slots into the initialization strategy network model, obtain the action of each ground base station in each time slot, and execute the action of each ground base station in each time slot to obtain the reward and cost of each ground base station in each time slot; the target range is the area covered by the air network and the ground network; S2, determining the reward advantage function of each time slot according to the state of each ground base station in each time slot, the state of the next time slot of each time slot, the reward of each time slot and the reward advantage function of each next time slot; and determining the target of the reward value of each time slot according to the reward advantage function of each time slot, the reward value network model of each time slot and the state of each time slot; S3, determining the cost advantage function of each time slot according to the state of each ground base station in each time slot, the state of the next time slot of each time slot, the cost of each time slot and the cost advantage function of each next time slot; and determining the target of the cost value of each time slot according to the cost advantage function of each time slot, the cost value network of each time slot and the state of each time slot; S4, obtaining a plurality of training data sets according to the state, action, reward advantage function, cost advantage function, reward value target and cost value target of each ground base station in each time slot, and training the initialization strategy network model corresponding to each ground base station according to each training data set until the training convergence condition is reached, thereby obtaining the optimal strategy network model of each ground base station; S5. Input the state of each of the ground base stations in the current time slot into each of the optimal strategy network models to obtain the optimal action of each of the ground base stations in the current time slot; the optimal action is obtained with the goal of maximizing the expected value of the cumulative discount reward and the expected value of the cumulative discount cost to be less than or equal to a first preset threshold; S6. Determine a beamforming vector of each ground base station in the current time slot according to the optimal action of each ground base station in the current time slot.
2. The method according to claim 1, characterized in that The action a of the ground base station n in time slot t′ n (t′) is expressed as: Wherein, 1, ..., j, ..., N respectively represent the numbers of each ground cell within the target range; 1, ..., l, ..., L respectively represent the numbers of each aerial device within the target range; 1, ..., i, ..., K respectively represent the numbers of each ground device within the target range; represents the ratio of the total transmission power of the ground base station n to the maximum transmission power at time slot t′; q n,k (t′) represents the ratio of the transmission power allocated by the ground base station n to the ground device k in the cell n at time slot t′ to the total transmission power of the ground base station n; α n,j,i (t′) represents the relevant parameters of the channel vector from the ground base station n to the ground equipment i in the corresponding cell j at time slot t′; μ n,l (t′) represents the relevant parameters of the channel vector of the ground base station n for the aerial device n at time slot t′; η n (t′) represents the noise related parameters of the ground base station n at time slot t′.
3. The method according to claim 1 or 2, characterized in that: The reward r of the ground base station n at time slot t′ n (t′) is expressed as: Among them, R n,k (t′) represents the rate of the ground device k in the cell n where the ground base station n is located at time slot t′; represents the rate loss caused by the interference of the ground base station n at time slot t′; P j,i (t′) represents the achievable rate of ground device i in cell j when the ground base station n does not exist at time slot t′ minus the achievable rate of ground device i in cell j when the ground base station n exists; represents the set of interfered ground devices of the ground base station n at time slot t′; represents the expected signal power received by the ground device i in the cell j at time slot t′; β n,j,i (t′) represents the interference power received by the ground equipment i in the cell j from the ground base station n at time slot t′; β j,i (t′) represents the interference power plus noise power received by the ground device i in the cell j at time slot t′.
4. The method according to claim 1 or 2, characterized in that: The cost c of the ground base station n in time slot t′ n (t′) is expressed as: c n (t′)=[c n,1 (t′),…,c n,l (t′),…,c n,L (t′)] Among them, c n,l (t′) represents the cost of ground base station n with respect to aerial device l at time slot t′; ρ l (t′) represents the interference power received by the air device l from all ground base stations at time slot t′; I max Indicates the maximum receiving interference power threshold that the air device l can tolerate.
5. The method according to claim 1 or 2, characterized in that: The step of determining the reward advantage function of each time slot in S2 according to the state of each ground base station in each time slot, the state of the next time slot of each time slot, the reward of each time slot and the reward advantage function of each next time slot includes: S21. For any time slot, the state of each ground base station in the time slot and the state of the next time slot of the time slot are input into the corresponding reward value network model to obtain the reward value of the time slot and the reward value of the next time slot; the reward value network model is obtained by training the data corresponding to the time slot in each training data set; The state s of the ground base station n at time slot t' n (t′) is expressed as: represents the local information of the ground base station n at time slot t′; represents the information of the ground base station that interferes with each of the ground devices in the cell n where the ground base station n is located at time slot t′; in,pri (t′) represents the information of the ground base station that is the interferer of each of the aerial devices at time slot t′; Indicates the information of each of the ground devices interfered in the cell n where the ground base station n is located at time slot t′; Information indicating interference caused by the ground base station n to each of the aerial devices at time slot t′; in, represents the compressed version of the downlink channel of the ground equipment k in the cell n where the ground base station n estimates itself at time slot t′; n,k (t′-1) represents the transmission power of the ground base station n to the corresponding ground device k at the previous time slot (t′-1); R n,k (t′-1) represents the rate of the ground device k in the cell n where the ground base station n is located at the previous time slot (t′-1); represents the expected signal power received by the ground device k in the cell n where the ground device is located at the previous time slot (t′-1); β n,k (t′-1) represents the interference power plus noise power received by the ground device k in the cell n where the ground device n is located at the previous time slot (t′-1); represents the compressed version of the interference channel estimated by the ground base station n to the air device l at time slot t′; n,l (t′-1) The interference power received by the aerial device l from the ground base station n at the previous time slot (t′-1); Indicates from Information of interfering ground base stations; represents the set of interfering ground base stations of the ground device k in the cell n where the ground base station n is located at the previous time slot (t′-1); It is a guarantee The threshold value of Indicates that at time slot t′, The information of interfering ground base stations in represents the set of interfering ground base stations of the space device in the previous time slot (t′-1); It is a guarantee The threshold value of represents the set of interfering ground base stations of the ground base station n at the previous time slot (t′-1), It is a guarantee The threshold value of ρl(t′-1) represents the interference power received by the aerial device l from all ground base stations at the previous time slot (t′-1); S22, obtaining a reward advantage function of the time slot according to the reward value of the time slot, the reward value of the next time slot and the reward advantage function of the next time slot; The reward advantage function in the mth training data in the training data set for: Wherein, m represents the index of each training data in the training data set corresponding to time slot t′; represents the reward advantage function in the (m+1)th training data in the training data set; δ m represents the temporal difference error in the mth training data in the training data set; r m represents the reward of the (m+1)th training data in the training data set; Represents the current reward value network model; represents the reward value in the mth training data in the training data set; represents the reward value of the (m+1)th training data in the training data set; B M represents the number of data in the training data set; γ is the discount factor; λ is the generalized advantage estimation parameter.
6. The method according to claim 1 or 2, characterized in that: The step of determining the target of the reward value of each time slot according to the reward advantage function of each time slot, the reward value network model of each time slot and the state of each time slot in S2 includes: S23. For any time slot, according to the state of the time slot and the initialized reward value network model, obtain the reward value of the time slot; S24, taking the sum of the reward value of the time slot and the reward advantage function of the time slot as the target of the reward value of the time slot; The target of the reward value in the mth training data for:
7. The method according to claim 1 or 2, characterized in that: The step of determining the cost advantage function of each time slot in S3 according to the state of each ground base station in each time slot, the state of the next time slot of each time slot, the cost of each time slot and the cost advantage function of each next time slot includes: S31. For any time slot, input the state of each ground base station in the time slot and the state of the next time slot of the time slot into the corresponding cost value network model to obtain the cost value of the time slot and the cost value of the next time slot; the cost value network model is obtained by training the data corresponding to the time slot in the training data set; S32, according to the cost value of the time slot, the cost value of the next time slot and the cost advantage function of the next time slot, obtain the cost advantage function of the time slot; the cost advantage function in the mth training data for: in, represents the cost advantage function of the (m+1)th training data in the training data set; represents the temporal difference error in the mth training data in the training data set; c m Represents the cost in the mth training data; Represents the current cost-value network model; Represents the cost value in the mth training data; represents the cost value in the (m+1)th training data; γ represents the discount factor; λ represents the generalized advantage estimation parameter.
8. The method according to claim 1 or 2, characterized in that: The step of determining the target of the cost value of each time slot according to the cost advantage function of each time slot, the cost value network of each time slot and the state of each time slot in S3 includes: S33, for any time slot, obtaining the cost value of the time slot according to the state of the time slot and the initialized cost value network model; S34, taking the sum of the cost value of the time slot and the cost advantage function of the time slot as the target of the cost value of the time slot; The target cost value in the mth training data for:
9. The method according to claim 1 or 2, characterized in that: The step of training the initialization strategy network model corresponding to each of the ground base stations according to each of the training data sets in S4 until a training convergence condition is reached to obtain the optimal strategy network model of each of the ground base stations includes: S41, updating a cost parameter vector according to an average value of the cost of the ground base station in each of the time slots and a learning rate of the strategy network to obtain an updated cost parameter vector; The updated cost parameter vector includes: Among them, α v represents the learning rate of the cost parameter vector; v represents the cost parameter vector; represents the average cost in the training dataset, b represents the threshold vector of discounted cumulative cost; v mmx represents the maximum value of the cost parameter vector; Indicates that Clipped to the interval [0,v max ]; S42, obtain the policy network model obtained by training the previous training data set as π θ′ ; Among them, let the initialization strategy network model also be π θ′ ; S43. For any ground base station, multiple sets of training are required; in each set of training, multiple sets of sub-data sets are obtained according to a training data set containing training data of multiple time slots corresponding to the ground base station; each set of sub-data sets includes training data of multiple time slots; S44, training the initialization strategy network model of the ground base station according to each sub-data set in the current group to obtain the latest strategy network parameters; S45, updating the current policy network parameters to the latest policy network parameters to obtain a current policy network model; S46, using the current policy network model as the initialization policy network model, and executing the steps in S44-S45 in sequence according to each sub-data set in other groups, until the latest current policy network model meets the training convergence condition or reaches the preset maximum number of training times, and using the latest current policy network model as the optimal policy network model of the ground base station; The training convergence conditions include: Among them, D KL (π θ ,π θ′ ) represents the KL divergence between the current policy network model and the initialized policy network model; s m represents the state vector in the mth training data in the training data set; ε represents the second preset threshold.
10. The method according to claim 9, characterized in that The step of training the initialization strategy network model of the ground base station according to each sub-data set in the current group to obtain the latest strategy network parameters in S44 includes: S441, determining the update parameter value of the current policy network model according to the state, action, reward advantage function, cost advantage function, parameter vector of the updated cost and the number of sub-data sets in the current group in each of the sub-data sets; The updated parameter value is expressed as: Among them, B S represents the data length of the sub-data set; θ represents the current strategy network parameter; Representing the loss function of the current policy network model; express The gradient with respect to θ; π θ Represents the current strategy network model; π θ′ Represents the policy network model trained by the previous training data set; Represents the reward advantage function in the bth data of each sub-data set in any group; represents the cost advantage function of the bth data in each sub-data set in any group; a b represents the action in the bth data of each sub-data set in any group; s b Indicates the state of the bth data in each sub-data set in any group; S442, acquiring the latest policy network parameters according to the current policy network parameters of the initialized policy network model, the policy network learning rate and the update parameter value; Among them, the latest strategy network parameters are:
11. A dynamic beamforming device for spectrum sharing between airborne networks and ground networks, characterized in that: The device comprises: The first acquisition module is used to input the state of at least one ground base station in a target range in multiple time slots into the initialization strategy network model, obtain the action of each ground base station in each time slot, and execute the action of each ground base station in each time slot to obtain the reward and cost of each ground base station in each time slot; the target range is the area covered by the air network and the ground network; The first determination module is used to determine the reward advantage function of each time slot according to the state of each ground base station in each time slot, the state of the next time slot of each time slot, the reward of each time slot and the reward advantage function of each next time slot; and determine the target of the reward value of each time slot according to the reward advantage function of each time slot, the reward value network model of each time slot and the state of each time slot; The second determination module is used to determine the cost advantage function of each time slot according to the state of each ground base station in each time slot, the state of the next time slot of each time slot, the cost of each time slot and the cost advantage function of each next time slot; and determine the target of the cost value of each time slot according to the cost advantage function of each time slot, the cost value network of each time slot and the state of each time slot; A training module, used to obtain multiple training data sets according to the state, action, reward advantage function, cost advantage function, reward value target and cost value target of each ground base station in each time slot, and train the initialization strategy network model corresponding to each ground base station according to each training data set until the training convergence condition is reached, so as to obtain the optimal strategy network model of each ground base station; A second acquisition module is used to input the state of each ground base station in the current time slot into each optimal strategy network model to obtain the optimal action of each ground base station in the current time slot; the optimal action is obtained with the goal of maximizing the expected value of the cumulative discount reward and the expected value of the cumulative discount cost to be less than or equal to a first preset threshold; The solution module is used to determine the beamforming vector of each ground base station in the current time slot according to the optimal action of each ground base station in the current time slot.
12. A communication device, comprising a transceiver, a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 10 are implemented.
13. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 10 are implemented.
14. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 10 are implemented.
