Non-terrestrial communication method and apparatus

Abstract

Disclosed in the embodiments of the present disclosure are a non-terrestrial communication method and apparatus. The method comprises: acquiring, at the current time step, the data transmission rate between each terminal among at least one terminal and a network device; inputting state information at the current time step into a first model, so as to obtain a first action executed by the network device in the state of the current time step, wherein the first action is used for indicating the displacement of the network device at the current time step; and on the basis of the first action, determining the position of the network device at a subsequent time step, wherein the first model is used for determining the position of the network device at each time step within a first period, and the total data transmission rate of the at least one terminal is the largest within the first period. Thus, a network device can move rationally even when a terminal is moving, thereby effectively ensuring the throughput of a system, making the system achieve communication with a short delay and a high data rate, and improving the communication efficiency of the system.

Description

Non-terrestrial communication method and apparatus TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of communication, and particularly relates to a non-terrestrial communication method and apparatus. BACKGROUND

[0002] Non-terrestrial (also referred to as non-terrestrial) networks (NTN) are an important technology introduced in the 5th generation mobile communication technology (5G), which provides wireless resources through satellites or unmanned aerial vehicles instead of ground base stations.

[0003] Unmanned aerial vehicle-mounted base stations (UAV-BS) can be used as a supplement to ground base stations (BS) to provide additional capacity in temporarily overcrowded situations.

[0004] SUMMARY

[0005] Embodiments of the present disclosure provide a non-terrestrial communication method and apparatus.

[0006] In a first aspect, a non-terrestrial communication method is provided, and the method comprises:

[0007] obtaining a data transmission rate between each terminal in at least one terminal and the network device at a current time step;

[0008] inputting state information at the current time step into a first model to obtain a first action performed by the network device in the state at the current time step, wherein the first action is used to indicate a displacement of the network device at the current time step;

[0009] determining a position of the network device at a next time step based on the first action;

[0010] the first model is used to determine the position of the network device at each time step in a first period, wherein a total data transmission rate of the at least one terminal in the first period is maximum.

[0011] In a second aspect, a network device is provided, and the network device comprises:

[0012] a processing module configured to obtain a data transmission rate between each terminal in at least one terminal and the network device at a current time step;

[0013] The processing module is further configured to input state information at the current time step into the first model to obtain a first action performed by the network device in the state at the current time step, wherein the first action is used to indicate a displacement of the network device at the current time step.

[0014] The processing module is further configured to determine a position of the network device at a next time step based on the first action.

[0015] The first model is used to determine positions of the network device at each time step in a first period, wherein a total data transmission rate of the at least one terminal in the first period is maximum.

[0016] The scheme provided by the embodiments of the present disclosure is to obtain a data transmission rate between each terminal in the at least one terminal and the network device at a current time step; input state information at the current time step into the first model to obtain a first action performed by the network device in the state at the current time step, wherein the first action is used to indicate a displacement of the network device at the current time step; determine a position of the network device at a next time step based on the first action; and the first model is used to determine positions of the network device at each time step in a first period, wherein a total data transmission rate of the at least one terminal in the first period is maximum. Thus, even in the case of terminal movement, the network device can also make reasonable movement, effectively guarantee the throughput of the system, realize low-delay and high-data-rate communication, improve the communication efficiency of the system, and guarantee the communication quality of the system. BRIEF DESCRIPTION OF DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure or the background art, the drawings needed to be used in the embodiments of the present disclosure or the background art will be described below.

[0018] FIG. 1A is a schematic diagram of an architecture of a communication system provided by an embodiment of the present disclosure;

[0019] FIG. 1B is a schematic diagram of an architecture of a space-air-sea (SAS) communication system provided by an embodiment of the present disclosure;

[0020] FIG. 2A is an interaction schematic diagram of a non-ground communication method provided by an embodiment of the present disclosure;

[0021] FIG. 2B is a schematic diagram of a system model provided by an embodiment of the present disclosure;

[0022] FIG. 2C is a schematic diagram of a network device deployment algorithm provided by an embodiment of the present disclosure;

[0023] FIG. 2D is a schematic diagram of a model iteration process provided by an embodiment of the present disclosure;

[0024] FIGS. 3A-3B are flow diagrams of a non-terrestrial communication method according to embodiments of the present disclosure;

[0025] FIG. 4 is a structural diagram of a network device according to embodiments of the present disclosure;

[0026] FIG. 5A is a structural diagram of a communication device according to embodiments of the present disclosure;

[0027] FIG. 5B is a structural diagram of a chip according to embodiments of the present disclosure. DETAILED DESCRIPTION

[0028] Embodiments of the present disclosure provide a non-terrestrial communication method and device.

[0029] In a first aspect, embodiments of the present disclosure provide a non-terrestrial communication method, the method comprising:

[0030] obtaining a data transmission rate between each of at least one terminal and the network device at a current time step;

[0031] inputting state information at the current time step into a first model to obtain a first action performed by the network device in the state at the current time step, wherein the first action is used to indicate a displacement of the network device at the current time step;

[0032] determining a position of the network device at a next time step based on the first action;

[0033] the first model is used to determine a position of the network device at each time step in a first period, wherein a total data transmission rate of the at least one terminal in the first period is maximum.

[0034] In the above embodiments, even if the terminals move, the network device can still move reasonably, effectively guaranteeing the throughput of the system, achieving low-delay and high-data-rate communication, improving the communication efficiency of the system, and guaranteeing the communication quality of the system.

[0035] In some embodiments of the first aspect, in some embodiments, the state information at the current time step includes a position of the network device at the current time step and a position of each terminal at the current time step.

[0036] In some embodiments of the first aspect, in some embodiments, the method further comprises:

[0037] inputting the state information at the current time step, the first action, and a reward corresponding to the first action into a second model to obtain quality information;

[0038] The quality information is used to indicate a quality of performing the first action in the state at the current time step.

[0039] In some embodiments of the first aspect, the reward includes a positive reward and a negative reward; wherein,

[0040] The first action that increases the total data transmission rate of the terminal at the current time step compared with the previous time step corresponds to a positive reward; or,

[0041] The first action that decreases the total data transmission rate of the terminal at the current time step compared with the previous time step corresponds to a negative reward; or,

[0042] The first action that causes the network device to exceed the predefined boundary corresponds to a negative reward.

[0043] In some embodiments of the first aspect, the quality information is further used to update the weight parameters of the first model.

[0044] In some embodiments of the first aspect, the second model further includes an experience replay (ER) memory, and the ER memory stores an experience of each historical time step.

[0045] The experience of a historical time step includes: state information at the historical time step, the first action at the historical time step, a reward corresponding to the first action at the historical time step, and state information of a next time step of the historical time step.

[0046] In some embodiments of the first aspect, the first model and the second model are actor-critic deep Q-learning (ACDQL) models; the first model is an actor deep neural network model, and the second model is a critic deep neural network model.

[0047] In some embodiments of the first aspect, the method further includes:

[0048] The weight parameters in the first model are updated by using a policy gradient ascent algorithm.

[0049] In some embodiments of the first aspect, the method further includes:

[0050] The weight parameters in the second model are updated by using a gradient descent algorithm.

[0051] In a second aspect, the embodiments of the present disclosure provide a network device, which comprises a transceiver module and a processing module; wherein the network device is configured to perform the method in the first aspect and the optional implementation manners of the first aspect.

[0052] In a third aspect, the embodiments of the present disclosure provide a network device, which comprises one or more processors; wherein the network device is configured to perform the method in the first aspect and the optional implementation manners of the first aspect.

[0053] In a fourth aspect, the embodiments of the present disclosure provide a communication system, which comprises a terminal and a network device; wherein the network device is configured to perform the method in the first aspect and the optional implementation manners of the first aspect.

[0054] In a fifth aspect, the embodiments of the present disclosure provide a storage medium, which stores instructions, when the instructions are executed on a communication device, causing the communication device to perform the method in the first aspect and the optional implementation manners of the first aspect.

[0055] In a sixth aspect, the embodiments of the present disclosure provide a program product, which, when executed by a communication device, causes the communication device to perform the method in the first aspect and the optional implementation manners of the first aspect.

[0056] In a seventh aspect, the embodiments of the present disclosure provide a computer program, which, when executed on a computer, causes the computer to perform the method in the first aspect and the optional implementation manners of the first aspect.

[0057] In an eighth aspect, the embodiments of the present disclosure provide a chip or chip system. The chip or chip system comprises a processing circuit configured to perform the method in the first aspect and the optional implementation manners of the first aspect.

[0058] It can be understood that the terminal, the access network device, the core network device, the communication system, the storage medium, the program product, the computer program, the chip or the chip system are all configured to perform the method provided by the embodiments of the present disclosure. Therefore, the beneficial effects achieved thereby can refer to the beneficial effects in the corresponding method, which will not be described herein.

[0059] The embodiments of the present disclosure provide a non-terrestrial communication method and device. In some embodiments, the terms of non-terrestrial communication method, information processing method, and communication method can be replaced with each other, the terms of non-terrestrial communication device, information processing device, and communication device can be replaced with each other, and the terms of information processing system and communication system can be replaced with each other.

[0060] The embodiments of the present disclosure are not exhaustive, but only illustrate some embodiments, and are not specific limitations on the protection scope of the present disclosure. In the case of no contradiction, each step in an embodiment can be implemented as an independent embodiment, and the steps can be combined arbitrarily, for example, the scheme after removing part of the steps in an embodiment can also be implemented as an independent embodiment, and the order of the steps in an embodiment can be exchanged arbitrarily, in addition, the optional implementation manners in an embodiment can be combined arbitrarily; in addition, the embodiments can be combined arbitrarily, for example, part or all steps of different embodiments can be combined arbitrarily, an embodiment can be combined with optional implementation manners of other embodiments arbitrarily.

[0061] In each embodiment of the present disclosure, the terms and / or descriptions between the embodiments are consistent if there is no special description and logical conflict, and can be referred to each other, and the technical features in different embodiments can be combined to form new embodiments according to their inherent logical relationship.

[0062] The terms used in the embodiments of the present disclosure are only for the purpose of describing the specific embodiments, and not as a limitation on the present disclosure.

[0063] In the embodiments of the present disclosure, unless otherwise specified, the elements expressed in singular form, such as "one", "a", "the", "above", "said", "preceding", "this" and the like, can represent "one and only one", and can also represent "one or more", "at least one" and the like. For example, in the case of using articles such as "a", "an", "the" and the like in English, the noun after the article can be understood as singular expression, and can also be understood as plural expression.

[0064] In the embodiments of the present disclosure, "a plurality of" means two or more.

[0065] In some embodiments, the terms "at least one of", "one or more", "a plurality of", "multiple" and the like can be replaced with each other.

[0066] In some embodiments, "at least one of A, B", "A and / or B", "in one case A, in another case B", "responsive to case A, responsive to case B" and the like, can be interpreted to include both cases, A and B, in some embodiments, A (A is performed regardless of B), in some embodiments, B (B is performed regardless of A), in some embodiments, selected from the group consisting of A and B (the selection between A and B is an option), in some embodiments, A and B (both A and B are performed).

[0067] In some embodiments, "A or B" and the like, can be interpreted to include both cases, A and B, in some embodiments, A (A is performed regardless of B), in some embodiments, B (B is performed regardless of A), in some embodiments, selected from the group consisting of A and B (the selection between A and B is an option).

[0068] In some embodiments, the prefix words "first", "second" and the like in the disclosure do not limit the position, order, priority, number or content of the described objects, and the description of the described objects should be understood in the context of the claims or embodiments, and should not be construed as redundant limitations. For example, the described object is "field", and the ordinal words before "field" in "first field" and "second field" do not limit the position or order between "fields", and "first" and "second" do not limit whether the "fields" modified by them are in the same message or not, nor do they limit the order of "first field" and "second field". For another example, the described object is "level", and the ordinal words before "level" in "first level" and "second level" do not limit the priority between "levels". For another example, the number of described objects is not limited by ordinal words, and can be one or more. For example, "first device", where the number of "devices" can be one or more. In addition, the objects modified by different prefix words can be the same or different, for example, the described object is "device", and "first device" and "second device" can be the same device or different devices, and their types can be the same or different; for another example, the described object is "information", and "first information" and "second information" can be the same information or different information, and their contents can be the same or different.

[0069] In some embodiments, "including A", "containing A", "for indicating A", "carrying A" can be interpreted as directly carrying A, or indirectly indicating A.

[0070] In some embodiments, the terms "in response to", "in response to determining", "in the case of", "when", "when", "if", "if" and the like can be replaced with each other.

[0071] In some embodiments, the terms "greater than", "greater than or equal to", "not less than", "more than", "more than or equal to", "not less than", "higher than", "higher than or equal to", "not lower than", "above", and the like can be replaced with each other, and the terms "less than", "less than or equal to", "not greater than", "fewer than", "fewer than or equal to", "not more than", "lower than", "lower than or equal to", "not higher than", "below", and the like can be replaced with each other.

[0072] In some embodiments, the apparatuses and devices can be interpreted as physical or virtual, and their names are not limited to the names described in the embodiments, and in some cases can also be understood as "equipment", "device", "circuit", "network element", "node", "function", "unit", "section", "system", "network", "chip", "chip system", "entity", "subject", and the like.

[0073] In some embodiments, "network" can be interpreted as an apparatus included in the network, such as an access network device, a core network device, and the like.

[0074] In some embodiments, an “access network device (AN device)” can also be referred to as a “radio access network device (RAN device),” a “base station (BS),” a “radio base station,” a “fixed station,” and in some embodiments can also be understood as a “node,” an “access point,” a “transmission point (TP),” a “reception point (RP),” a “transmission / reception point (TRP),” a “panel,” an “antenna panel,” an “antenna array,” a “cell,” a “macro cell,” a “small cell,” a “femto cell,” a “pico cell,” a “sector,” a “cell group,” a “serving cell,” a “carrier,” a “component carrier,” a “bandwidth part (BWP),” and the like.

[0075] In some embodiments, a "terminal" or "terminal device" can be referred to as a "user equipment (UE)", "user terminal", "mobile station (MS)", "mobile terminal (MT)", subscriber station, mobile unit, subscriber unit, wireless unit, remote unit, mobile device, wireless device, wireless communication device, remote device, mobile subscriber station, access terminal, mobile terminal, wireless terminal, remote terminal, handset, user agent, mobile client, client, etc.

[0076] In some embodiments, an access network device, a core network device, or a network device can be replaced with a terminal. For example, the embodiments of the present disclosure can also be applied to a structure in which communication between an access network device, a core network device, or a network device and a terminal is replaced with communication between a plurality of terminals (e.g., device-to-device (D2D), vehicle-to-everything (V2X), etc.). In this case, the terminal can also be configured to have all or part of the functions of the access network device. In addition, the terms "uplink", "downlink", etc. can also be replaced with terms corresponding to the inter-terminal communication (e.g., "side"). For example, an uplink channel, a downlink channel, etc. can be replaced with a side channel, and an uplink, a downlink, etc. can be replaced with a side link.

[0077] In some embodiments, a terminal can be replaced with an access network device, a core network device, or a network device. In this case, the access network device, the core network device, or the network device can also be configured to have all or part of the functions of the terminal.

[0078] In some embodiments, the data, information, etc. can be obtained in compliance with the laws and regulations of the country where the location is located.

[0079] In some embodiments, the data, information, etc. can be obtained after obtaining the consent of the user.

[0080] In addition, each element, each row, or each column in the table of the embodiments of the present disclosure can be implemented as an independent embodiment, and any combination of any element, any row, or any column can also be implemented as an independent embodiment.

[0081] FIG. 1A is a schematic diagram of an architecture of a communication system according to an embodiment of the present disclosure.

[0082] As shown in FIG. 1A, the communication system 100 includes a network device 101 and a terminal 102.

[0083] In some embodiments, the network device 101 is, for example, a node or device that accesses a terminal to a wireless network, and the network device can include at least one of a node such as a drone or a satellite in a non-terrestrial communication network, an evolved NodeB (eNB) in a 5G communication system, a next generation eNB (ng-eNB), a next generation NodeB (gNB), a next generation RAN node (NG-RAN node), a node B (NB), a home node B (HNB), a home evolved node B (HeNB), a wireless backhaul device, a radio network controller (RNC), a base station controller (BSC), a base transceiver station (BTS), a base band unit (BBU), a mobile switching center, a base station in a 6G communication system, an Open RAN, a Cloud RAN, a base station in other communication systems, an access node in a Wi-Fi system, but is not limited thereto.

[0084] In some embodiments, the technical solutions of the present disclosure can be applied to an Open RAN architecture, at this time, the interfaces between the access network devices or within the access network devices involved in the embodiments of the present disclosure can become internal interfaces of the Open RAN, and the processes and information interactions between these internal interfaces can be realized through software or programs.

[0085] In some embodiments, the network device can be composed of a central unit (CU) and a distributed unit (DU), where the CU can also be referred to as a control unit. The CU-DU structure can split the protocol layers of the access network device, with some of the protocol layers being controlled by the CU and the rest of the protocol layers or all of the protocol layers being distributed in the DU and controlled by the CU, but is not limited thereto.

[0086] In some embodiments, the terminal 102 includes at least one of a mobile phone, a wearable device, an Internet of Things device, a communication-capable automobile, a smart automobile, a tablet computer (Pad), a wireless transceiver-equipped computer, a virtual reality (VR) terminal device, an augmented reality (AR) terminal device, a wireless terminal device in industrial control, a wireless terminal device in self-driving, a wireless terminal device in remote medical surgery, a wireless terminal device in a smart grid, a wireless terminal device in transportation safety, a wireless terminal device in a smart city, a wireless terminal device in a smart home, and a RedCap terminal, but is not limited thereto.

[0087] It can be understood that the communication system described in the embodiments of the present disclosure is for more clearly illustrating the technical solutions of the embodiments of the present disclosure, and does not constitute a limitation on the technical solutions proposed in the embodiments of the present disclosure. Those skilled in the art can know that, as the system architecture evolves and new business scenarios appear, the technical solutions proposed in the embodiments of the present disclosure are also applicable to similar technical problems.

[0088] The following embodiments of the present disclosure can be applied to the communication system 100 shown in FIG. 1A or part of the subject, but are not limited thereto. The subjects shown in FIG. 1A are exemplary, and the communication system can include all or part of the subjects in FIG. 1A, or other subjects other than those in FIG. 1A. The number and form of each subject is arbitrary, each subject can be physical or virtual, the connection relationship between each subject is exemplary, each subject can not be connected or can be connected, the connection can be in any way, can be direct connection or indirect connection, can be wired connection or wireless connection.

[0089] Embodiments of the present disclosure can be applied to a Non-terrestrial Network (NTN), Long Term Evolution (LTE), LTE-Advanced (LTE-A), LTE-Beyond (LTE-B), SUPER 3G, IMT-Advanced, 4th generation mobile communication system (4G), 5th generation mobile communication system (5G), 5G new radio (NR), Future Radio Access (FRA), New-Radio Access Technology (RAT), New Radio (NR), New radio access (NX), Future generation radio access (FX), Global System for Mobile communications (GSM (registered trademark)), CDMA2000, Ultra Mobile Broadband (UMB), IEEE 802.11 (Wi-Fi (registered trademark)), IEEE 802.16 (WiMAX (registered trademark)), IEEE 802.20, Ultra-WideBand (UWB), Bluetooth (Bluetooth (registered trademark)), Public Land Mobile Network (PLMN) network, Device-to-Device (D2D) system, Machine to Machine (M2M) system, Internet of Things (IoT) system, Vehicle-to-Everything (V2X), system using other communication methods, next-generation system expanded based thereon, and the like. Further, a plurality of systems can be applied in combination (for example, combination of LTE or LTE-A and 5G, and the like).

[0090] In some embodiments, non-terrestrial (also referred to as non-terrestrial) networks (NTN) are an important technology introduced in the 5th generation mobile communication technology (5G) that provides wireless resources through satellites or unmanned aerial vehicles (UAVs) instead of ground base stations.

[0091] In the 6th generation mobile communication technology (6G), however, 6G networks are able to meet the requirements of users for high data speeds and low latency, and maintain network connectivity worldwide. On the other hand, with the growth of maritime traffic, maritime communication in the deep sea is also expanding. To serve critical mission applications, more and more maritime end users need high throughput and low latency.

[0092] In some embodiments, uncrewed aerial vehicle-mounted base stations (UAV-BSs) can be used as a supplement to ground base stations (BSs) to provide additional capacity in temporarily overcrowded situations, such as sports events in stadiums. UAV-BSs are expected to be widely used in 5G and 6G communications in urban microcells, which require a dense network of access points to provide ultra-reliable low-latency communication support.

[0093] In some embodiments, in the deep sea, satellite connections will make 6G networks possible, but the service capacity is very limited due to the long propagation delay between the end user and the satellite. UAV-BSs bring many advantages to supplement cellular connections, but at the same time, they also pose several key challenges that must be addressed to establish an effective UAV-assisted cellular network.

[0094] In some embodiments, one of the key challenges of UAV-BSs is where they should be placed in the sky to maximize network performance, and how they should move in the sky to maintain their high performance over time.

[0095] However, traditional methods are not suitable for dynamic environments with constantly changing network topologies, as they must be re-run to adapt to the new topology even if the network has changed minimally, which will result in a huge computational overhead.

[0096] Therefore, embodiments of the present application propose uncrewed aerial vehicles (UAVs) as aerial backhaul and relay mediums in satellite and coastal base station assisted maritime communication networks. They can provide cost-effective Internet connectivity for users without infrastructure. They can also quickly take over as service providers when ground base stations fail. To ensure the connectivity of the system to maritime / remote users, a low earth orbit satellite, UAV, maritime (remote) user, coastal (offshore) BS, and island BS integrated network is proposed.

[0097] Further, an advanced Space-Air-Sea (SAS) / Space-Air-Land (SAL) 6G network architecture is proposed for maritime traffic or remote areas in ocean waters. It can be as shown in FIG. IB.

[0098] The non-terrestrial communication method and apparatus provided by the present disclosure will be described in detail below with reference to the accompanying drawings.

[0099] FIG. 2A is an interaction diagram of a non-terrestrial communication method according to an embodiment of the present disclosure. As shown in FIG. 2A, the present embodiment relates to a non-terrestrial communication method, which comprises:

[0100] In step S2101, the network device 101 obtains the data transmission rate of each terminal 102 at the current time step.

[0101] In some embodiments, the network device 101 can obtain the data transmission rate between the terminal 102 and the network device 101 at the current time step.

[0102] In some embodiments, the number of terminals 102 is one or more.

[0103] In some embodiments, the terminal 102 is a terminal within a predetermined area range.

[0104] In some embodiments, the terminal 102 is a terminal within the service range that the network device 101 can provide.

[0105] In some embodiments, the service range that the network device 101 can provide can be pre-set.

[0106] In some embodiments, the number of network devices 101 can also be multiple.

[0107] In some embodiments, the number of network devices 101 is multiple, and each network device 101 can obtain the data transmission rate of each terminal 102 at the current time step.

[0108] In some embodiments, the network device 101 can be a UAV-BS.

[0109] In some embodiments, the terminal 102 can be an endpoint.

[0110] As an example, for ease of description, the number of network devices 101 is one.

[0111] In some embodiments, the preset area range can be represented as a D x D area range.

[0112] As an example, the system model can be as shown in FIG. 2B.

[0113] In some embodiments, a path loss model is used to model the path loss between the network device 101 and the terminal 102, which depends on the Line of Sight (LoS) probability and the Non-Line of Sight (NLoS) probability of the connection between the terminal 102 and the network device 101. Since the network device 101 is deployed in a three-dimensional space above the ground, the possibility of establishing a LoS connection with the terminal 102 on the ground is high.

[0114] The Line of Sight (LoS) probability between the network device 101 and the terminal 102 can be represented as formula (1):

[0115] where (k1, k2) are environment type variables set according to the environment type, θ i is the elevation angle between the i-th terminal 102 and the network device 101.

[0116] where the elevation angle θ i is calculated as shown in formula (2): θ i = arctan(h uav / r i ).(2)

[0117] where h uav is the altitude of the network device 101, r i is the horizontal distance between the i-th terminal 102 and the network device 101, which can be obtained by formula (3):

[0118] where (x i , y i ) are the 2D coordinates of the i-th terminal 102, (x uav , yuav h uav ) is the three-dimensional coordinates (3D coordinates) of the network device 101.

[0119] Further, the calculation of the path loss related to the LoS connection can be shown as formula (4):

[0120] wherein f c is the carrier frequency, c is the speed of light, δ LoS is the additional environmental related path loss of the LoS connection.

[0121] Due to the existence of obstacles in the environment such as high-rise buildings, forests, etc., especially in urban and dense urban environments, some signals from the network device 101 reach the terminal 102 through the NLoS link. The NLoS probability between the network device 101 and the i-th terminal 102, as a supplement to the LoS probability, can be represented as formula (5):

[0122] P i(NLoS) = 1 - P i(LoS) . (5)

[0123] The calculation of the path loss related to the NLoS connection can be shown as formula (6):

[0124] wherein δ NLoS is the additional environmental related path loss of the NLoS connection.

[0125] Based on the above formulas, the average path loss between the network device 101 and the i-th terminal 102 can be obtained:

[0126] L i(avg) = P i(LoS) × L i(LoS) + P i(NLoS) × L i(NLoS) . (7)

[0127] Further, based on the average path loss corresponding to each terminal 102, the downlink data transmission rate R i corresponding to each terminal 102 can be determined. Specifically, the calculation formula can be shown as formula (11) as follows.

[0128] In the embodiments of the present application, the terminals 102 (users / endpoints) are also mobile, and their mobility model follows a Gauss-Markov mobility model. In the Gauss-Markov mobility model, each terminal 102 is assigned an initial random velocity and direction.

[0129] In addition, as an example, in the embodiments of the present application, the network device 101 should constantly change its location and move towards the location of the maximum system throughput, that is, the goal is to move the network device 101 to a path that maximizes the total data transmission rate of users in the time step t in the first period T. This goal can be represented as an optimization problem as shown below:

[0130] 0≤x uav ≤x max ,

[0131] 0≤y uav ≤y max . (10)

[0132] wherein the constraint in the above formula (9) ensures that the total power allocated by the network device 101 to the terminals 102 does not exceed the transmission power of the network device 101, wherein p i is the power allocated to the i-th terminal 102, and p uav is the total transmission power of the network device 101. The constraint in the above formula (10) ensures that the movement of the network device 101 does not exceed the predefined boundary, wherein x max and y max are the lengths of the predefined boundary in the x-axis and y-axis directions.

[0133] wherein R i is the downlink data transmission rate between the i-th terminal 102 and the network device 101, and can be represented as:

[0134] wherein B is the total bandwidth of the network device 101, which is evenly allocated to each terminal 102, and N is the number of terminals 102. γ i is the signal to noise ratio (SNR) of the i-th terminal 102 received by the network device 101:

[0135] wherein σ noise is the noise power, and G i is the power gain between the network device 101 and the i-th terminal 102, which can be calculated according to the following formula:

[0136] wherein, is a transmit antenna gain of the network device 101 to the i-th terminal 102, is a receive antenna gain of the network device 101 to the i-th terminal 102, d0is a far-field reference distance, d i is a distance between the network device 101 and the i-th terminal 102, which can be represented as:

[0137] At step S2102, the network device 101 determines a first action based on the first model.

[0138] In some embodiments, the network device 101 can input the state information at the current time step into the first model to obtain the first action performed at the state at the current time step.

[0139] In some embodiments, the first action is used to indicate the displacement of the network device 101 at the current time step.

[0140] In some embodiments, the first model can be used to determine the position of the network device 101 at each time step within the first period, that is, the first model can be used to determine the moving path of the network device 101 within the first period.

[0141] In some embodiments, the path of the network device 101 determined by the first model can maximize the system throughput, that is, the total data transmission rate of at least one terminal 102 in the system within the first period.

[0142] In some embodiments, the first model is an actor deep neural network (DNN) model in an actor-critic deep Q-learning (ACDQL) model.

[0143] Optionally, the network device 101 can be regarded as an agent in the ACDQL model.

[0144] In some embodiments, the state information at the current time step t is the position of the network device 101 at the current time step t and the position of each terminal 102 at the current time step t, which can be represented as:

[0145] s t = (x uav (t), y uav (t), x1(t), y1(t), …, x N(t),y N (t))).

[0146] In some embodiments, the state information described above is input into the first model, and a first action (or move) (act) in the state at the current time step can be obtained. In embodiments of the present application, the action space is defined as the displacement of the network device 101 on the x-axis and y-axis at each time step, which can be represented by and respectively, so the first action performed in the state at the current time step t can be represented as:

[0147] wherein the displacement of the network device 101 on the x-axis and y-axis at each time step can take any value in the continuous range [-k, k], where k is the maximum speed of the network device 101.

[0148] Further, the goal is to maximize the total data rate of the users over the first period T, so any movement of the network device 101 that increases the total data rate of the endpoints (i.e. users) at time t compared to time t-1 will immediately receive a positive reward. Conversely, if the movement at time t results in a total data rate of the users that is less than at time t-1, a penalty will be incurred. The reward r corresponding to the first action described above can be determined based on the following formula t :

[0149] Based on equation (15), it can be seen that the network device 101 obtains a larger negative reward (i.e. penalty) when it moves outside the boundary.

[0150] In some embodiments, the aforementioned optimization problem can be represented as a Markov Decision Process (MDP), which can be solved using a Reinforcement Learning (RL) algorithm. The goal is to find the optimal policy π opt that maps each state to the action that maximizes the long-term reward. The policy function in the model should return the long-term cumulative reward that the network device 101 (i.e. the agent in the model) obtains by performing action a in state s. From the policy function, the network device 101 can choose the action at each time step that maximizes the long-term cumulative reward, thereby maximizing the total data rate of the users. Therefore, to solve the position optimization problem of the network device 101, only a policy needs to be implemented to discover the optimal policy function.

[0151] In some embodiments, the goal of the first model described above is to obtain the optimal policy function π θa that maximizes the long-term expected reward.

[0152] In some embodiments, a policy gradient method can be used to generate the optimal policy.

[0153] As an example, the policy gradient on the first model can be represented as:

[0154] where Q(s t ,a t ) is the quality of performing action a t in state s t , which is derived from the second model.

[0155] Further, since the algorithm in the embodiments of the present application is designed for continuous action space, the first model described above can use a Gaussian distribution to select actions, which is a well-known multi-parameter distribution of continuous random variables. More specifically, the first model in the embodiments of the present application can select the optimal action for each input state and output the mean (μ) and standard deviation (σ) of the Gaussian distribution. Therefore, according to the parameterized policy of the Gaussian distribution, it can be represented according to equation (17):

[0156] It should be noted that in the actor network, the standard deviation of the Gaussian distribution is generally considered to be a fixed value in the literature. However, in the embodiments of the present application, the standard deviation σ is taken as one of the outputs of the first model (the actor deep neural network model) so that the agent can dynamically balance between exploration and exploitation. By taking σ as one of the outputs of the first model, when the output σ value is larger, the agent can explore more, and when the output σ value is smaller, the agent will exploit more. With the convergence of the algorithm described above, the σ value output by the first model tends to zero.

[0157] In some embodiments, the first model described above uses a gradient ascent algorithm on the policy gradient, as shown in equation (16) above, in order to update its parameters and gradually converge to the optimal policy, so that the first model can iteratively update θ a , which can be represented as equation (18):

[0158] where α a is the learning rate of the first model.

[0159] In step S2103, the network device 101 determines the quality information of the first action based on the second model.

[0160] In some embodiments, the network device 101 can input the state information at the current time step, the first action described above, and the reward corresponding to the first action into the second model to obtain the quality information.

[0161] In some embodiments, the quality information is quality information corresponding to the first action.

[0162] In some embodiments, the quality information is used to indicate the quality of the first action performed by the network device 101 in the current time step in the state.

[0163] In some embodiments, the quality information is used to update the weight parameters in the first model.

[0164] In some embodiments, the quality information is used in a gradient ascent algorithm to update the weight parameters in the first model (for example, as shown in equation 18).

[0165] In some embodiments, the quality information of the first action can be represented as Q(s t ,a t ).

[0166] In some embodiments, the second model is a Critic deep neural network DNN model in an Actor-Critic Deep Q-Learning (ACDQL) model.

[0167] Optionally, the network device 101 can be regarded as an agent of the ACDQL model.

[0168] In some embodiments, the second model can use the Bellman equation to evaluate the quality of performing action a in state s, which is based on the expected discounted reward, which can be shown as equation (19):

[0169] where s' is the target state after performing action a in state s, a' is the action performed in state s', and γ is a discount factor quantifying the importance of future rewards.

[0170] Optionally, γ is a parameter that can be changed, and can predict future actions according to the current state.

[0171] In some embodiments, considering the random environment in the embodiments of the present application and the change of the value of Q(s,a) over time, a time difference (TD) is introduced in the second model, which is used to represent the difference between the newly predicted Q t (s,a) and the old Q t-1 (s,a). The time difference TD can be represented as equation (20):

[0172] Applying TD in the above equation (20) to the value function Q in the above equation (19) can convert the value function Q into equation (21):

[0173] wherein α c is a learning rate of the second model.

[0174] Further, in some embodiments, as can be seen from the above equation (21), the second model aims to minimize TD, and is taken as the true value or target of the second model, while Q t (s,a) is calculated as the predicted value of the second model. Therefore, in the second model, a gradient descent algorithm can be used to minimize the TD loss, as shown in equation (22):

[0175] L(θ c )=E[(y t -Q t (s,a)|θ c ) 2 ]。 (22)

[0176] wherein θ c stores the weight parameters of the above second model.

[0177] Further, θ c can be updated based on the calculated TD loss. Wherein the action value a satisfying the above equation (22) is determined by the first model.

[0178] In some embodiments, the above second model further includes an experience replay (ER) memory, and the ER memory stores the experience of each historical time step.

[0179] wherein optionally, the experience of one historical time step includes: state information at the above historical time step, the first action at the above historical time step, the reward corresponding to the first action at the above historical time step, and state information at the next time step of the above historical time step.

[0180] As an example, the experience stored in the ER memory can be represented as: (s1,a1,r1,s′2),(s2,a2,r2,s′3)…(s t ,a t ,r t ,s′ t+1 )。

[0181] In some embodiments, the ER memory can store all experiences of the agent as a circular buffer. By utilizing the ER memory, the second model will be able to select random batches of new and old experiences to train the DNN and minimize the TD loss. In some embodiments, the ER memory can help the agent solve the problem of similar consecutive interdependent states and enable the agent to converge quickly.

[0182] At step S2104, the network device 101 determines the position of the next time step.

[0183] In some embodiments, the network device 101 can determine the position of the network device 101 at the next time step based on the selected first action.

[0184] As an example, the algorithm based on the first model and the second model in the embodiments of the present application can be as shown in FIG. 2C.

[0185] As an example, the model iteration process in the embodiments of the present application can be as shown in FIG. 2D.

[0186] In some embodiments, the terms “eNB”, “gNB”, “base station”, “NG-RAN node” and the like can be replaced with each other.

[0187] In some embodiments, the terms “MME”, “CN”, “AMF”, “SMF” and the like can be replaced with each other.

[0188] In some embodiments, the terms “SGW”, “UPF” and the like can be replaced with each other.

[0189] In some embodiments, the determination or judgment can be made by a value represented by 1 bit (0 or 1), or by a true or false value (Boolean value) represented by true or false, or by comparison of numerical values (for example, comparison with a predetermined value), but is not limited thereto.

[0190] In some embodiments, the terms “carrier”, “band”, “frequency” and the like can be replaced with each other.

[0191] In some embodiments, the names of information and the like are not limited to the names described in the embodiments, and terms such as "information", "message", "signal", "signaling", "report", "configuration", "indication", "instruction", "command", "channel", "parameter", "field", "symbol", "symbol", "codebook", "codeword", "codepoint", "bit", "data", "program", "chip", and the like can be replaced with each other.

[0192] In some embodiments, the names of information and the like are not limited to the names described in the embodiments, and terms such as "information", "message", "signal", "signaling", "report", "configuration", "indication", "instruction", "command", "channel", "parameter", "field", "symbol", "symbol", "codebook", "codeword", "codepoint", "bit", "data", "program", "chip", and the like can be replaced with each other.

[0193] In some embodiments, terms such as "physical downlink shared channel (PDSCH)", "DL data", and the like can be replaced with each other, and terms such as "physical uplink shared channel (PUSCH)", "UL data", and the like can be replaced with each other.

[0194] In some embodiments, the terms "radio", "wireless", "radio access network (RAN)", "access network (AN)", "RAN-based", and the like can be replaced with each other.

[0195] In some embodiments, the terms "resource block (RB)", "physical resource block (PRB)", "sub-carrier group (SCG)", "resource element group (REG)", "PRB pair", "RB pair", "resource element (RE)", "sub-carrier", and the like can be replaced with each other.

[0196] In some embodiments, the terms "acquire", "obtain", "get", "receive", "transmit", "bidirectional transmission", "send and / or receive", and the like can be replaced with each other, which can be interpreted as receiving from other subjects, acquiring from protocols, obtaining from higher layers, obtaining by self-processing, implementing autonomously, and the like.

[0197] In some embodiments, the terms "send", "transmit", "report", "issue", "transmit", "bidirectional transmission", "send and / or receive", and the like can be replaced with each other.

[0198] In some embodiments, the terms "certain", "preset", "pre-set", "set", "indicated", "a certain", "any", "first", and the like can be replaced with each other, and "certain A", "preset A", "pre-set A", "set A", "indicated A", "a certain A", "any A", "first A" can be interpreted as A specified in advance in protocols and the like, can be interpreted as A obtained by setting, configuring, or indicating, and the like, can be interpreted as certain A, a certain A, any A, or first A, and the like, but are not limited thereto.

[0199] The communication method related to the embodiments of the present disclosure can include at least one of steps S2101-S2104. For example, step 2101 can be implemented as an independent embodiment, step 2102 can be implemented as an independent embodiment, step 2103 can be implemented as an independent embodiment, step 2104 can be implemented as an independent embodiment, steps 2101+2102 can be implemented as an independent embodiment, steps 2102+2103 can be implemented as an independent embodiment, steps 2103+2104 can be implemented as an independent embodiment, steps 2102+2103+2104 can be implemented as an independent embodiment, steps 2101+2102+2103 can be implemented as an independent embodiment, steps 2101+2102+2103+2104 can be implemented as an independent embodiment, and the like, but are not limited thereto.

[0200] In some embodiments, steps S2101 and S2102 can be exchanged in order or performed simultaneously.

[0201] In some embodiments, steps S2103 and S2104 are optional, and one or more of these steps can be omitted or replaced in different embodiments.

[0202] In some embodiments, other optional implementations described before or after the description of FIG. 2A can be referred to.

[0203] FIG. 3A is a flow diagram of a non-terrestrial communication method according to an embodiment of the present disclosure. As shown in FIG. 3A, the embodiments of the present disclosure relate to a non-terrestrial communication method, and the method is performed by the network device 101, and the method includes:

[0204] In step S3101, the data transmission rate of each terminal 102 at the current time step is obtained.

[0205] Optional implementations of step S3101 can be referred to optional implementations of step S2101 of FIG. 2A and other associated parts in the embodiments related to FIG. 2A, which will not be described here.

[0206] In step S3102, a first action is determined based on a first model.

[0207] Optional implementations of step S3102 can be referred to optional implementations of step S2102 of FIG. 2A and other associated parts in the embodiments related to FIG. 2A, which will not be described here.

[0208] In step S3103, quality information of the first action is determined based on a second model.

[0209] The optional implementation of step S3103 can be found in the optional implementation of step S2103 in Figure 2A and other related parts in the embodiments involved in Figure 2A, which will not be repeated here.

[0210] Step S3104: Determine the position of the next time step.

[0211] The optional implementation of step S3104 can be found in the optional implementation of step S2104 in Figure 2A and other related parts in the embodiments involved in Figure 2A, which will not be repeated here.

[0212] The communication method involved in the embodiments of this disclosure may include at least one of steps S3101 to S3104. For example, step 3101 may be implemented as an independent embodiment, step 3102 may be implemented as an independent embodiment, step 3103 may be implemented as an independent embodiment, step 3104 may be implemented as an independent embodiment, steps 3101+3102 may be implemented as an independent embodiment, steps 3102+3103 may be implemented as an independent embodiment, steps 3103+3104 may be implemented as an independent embodiment, steps 3101+3102+3103 may be implemented as an independent embodiment, steps 3101+3102+3103+3104 may be implemented as an independent embodiment, etc., but not limited thereto.

[0213] In some embodiments, steps S3101 and S3102 may be performed in an alternate order or simultaneously.

[0214] In some embodiments, steps S3103 and S3104 are optional, and one or more of these steps may be omitted or substituted in different embodiments.

[0215] Figure 3B is a flowchart illustrating a non-terrestrial communication method according to an embodiment of the present disclosure. As shown in Figure 3B, this embodiment of the disclosure relates to a non-terrestrial communication method, which is executed by access network device 101, and includes:

[0216] Step S3201: Obtain the data transmission rate of each terminal 102 at the current time step.

[0217] The optional implementation of step S3201 can be found in step S2101 of Figure 2A, the optional implementation of step S3101 of Figure 3A, and other related parts in the embodiments involved in Figures 2A and 3A, which will not be repeated here.

[0218] Step S3202: Based on the first model, determine the first action.

[0219] The optional implementation of step S3202 can refer to the optional implementation of step S2102 in FIG. 2A, step S3102 in FIG. 3A, and other associated parts in the embodiments related to FIG. 2A and FIG. 3A, which will not be repeated here.

[0220] In step S3203, the position of the next time step is determined.

[0221] The optional implementation of step S3203 can refer to the optional implementation of step S2104 in FIG. 2A, step S3104 in FIG. 3A, and other associated parts in the embodiments related to FIG. 2A and FIG. 3A, which will not be repeated here.

[0222] The communication method related to the embodiments of the present disclosure can include at least one of steps S3201-S3203. For example, step 3201 can be implemented as an independent embodiment, step 3202 can be implemented as an independent embodiment, step 3203 can be implemented as an independent embodiment, steps 3201+3202 can be implemented as an independent embodiment, steps 3202+3203 can be implemented as an independent embodiment, steps 3201+3202+3203 can be implemented as an independent embodiment, and the like, but are not limited thereto.

[0223] The following is an exemplary introduction to the method described in the above embodiments.

[0224] In the embodiments of the present application, in the case of user movement (i.e. the terminal 102 in the foregoing embodiments of the present application), a dynamic positioning scheme for UAV-BS (i.e. the network device 101 in the foregoing embodiments of the present application) based on reinforcement learning (RL) is developed.

[0225] In the embodiments of the present application, a new reward function based on the RL algorithm is proposed, on the basis of which the UAV-BS agent obtains some positive or negative rewards after selecting and executing each action. The proposed reward function aims to keep the UAV base station within the boundary of the region of interest and maximize the total data rate of the user.

[0226] In the embodiments of the present application, a natural disaster scenario is considered, in which the ground base stations are destroyed and the UAV-BS should provide downlink internet connectivity for the ground endpoints. However, the system model can be extended to scenarios such as over-crowded events where UAV-BS and ground BS coexist. In the case of coexistence of ground and air BSs, each user is served by either a ground BS or a UAV-BS according to the received signal quality. It is assumed that the area where the users are located is a D x D square kilometer area, and the initial location of the users is randomly selected within the entire area. As shown in FIG. 2B, FIG. 2B is a schematic diagram of a single UAV-BS based model provided by the embodiments of the present application.

[0227] In the initial stage, the position of the UAV-BS in the air is random at a fixed height h uav . There are N users in total, and the two-dimensional coordinates of each ground user endpoint i are denoted as (x i ,y i ), and (x uav ,y uav ,h uav ) represents the three-dimensional coordinates of the UAV-BS. It is assumed that there is a Line of Sight (LoS) and Non-Line of Sight (NLoS) connection environment at the same time. The user is movable, and its migration model follows the Gaussian-Markov migration model. In the Gaussian-Markov migration model, an initial random speed and direction are assigned to each user endpoint.

[0228] In some embodiments, the UAV-BS is regarded as an ACDQL agent and is equipped with an ACDQL-based algorithm. At each time step t, the UAV-BS selects the best motion in the x coordinate and the y coordinate, respectively, with the purpose of maximizing the user data rate. The actions selected in the x and y coordinates can take any value in the range [-k, k], where the value of k depends on the maximum speed of the UAV-BS.

[0229] First, Table 1 below gives the parameter symbols used in the embodiments of the present application and their descriptions.

[0230] Table 1 Parameter symbols used in the embodiments of the present application and corresponding descriptions

[0231] In some embodiments, generally one UAV-BS is not enough to serve all users, and the system model can be extended to have multiple UAV-BSs, where each UAV-BS is regarded as an ACDQL-based agent. For the convenience of description, the embodiments take one UAV-BS as an example for illustration.

[0232] Channel model:

[0233] In some embodiments, a path loss model can be employed to model the path loss between the UAV-BS and the ground terminals, which depends on the LoS probability and NLoS probability between the terminals and the UAV-BS. Since the UAV-BSs are deployed in the three-dimensional space above the ground, the probability of establishing a LoS connection with the ground users is high. The line-of-sight probability between the UAV-BS and the ground endpoints can be expressed as Equation (1):

[0234] where k1 and k2 are set according to the environment type, θ i is the elevation angle between the UAV-BS and the i-th user. The elevation angle calculation formula is shown in Equation (2).

[0235] θ i = arctan(h uav / r i ). (2)

[0236] where r i is the horizontal distance between the i-th endpoint and the UAV-BS, which is obtained from Equation (3).

[0237] The path loss related to the LoS connection is calculated as Equation (4).

[0238] where f c is the carrier frequency, c is the speed of light, and δ LoS is the environment-dependent path loss added to the LoS connection.

[0239] Due to the presence of obstacles in the environment such as high-rise buildings, trees, etc., especially in urban and dense urban environments, some signals from the UAV-BS reach the user through NLoS links. The NLoS probability between the UAV-BS and the i-th endpoint is complementary to the LoS probability, as shown in Equation (5).

[0240] P i(NLoS) = 1 - P i(LoS) . (5)

[0241] The path loss related to the NLoS connection is shown in Equation (6), where δ NLoS is the additional environment-dependent path loss related to the NLoS connection.

[0242] Finally, the average path loss from the UAV-BS to the i-th user is obtained, as shown in Equation (7).

[0243] L i(avg) = P i(LoS) x L i(LoS)+P i(NLoS) ×L i(NLoS) 。 (7)

[0244] Optimization problem formulation:

[0245] In some embodiments, the UAV-BS should constantly change its position and move towards the position of maximum system throughput, where the system throughput is defined as the data rate achieved by the ground endpoints.

[0246] Therefore, the goal is to move the UAV-BS on a path that maximizes the total data rate of users over a time period T (i.e., the first period in the aforementioned embodiments of the present application) at a time step t, which can be represented as:

[0247] The constraint in equation (9) ensures that the total allocated power from the UAV-BS to the endpoints does not exceed the transmit power of the UAV-BS, which is represented by p uav .

[0248] 0≤x uav ≤x max ,

[0249] 0≤y uav ≤y max . (10)

[0250] Subsequently, the constraint in equation (10) ensures that the UAV-BS does not fly out of the predefined boundaries.

[0251] R i is the data transmission rate between the UAV-BS and the i-th user, which can be calculated as shown in equation (11) below, where B is the total bandwidth of the UAV-BS allocated to the endpoints on average.

[0252] γ i is the signal-to-noise ratio (SNR) of the i-th user received by the UAV-BS, which can be represented as equation (12).

[0253] In the equation, σ noise is the noise power, G i is the power gain of the UAV-BS to the i-th endpoint, which can be calculated as equation (13).

[0254] where and are the antenna gains of the UAV-BS transmitting and receiving to the i-th user. d0 is the far-field reference distance, d i is the distance from the i-th endpoint to the UAV-BS, which can be calculated as shown in equation (14).

[0255] The objective function presented in equation (8) is a Mixed-Integer Nonlinear Programming (MINLP) formulation whose aim is to maximize the sum of data rates of users in a long-term operation T. The MINLP problem is proven to be a non-deterministic polynomial hard problem.

[0256] Therefore, to solve the non-deterministic polynomial hard problem of maximizing the sum of data rates of ground endpoints in a UAV-assisted cellular network, a solution of Actor-Critic Deep Q-Learning (ACDQL) model is proposed.

[0257] ACDQL-BASED UAV-BS deployment algorithm:

[0258] To maximize the network throughput, i.e., the sum of data rates of users, at each time step t, the UAV-BS needs to perceive the state of the environment and decide what motion is best to take. Since the decision of the next position of the UAV-BS depends on the current state of the environment, solving the UAV-BS three-dimensional position optimization problem can be represented as a Markov Decision Process (MDP) and can be solved using Reinforcement Learning (RL) algorithms.

[0259] The MDP consists of four main elements (S, A, π, R) that should be defined for the environment so that it can be solved using RL. S is the state space that contains all possible states of the environment, while A is the action space that contains all actions that the RL agent can take. Here, π represents the policy function, i.e., the mapping from states to probability distributions over actions, and R represents the reward function.

[0260] The goal is to find the optimal policy π opt that maps each state to an action that maximizes the long-term reward.

[0261] In some embodiments, with respect to the three-dimensional deployment of the UAV-BS in the embodiments of the present application, the MDP elements are as follows.

[0262] State space: The state space in the model is represented as the position of the UAV-BS and the positions of all ground endpoints at each time step t, which is represented as:

[0263] s t = (x uav (t), y uav(t), x1(t), y1(t),..., x N (t), y N (t))).

[0264] Action space: The action space of the model is defined as the displacement of the UAV-BS in the x-axis and y-axis at each time step t, denoted by and , respectively. The displacement of the UAV-BS along the x and y coordinates at each time step can take any value in the continuous range [-k, k], where k is the maximum speed of the UAV-BS. Therefore, the action space of the proposed method can be represented as:

[0265] Reward function: The objective of the model is to maximize the total data rate of users over the time period T, therefore any movement of the UAV-BS that increases the total data rate of the endpoints (i.e. users) at time t compared to time t-1 will immediately receive a positive reward. Conversely, if the movement of the agent at time t results in a total data rate of users that is less than at time t-1, then a penalty will be incurred. Therefore, the reward function is represented as:

[0266] In equation (15), the larger negative reward (i.e. penalty) received by the agent when moving outside the boundaries defines the boundary constraints of the UAV-BS agent to comply with the aforementioned optimization problem presented in equation (10).

[0267] Policy function: The policy function in the model should return the long-term cumulative reward that the UAV-BS agent would receive by performing action a in state s. From the policy function, the agent can choose the action at each time step that maximizes the long-term cumulative reward, and thus maximizes the total data rate of users. Therefore, to solve the position optimization problem of the UAV-BS, only a policy needs to be implemented to discover the optimal policy function.

[0268] In some embodiments, the proposed derivation strategy of the optimal policy function is based on the continuous ACDQL algorithm. The actor-critic method consists of two deep neural networks (DNNs), namely the actor DNN (i.e. the first model in the aforementioned embodiments of the present application) and the critic DNN (i.e. the second model in the aforementioned embodiments of the present application). The actor DNN receives the environment state as its input and maps it to a probability distribution over actions a, where a stores the weights of the actor DNN to predict what action is the best choice. In other words, the actor network represents the required policy function.

[0269] On the other hand, the critic DNN implements a value function of the form Q(s, a) that evaluates the quality of the action a chosen by the actor network in state s.

[0270] As an example, as shown in FIG. 2C, FIG. 2C is an example of a UAV-BS deployment algorithm based on ACDQL proposed in embodiments of the present application.

[0271] Actor Network: The goal of the actor network is to obtain the optimal policy function that maximizes the long-term expected reward Therefore, a policy gradient method is used in the actor network to generate the optimal policy. The policy gradient on the actor DNN can be derived by equation (16), where Q(s t ,a t ) is the quality of performing action a in state s t , which is computed by the critic network.

[0272] In some embodiments, since the UAV-BS deployment algorithm proposed in embodiments of the present application is designed for continuous action space, the actor network uses a Gaussian distribution to select actions, which is a well-known multi-parameter distribution of continuous random variables. More specifically, the actor network proposed in embodiments of the present application selects the optimal action by each input state, outputting the mean (μ) and standard deviation (σ) of the Gaussian distribution. Therefore, the parameterized policy according to the Gaussian distribution can be represented according to equation (17):

[0273] In the actor network, the standard deviation of the Gaussian distribution is generally considered a fixed value in the literature. However, in embodiments of the present application, σ is taken as one of the outputs of the actor DNN, so that the agent can dynamically balance between exploration and exploitation. By taking σ as one of the outputs of the actor DNN, when the value of σ is larger, the agent can perform more exploration, and when the value of σ is smaller, it will perform more exploitation. As the UAV-BS deployment algorithm converges, the value of σ tends to zero.

[0274] In some embodiments, the actor network uses a gradient ascent technique on the policy gradient, as shown in equation (16), in order to update its parameters and gradually converge to the optimal policy, so that the actor network iteratively updates θ a , which is represented by equation (18), where a a is the learning rate of the actor DNN.

[0275] Critic Network: The critic network is used to evaluate the quality of the action selected by the actor network, which uses the Bellman equation to evaluate the quality of performing action a in state s, which is based on the expected discounted reward, denoted as equation (19), where s' is the target state after performing action a in state s, and γ is a discount factor quantifying the importance of future rewards.

[0276] In some embodiments, considering the stochastic environment in the embodiments of the present application and the change of the value of Q(s, a) over time, a time difference (TD) is introduced in the critic network to represent the difference between the newly predicted Q t (s, a) and the old Q t-1 (s, a). The TD can be represented as equation (20).

[0277] By applying TD in equation (19), the value function Q is converted to equation (21), where α c represents the learning rate of the critic DNN.

[0278] As can be seen from equation (21), the DNN of the critic network aims to minimize TD, taking as the true value or target of the network, while Q t (s, a) is calculated as the predicted value of the network. Therefore, the gradient descent technique is used in the critic network to minimize the TD loss, as shown in equation (22), where θ c stores the weights of the critic DNN.

[0279] L(θ c )=E[(y t -Q t (s,a)|θ c ) 2 ]。 (22)

[0280] Then, θ c is optionally updated with the calculated TD loss. The action value that satisfies equation (22) is determined by the actor network.

[0281] In addition, in some embodiments of the embodiments of the present application, an experience replay (ER) memory is used in the critic network, which is a circular buffer to store (s t ,a t ,r t ,s′ t+1All experiences of the agent are stored in the form of ER memory. By utilizing the ER memory, the critic network will be able to select random batches of new and old experiences to train the DNN and minimize the TD loss. The ER memory can help the agent to solve the problem of similar continuous interdependent states and enable the agent to converge quickly.

[0282] As an example, the flow chart as shown in FIG. 2D illustrates the steps of the continuous policy based on ACDQL proposed in the embodiments of the present application.

[0283] The embodiments of the present disclosure also propose a device for implementing any of the above methods, for example, a device comprising units or modules for implementing the steps performed by the terminal in any of the above methods. For another example, another device is also proposed, comprising units or modules for implementing the steps performed by the network equipment (such as access network equipment, core network function node, core network equipment, etc.) in any of the above methods.

[0284] It should be understood that the division of each unit or module in the above apparatus is only a logical function division, and all or part of them can be integrated into a physical entity or physically separated in actual implementation. In addition, the units or modules in the apparatus can be implemented in the form of processor calling software: for example, the apparatus includes a processor, the processor is connected with a memory, the memory stores instructions, and the processor calls the instructions stored in the memory to realize any of the above methods or realize the functions of each unit or module of the above apparatus, wherein the processor is a general processor such as a central processing unit (CPU) or a microprocessor, and the memory is a memory in the apparatus or a memory outside the apparatus. Alternatively, the units or modules in the apparatus can be implemented in the form of hardware circuit, and the functions of part or all of the units or modules can be realized by the design of hardware circuit. The above hardware circuit can be understood as one or more processors; for example, in one implementation, the above hardware circuit is an application-specific integrated circuit (ASIC), and the functions of part or all of the units or modules are realized by the design of the logical relationship of elements in the circuit; for another example, in another implementation, the above hardware circuit is a programmable logic device (PLD), and a field programmable gate array (FPGA) is taken as an example, which can include a large number of logic gate circuits, and the connection relationship between the logic gate circuits is configured by a configuration file, so as to realize the functions of part or all of the above units or modules. All units or modules of the above apparatus can be all implemented in the form of processor calling software, or all implemented in the form of hardware circuit, or part implemented in the form of processor calling software and the remaining part implemented in the form of hardware circuit.

[0285] In the embodiments of the present disclosure, the processor is a circuit with signal processing capability. In one implementation, the processor can be a circuit with instruction reading and running capability, such as a central processing unit (CPU), a microprocessor, a graphics processing unit (GPU) (which can be understood as a microprocessor), a digital signal processor (DSP), or the like. In another implementation, the processor can implement certain functions through a logical relationship of hardware circuits, and the logical relationship of the hardware circuits is fixed or can be reconfigured. For example, the processor is a hardware circuit implemented by an application-specific integrated circuit (ASIC) or a programmable logic device (PLD), such as an FPGA. In the reconfigurable hardware circuit, the processor loads a configuration document to implement the configuration of the hardware circuit. It can be understood that the processor loads instructions to implement the functions of the above part or all units or modules. In addition, it can also be a hardware circuit designed for artificial intelligence, which can be understood as an ASIC, such as a neural network processing unit (NPU), a tensor processing unit (TPU), a deep learning processing unit (DPU), and the like.

[0286] FIG. 4 is a structural schematic diagram of a network device according to an embodiment of the present disclosure. As shown in FIG. 4, the network device 4100 can include at least one of a transceiver module 4101, a processing module 4102, and the like. In some embodiments, the processing module 4102 is configured to obtain a data transmission rate between each terminal in at least one terminal and the network device at a current time step; the processing module 4102 is further configured to input state information at the current time step into a first model to obtain a first action performed by the network device in the state at the current time step, wherein the first action is used to indicate a displacement of the network device at the current time step; and the processing module 4102 is further configured to determine a position of the network device at a next time step based on the first action. The first model is used to determine the position of the network device at each time step in a first period, and a total data transmission rate of the at least one terminal in the first period is maximum.

[0287] Optionally, the state information at the current time step comprises a position of the network device at the current time step and a position of each terminal at the current time step.

[0288] Optionally, the processing module 4102 is further configured to:

[0289] input the state information at the current time step, the first action, and the reward corresponding to the first action into a second model to obtain quality information;

[0290] The quality information is used to indicate a quality of performing the first action in the state at the current time step.

[0291] Optionally, the reward comprises a positive reward and a negative reward; wherein,

[0292] The reward corresponding to the first action that increases the total data transmission rate of the terminal at the current time step compared with the previous time step is a positive reward; or,

[0293] The reward corresponding to the first action that decreases the total data transmission rate of the terminal at the current time step compared with the previous time step is a negative reward; or,

[0294] The reward corresponding to the first action that makes the network device exceed a predefined boundary is a negative reward.

[0295] Optionally, the quality information is further used to update a weight parameter of the first model.

[0296] Optionally, the second model further comprises an experience replay (ER) memory, and the ER memory stores an experience of each historical time step;

[0297] The experience of a historical time step comprises state information at the historical time step, a first action at the historical time step, a reward corresponding to the first action at the historical time step, and state information at a next time step of the historical time step.

[0298] Optionally, the first model and the second model are actor-critic deep Q-learning (ACDQL) models; the first model is an actor deep neural network model, and the second model is a critic deep neural network model.

[0299] Optionally, the processing module 4102 is further configured to:

[0300] The weight parameter in the first model is updated by using a policy gradient ascent algorithm.

[0301] Optionally, the processing module 4102 is further configured to:

[0302] The weight parameters in the second model are updated using a gradient descent algorithm.

[0303] Optionally, the transceiver module is configured to perform at least one of the communication steps (e.g., receiving and / or sending) performed by the network device 101 in any of the above methods.

[0304] Optionally, the processing module is configured to perform at least one of the other steps (e.g., steps S2101, S2102, S2103, S2104, but not limited thereto) performed by the network device 101 in any of the above methods.

[0305] FIG. 5A is a structural schematic diagram of a communication device 5100 according to an embodiment of the present disclosure. The communication device 5100 can be a network device (e.g., an access network device, a core network device, etc.), a terminal (e.g., a user equipment, etc.), a chip, a chip system, or a processor supporting the network device to implement any of the above methods, or a chip, a chip system, or a processor supporting the terminal to implement any of the above methods. The communication device 5100 can be used to implement the methods described in the above method embodiments, and details can be referred to the descriptions in the above method embodiments.

[0306] As shown in FIG. 5A, the communication device 5100 includes one or more processors 5101. The processor 5101 can be a general-purpose processor or a special-purpose processor, for example, a baseband processor or a central processing unit. The baseband processor can be used to process communication protocols and communication data, and the central processing unit can be used to control the communication device (e.g., a base station, a baseband chip, a terminal device, a terminal device chip, a DU or a CU, etc.), execute programs, and process data of the programs. The communication device 5100 is configured to implement any of the above methods.

[0307] In some embodiments, the communication device 5100 further includes one or more memories 5102 configured to store instructions. Optionally, all or part of the memory 5102 can also be located outside the communication device 5100.

[0308] In some embodiments, the communication device 5100 further includes one or more transceivers 5103. When the communication device 5100 includes one or more transceivers 5103, the transceiver 5103 performs at least one of the communication steps (e.g., receiving and / or sending) in the above methods, and the processor 5101 performs at least one of the other steps.

[0309] In some embodiments, the transceiver can include a receiver and / or a transmitter, which can be separate or integrated together. Optionally, the terms transceiver, transceiving unit, transceiver, transceiving circuit, etc. can be replaced by each other, the terms transmitter, transmitting unit, transmitter, transmitting circuit, etc. can be replaced by each other, and the terms receiver, receiving unit, receiver, receiving circuit, etc. can be replaced by each other.

[0310] In some embodiments, the communication device 5100 can include one or more interface circuits 5104. Optionally, the interface circuit 5104 is connected with the memory 5102, and the interface circuit 5104 can be used to receive signals from the memory 5102 or other devices, and can be used to send signals to the memory 5102 or other devices. For example, the interface circuit 5104 can read instructions stored in the memory 5102 and send the instructions to the processor 5101.

[0311] The communication device 5100 described in the above embodiments can be a network device or a terminal, but the scope of the communication device 5100 described in the present disclosure is not limited thereto, and the structure of the communication device 5100 can not be limited by Figure 5A. The communication device can be a standalone device or can be part of a larger device. For example, the communication device can be: (1) a standalone integrated circuit (IC), or a chip, or a chip system or subsystem; (2) a set of one or more ICs, which can optionally also include storage components for storing data, programs; (3) an ASIC, such as a Modem; (4) a module that can be embedded in other devices; (5) a receiver, a terminal device, a smart terminal device, a cellular phone, a wireless device, a handset, a mobile unit, a vehicle-mounted device, a network device, a cloud device, an artificial intelligence device, etc.; (6) other devices, etc.

[0312] Figure 5B is a structural schematic diagram of a chip 5200 according to an embodiment of the present disclosure. For the case where the communication device 5100 is a chip or a chip system, the structural schematic diagram of the chip 5200 shown in Figure 5B can be referred to, but is not limited thereto.

[0313] The chip 5200 includes one or more processors 5201, and the chip 5200 is configured to execute any of the above methods.

[0314] In some embodiments, the chip 5200 further includes one or more interface circuits 5202. Optionally, the interface circuit 5202 is connected with the memory 5203, and the interface circuit 5202 can be used to receive signals from the memory 5203 or other devices, and can be used to send signals to the memory 5203 or other devices. For example, the interface circuit 5202 can read instructions stored in the memory 5203 and send the instructions to the processor 5201.

[0315] In some embodiments, the interface circuit 5202 performs at least one of the communication steps of sending and / or receiving in the above-described methods, and the processor 5201 performs at least one of the other steps.

[0316] In some embodiments, the terms interface circuit, interface, transceiver pin, transceiver, etc. can be replaced by each other.

[0317] In some embodiments, the chip 5200 further includes one or more memories 5203 for storing instructions. Optionally, all or part of the memories 5203 can be outside the chip 5200.

[0318] The disclosure also proposes a storage medium having instructions stored thereon, which, when executed on the communication device 5100, causes the communication device 5100 to perform any of the above methods. Optionally, the storage medium is an electronic storage medium. Optionally, the storage medium is a computer-readable storage medium, but is not limited to this, and it can also be a storage medium readable by other devices. Optionally, the storage medium can be a non-transitory storage medium, but is not limited to this, and it can also be a transitory storage medium.

[0319] The disclosure also proposes a program product, which, when executed by the communication device 5100, causes the communication device 5100 to perform any of the above methods. Optionally, the program product is a computer program product.

[0320] The disclosure also proposes a computer program, which, when executed on a computer, causes the computer to perform any of the above methods.

[0321] In the embodiments described above, all or part of the system, device, and unit can be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part of the system, device, and unit can be implemented in the form of a computer program product. The computer program product includes one or more computer programs. When the computer programs are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present disclosure are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer programs can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another computer-readable storage medium, for example, the computer programs can be transferred from one website, computer, server, or data center to another website, computer, server, or data center through a wired (for example, coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (for example, infrared, wireless, microwave, etc.) manner. The computer-readable storage medium can be any available medium accessible by a computer or a data storage device such as a server, data center, etc. integrated with one or more available media. The available media can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a high-density digital video disc (DVD)), or a semiconductor medium (for example, a solid state disk (SSD)), etc.

[0322] Those skilled in the art can appreciate that the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether the functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. A skilled person can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present disclosure.

[0323] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the system, device, and unit described above can refer to the corresponding processes in the foregoing method embodiments, which will not be described here.

[0324] The above is only a specific implementation of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical range disclosed in the present disclosure, which should be covered within the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be subject to the protection scope of the claims.

Claims

1. A non-terrestrial communication method, characterized by, The method is performed by a network device, and the method comprises: obtaining a data transmission rate between each of at least one terminal and the network device at a current time step; inputting state information at the current time step into a first model to obtain a first action performed by the network device in the state at the current time step, wherein the first action is used to indicate a displacement of the network device at the current time step; determining a position of the network device at a next time step based on the first action; the first model is used to determine the position of the network device at each time step in a first period, wherein a total data transmission rate of the at least one terminal in the first period is maximum.

2. The method of claim 1, wherein, The state information at the current time step is the position of the network device at the current time step and the position of each terminal at the current time step.

3. The method of claim 2, wherein, The method further comprises: inputting the state information at the current time step, the first action, and a reward corresponding to the first action into a second model to obtain quality information; wherein the quality information is used to indicate a quality of performing the first action in the state at the current time step.

4. The method of claim 3, wherein, The reward comprises a positive reward and a negative reward; wherein, the reward corresponding to the first action that increases the total data transmission rate of the terminal at the current time step compared with a previous time step is a positive reward; or, the reward corresponding to the first action that reduces the total data transmission rate of the terminal at the current time step compared with the previous time step is a negative reward; or, the reward corresponding to the first action that makes the network device exceed a predefined boundary is a negative reward.

5. The method of claim 3, wherein, The quality information is further used to update a weight parameter of the first model.

6. The method of claim 3, wherein, The second model further comprises an experience replay (ER) memory, and the ER memory stores an experience of each historical time step; wherein the experience of a historical time step comprises state information at the historical time step, a first action at the historical time step, a reward corresponding to the first action at the historical time step, and state information of a next time step of the historical time step.

7. The method of any one of claims 1-6, wherein the first model and the second model are actor-critic deep Q-learning (ACDQL) models; wherein the first model is an actor deep neural network model, and the second model is a critic deep neural network model.

8. The method of claim 7, wherein, The method further comprises: updating the weight parameter in the first model by using a policy gradient ascent algorithm.

9. The method of claim 7, wherein, The method further comprises: updating the weight parameter in the second model by using a gradient descent algorithm.

10. A network device, comprising: The network device comprises: a processing module configured to obtain a data transmission rate between each of at least one terminal and the network device at a current time step; the processing module is further configured to input state information at the current time step into a first model to obtain a first action performed by the network device in the state at the current time step, wherein the first action is used to indicate a displacement of the network device at the current time step; The processing module is further configured to determine a position of the network device at a next time step based on the first action. The first model is configured to determine a position of the network device at each time step in a first period, wherein a total data transmission rate of the at least one terminal in the first period is maximum.

11. A network device, comprising: The network device comprises: one or more processors; The network device is configured to perform the non-terrestrial communication method of any one of claims 1-9.

12. A storage medium, the storage medium storing instructions, wherein, The instructions, when executed on the communication device, cause the communication device to perform the non-terrestrial communication method of any one of claims 1-9.

Images