Communication method based low earth orbit satellite and computing device for executing the same

Abstract

A communication method for a low-orbit satellite and a computing device for performing the same are disclosed. A communication method according to one disclosed embodiment is a communication method of a communication system comprising a plurality of satellites having a Reconfigurable Intelligent Surface (RIS) panel, a plurality of remote user terminals, and one or more base station devices, and a communication method performed in a computing device having one or more processors and a memory storing one or more programs executed by said one or more processors, wherein when a remote user terminal receives a signal transmitted from a satellite, the method includes the steps of: calculating a received signal of the remote user terminal; calculating a data rate of the remote user terminal using the Signal to Noise Ratio (SNR) of the remote user terminal for the received signal; and optimizing parameters that are pre-set to maximize the data rate of the remote user terminal.

Description

Technology Field

[0001] Embodiments of the present invention relate to low-orbit satellite-based communication technology. Background Technology

[0003] Recently, the number of IoT devices has been rapidly increasing due to the development of IoT (Internet of Things) technology. Utilizing networks of these IoT devices can increase efficiency in various fields such as transportation, healthcare, industry, energy, and marine, and improve people's quality of life. However, internet access remains restricted in certain regions of the world, and Low Earth Orbit (LEO) satellite networks are being developed to address this issue.

[0004] However, satellite communication resources for Remote User Equipment (RUE) are limited due to uneven geographical distribution and dynamic ground traffic demand characteristics. In particular, there is a problem in that the signal transmission distance and coverage range of Low Earth Orbit satellites are restricted in the sub-terahertz (THz) frequency band due to high signal attenuation, molecular absorption, and spatial loss. Therefore, measures are required to maximize the coverage and data transmission rate of Low Earth Orbit satellites. Prior art literature

[0006] Korean Registered Patent Publication No. 10-2393301 (May 2, 2022) The problem to be solved

[0007] An embodiment of the present invention is to provide a low-orbit satellite-based communication method capable of maximizing the data rate for a remote user terminal of a low-orbit satellite, and a computing device for performing the same. means of solving the problem

[0009] A communication method according to one disclosed embodiment is a communication method of a communication system comprising a plurality of satellites having a Reconfigurable Intelligent Surface (RIS) panel, a plurality of remote user terminals, and one or more base station devices, and is a communication method performed in a computing device having one or more processors and a memory storing one or more programs executed by said one or more processors, wherein when a remote user terminal receives a signal transmitted from a satellite, the method comprises: a step of calculating a received signal of the remote user terminal; a step of calculating a data rate of said remote user terminal using the Signal to Noise Ratio (SNR) of said remote user terminal with respect to said received signal; and a step of optimizing parameters that are pre-set to maximize the data rate of said remote user terminal.

[0010] The above received signal (y u ) can be calculated through the following mathematical formula.

[0011] (Mathematical formula)

[0012]

[0013] L tot : Total loss until the signal is received by the remote user terminal

[0014] p u : Transmission power of base station device

[0015] g u : Channel from the last RIS panel to the remote user terminal

[0016] : Transmission vector of base station device

[0017] R s : Total number of RIS panels

[0018] z u AWGN (Additive White Gaussian Noise) received from a remote user terminal

[0019] r : RIS index

[0020] : Channel matrix between base station device and RIS panel r

[0021] : Phase shift of RIS panel r

[0022] SNR(Signal to Noise Ratio)( ) can be calculated using the following mathematical formula.

[0023] (Mathematical formula)

[0024]

[0025] p s,u : Power for transmitting signals from a satellite to a remote user terminal

[0026] : Elements of the association matrix for remote user terminals

[0027] N0: Noise spectral density

[0028] : Beamforming vector of base station device

[0029] The data rate (R) of the above remote user terminal u,t ) can be calculated using the following mathematical formula.

[0030] (Mathematical formula)

[0031]

[0032] B u : Total bandwidth available on the remote user terminal

[0033] The step of optimizing the pre-set parameters includes power (p) for transmitting a signal from the satellite to the remote user terminal to satisfy the objective function of the following mathematical formula, and the association matrix between the satellite's RIS panel and the remote user terminal ( ), phase shift of the satellite's RIS panel ( ), and the elevation angle of the satellite ( ) can be optimized.

[0034] (Mathematical formula)

[0035]

[0036] R u,t : Data rate of remote user terminal u in time slot t

[0037] T: Total number of time slots

[0038] U : Total number of remote user terminals

[0039] The step of optimizing pre-configured parameters is an association matrix between the RIS panel and the remote user terminal ( ), phase shift of the RIS panel ( ), and the elevation angle of the satellite ( A first optimization step of optimizing the power (p) using a preset first algorithm after initializing ); and an association matrix between the RIS panel and the remote user terminal ( ), phase shift of the RIS panel ( ), and the elevation angle of the satellite ( It may include a second optimization step that optimizes ).

[0040] The first algorithm above is WOA (Whale Optimization Algorithm), and the first optimization step can calculate the optimal value of the power (p) through a fitness function according to the following mathematical formula.

[0041] (Mathematical formula)

[0042]

[0043] : Inequality function

[0044] : Pre-set constant

[0045] : Index function

[0046] The above index function becomes 0 if the above inequality function is 0, and has a value of 1 if the above inequality function is less than 0.

[0047] The above second algorithm is an algorithm using a Multi-Agent Reinforcement Learning model, and an association matrix between the RIS panel and the remote user terminal ( ), phase shift of the RIS panel ( ), and the elevation angle of the satellite ( ) can be optimized.

[0048] (Mathematical formula)

[0049]

[0050] The second optimization step above comprises the global state (s') of the plurality of satellites and the local state (s') of each satellite according to the following mathematical formula. s It may include the step of inputting ) into the above multi-agent reinforcement learning model.

[0051] (Mathematical formula)

[0052]

[0053]

[0054] : Position of satellite s

[0055] : Association matrix between the RIS panel of satellite s and the user terminal

[0056] : Phase shift of the RIS panel of satellite s

[0057] : Optimized power of satellite s

[0058] i s : The angle of intersection between the orbital plane of satellite s and the equator

[0059] ω s: The angle between the vernal equinox of satellite s and the intersection of the orbital plane and the equatorial plane

[0060] χ s : The angle formed by the direction of satellite s and the intersection of the orbital plane and the equatorial plane

[0061] The second optimization step may include: a step of calculating an action regarding the association between the RIS panel and the remote user terminal, the phase shift of the RIS panel, and the elevation angle of the phase based on the local state of each satellite in the multi-agent reinforcement learning model; and a step of granting a global reward for the calculated action to the plurality of satellites in the multi-agent reinforcement learning model.

[0062] The above second optimization step is the global compensation (R) according to the following mathematical formula. s',t It may further include a step of training the above multi-agent reinforcement learning model so that ) is maximized.

[0063] (Mathematical formula)

[0064]

[0065] s t ' : Local state of the satellite at time slot t

[0066] a t : Satellite behavior in time slot t

[0067] RUE: Remote User Terminal

[0068] : Penalty value

[0069] A computing device according to one disclosed embodiment comprises one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and is a computing device for performing a communication method of a communication system comprising a plurality of satellites having a Reconfigurable Intelligent Surface (RIS) panel, a plurality of remote user terminals, and one or more base station devices, wherein the one or more programs include: a command for calculating a received signal of a remote user terminal when the remote user terminal receives a signal transmitted from a satellite; a command for calculating a data rate of the remote user terminal using the Signal to Noise Ratio (SNR) of the remote user terminal with respect to the received signal; and a command for optimizing parameters pre-set to maximize the data rate of the remote user terminal. Effects of the invention

[0071] According to the disclosed embodiment, by optimizing the power of the satellite, the association matrix between the satellite's RIS panel and the remote user terminal, the phase shift of the satellite's RIS panel, and the elevation angle of the satellite, the data rate according to the received signal of the remote user terminal can be maximized and the coverage of each low-orbit satellite can be optimized. Brief explanation of the drawing

[0073] FIG. 1 is a schematic diagram showing a communication system using a low-orbit satellite according to an embodiment of the present invention. FIG. 2 is a schematic diagram showing the angle of a low-orbit satellite with respect to the center of the Earth used to control a coverage area in one embodiment of the present invention. FIG. 3 is a flowchart illustrating a process for maximizing the data speed of a remote user terminal in one embodiment of the present invention. FIG. 4 is a schematic diagram illustrating multi-agent reinforcement learning in one embodiment of the present invention. FIG. 5 is a block diagram illustrating a computing environment including a computing device suitable for use in exemplary embodiments. Specific details for implementing the invention

[0074] Hereinafter, specific embodiments of the present invention will be described with reference to the drawings. The following detailed description is provided to facilitate a comprehensive understanding of the methods, apparatus, and / or systems described herein. However, this is merely illustrative and the present invention is not limited thereto.

[0075] In describing the embodiments of the present invention, detailed descriptions of known technologies related to the present invention are omitted if it is determined that such detailed descriptions may unnecessarily obscure the essence of the invention. Furthermore, the terms described below are defined in consideration of their functions within the present invention, and these may vary depending on the intentions or practices of the user or operator. Therefore, such definitions should be based on the content throughout this specification. Terms used in the detailed description are intended merely to describe the embodiments of the present invention and should not be limiting in any way. Unless explicitly stated otherwise, expressions in the singular form include the meaning of the plural form. In this description, expressions such as "include" or "comprise" are intended to refer to certain characteristics, numbers, steps, actions, elements, parts thereof, or combinations thereof, and should not be interpreted to exclude the existence or possibility of one or more other characteristics, numbers, steps, actions, elements, parts thereof, or combinations thereof other than those described.

[0076] In the following description, terms such as "transmission," "communication," "transmission," "reception," and other terms of similar meaning regarding signals or information include not only the direct transfer of signals or information from one component to another but also the transfer through other components. In particular, "transmission" or "transmitting" a signal or information to one component refers to the final destination of the signal or information and does not mean a direct destination. The same applies to the "reception" of signals or information. Furthermore, in this specification, two or more data or information are "related" means that if one data (or information) is obtained, at least a portion of another data (or information) can be obtained based thereon.

[0077] Additionally, terms such as "first," "second," etc., may be used to describe various components, but said components should not be limited by said terms. These terms may be used for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be named the second component, and similarly, the second component may be named the first component.

[0079] FIG. 1 is a schematic diagram showing a communication system using a low-orbit satellite according to one embodiment of the present invention.

[0080] Referring to FIG. 1, a communication system (100) using a low-orbit satellite may include a satellite (102), a remote user terminal (104), and a base station device (106).

[0081] Multiple satellites (102) are provided. The satellites (102) may be Low Earth Orbit (LEO) satellites. Each satellite (102) may be placed in an optimal location to provide global coverage. The communication system (100) may include a minimum number of satellites (102) that satisfy pre-set requirements while providing global coverage.

[0082] Each satellite (102) can be associated with all remote user terminals (104) in the corresponding coverage area. Additionally, each remote user terminal (104) can equally share the available bandwidth in the sub-Hertz band of each satellite (102).

[0083] A Reconfigurable Intelligent Surface (RIS) panel may be provided on the satellite (102). The RIS panel may be referred to as an intelligent reflective surface. That is, the satellite (102) may be equipped with a RIS panel capable of controlling phase shift and increasing signal strength. The RIS panel can be programmed to change the electromagnetic field to concentrate, adjust, and enhance signal power for a target user.

[0084] In the disclosed embodiment, satellite coverage can be maximized by mounting an RIS panel on the satellite (102). This can be achieved by adjusting the phase shift of the RIS panel. The RIS panel may be composed of a metasurface capable of changing the reflection phase. At this time, the phase shift in the RIS panel can be adjusted to beamform to a target user. On the satellite (102), the RIS panel may be placed below the solar panel.

[0085] Each RID panel r is a matrix containing n reflecting elements. It may be configured as follows. Each reflective element may be equipped with a phase shifter capable of reconstructing the incident signal to prevent signal loss. The remote user terminal (104) can use various communication services via the satellite (102). Various heterogeneous devices may be used for the remote user terminal (104). The remote user terminal (104) may be located in various places and equipment such as homes, offices, ships, aircraft, and vehicles, but it may also be a device carried by each individual.

[0086] The base station device (106) can control the placement of the satellite (102). The base station device (106) may be configured to optimize the phase shift, phase transmission power, and beamforming of downlink communication through the satellite (102). The base station device (106) can implement this through scheduling policies, transmission power allocation, phase shift control, and phase elevation angle adjustment to maximize the coverage of each satellite (102). At this time, each satellite (102) acts as a kind of distributed agent and can be centrally controlled by the base station device (106). A detailed explanation of this will be provided later. Through such optimization, the base station device (106) can maximize the average data rate for the remote user terminal (104) of the satellite (102).

[0087] The communication system (100) may be a satellite-based Multiple Input Multiple Output (MIMO) system. The communication system (100) may be configured to minimize signal attenuation in the sub-terahertz (Sub-THz) band for a remote user terminal (104). The communication system (100) may provide high-speed data services to a remote user terminal (104) that is not under the coverage of a terrestrial network infrastructure. Additionally, it may provide wireless services in urban areas to meet demand that cannot be met by existing terrestrial networks.

[0089] FIG. 2 is a schematic diagram showing the angle of a low-orbit satellite relative to the center of the Earth used to control the coverage area in one embodiment of the present invention.

[0090] Referring to Fig. 2, the orbit of each satellite(s) can be assumed to be circular. The 3D position of the satellite(s) in the time slot t is It can be expressed as. And, the position of the satellite(s) in a specific orbit m in the next time slot It can be expressed as follows. Here, is the interval between each time slot, and v s is the speed of the satellite.

[0091] Meanwhile, the following three angles (equator-related satellite angles) can be used to coordinate the orientation and position of a satellite relative to the equator. 1) The first angle, inclination angle i, can be defined as the intersection angle between the orbital plane and the equator. An inclination angle greater than 90 degrees indicates that the satellite's motion is opposite to the direction of the Earth's rotation. 2) The second angle, angle ω, can be defined as the angle between the vernal equinox and the intersection point of the orbital plane and the equatorial plane. 3) The third angle, angle χ, can be defined as the angle formed by the satellite's orientation and the intersection point of the orbital plane and the equatorial plane.

[0092] Cartesian coordinates ( ) can be expressed by the following mathematical formula 1.

[0093] (Mathematical Formula 1)

[0094]

[0095] h s : Satellite altitude

[0096] R e Earth's radius

[0097] S m : A set of satellites placed in orbital plane m

[0098] Here, the distance between two satellites (s, s') can be calculated as shown in Equation 2 below.

[0099] (Mathematical Formula 2)

[0100]

[0101] δ: The angle between two satellites in a single orbit as viewed from the center of the Earth.

[0102] The angle δ between the two satellites (s, s') is It can be represented as, is the elevation angle of each satellite.

[0103] And, the distance (d) in time slot t between the base station device (b) and the satellite(s) b,s ) is represented by the following mathematical formula 3, and the distance (d) in the time slot t between the satellite(s) and the remote user terminal (u) s,u ) can be expressed by the following mathematical formula 4.

[0104] (Mathematical Formula 3)

[0105]

[0106] (Mathematical Formula 4)

[0107]

[0108] In addition, the elevation angle in time slot t between the base station device (b) and the satellite (s) can be represented by the following mathematical formula 5, and the elevation angle in time slot t between the satellite (s) and the remote user terminal (u) can be represented by the following mathematical formula 6.

[0109] (Mathematical Formula 5)

[0110]

[0111] (Mathematical Formula 6)

[0112]

[0113] Here, the line-of-sight (LoS) propagation distance and the minimum elevation angle of the base station device (b) and the remote user terminal (u) can determine the coverage area provided by the satellite(s) at each time slot t. In this case, as the minimum elevation angle increases, the coverage of the satellite decreases.

[0115] In the communication system (100), indirect communication links include links between satellite and base station devices, links between satellites, and links between satellites and remote user terminals. Here, since each link shares the same function, the same channel model can be used. Also, signal propagation in the sub-terahertz band is affected by spreading loss, molecular absorption loss near Earth, loss due to rain and clouds, and ionospheric plasma loss.

[0116] Diffusion loss can refer to the ratio of power emitted from an isotropic transmitter that can be detected by an isotropic receiver. Molecular absorption loss can represent the ratio of electromagnetic energy converted into kinetic energy within vibrating molecules. Losses due to rain and clouds can refer to the loss resulting from scattering and absorption by rain and clouds as the signal travels through the atmosphere. Ionospheric plasma loss can refer to the loss that occurs when the signal passes through the plasma of the ionosphere.

[0117] Here, the total loss that occurs when a signal of frequency f propagates over a distance d can be expressed by the following mathematical equation 7.

[0118] (Mathematical Formula 7)

[0119]

[0120] L abs : Molecular absorption loss

[0121] L rain : Loss due to rain

[0122] Lcloud : Loss caused by clouds

[0123] L atten : Attenuation coefficient (loss per unit distance along the signal propagation path)

[0124] c : speed of light

[0125] G b : Antenna gain of base station device

[0126] G u : Antenna gain of remote user terminal

[0127] d x ,d y : The size of the satellite's unit cell in the x and y directions

[0128] A: Satellite coverage

[0129] : Satellite elevation angle

[0130] d b,s : Distance between the satellite and the base station device

[0131] d s,s' : Distance between a satellite and another satellite

[0132] d s,u : Distance between the satellite and the remote user terminal

[0133] N r : Number of reflective elements in the RIS panel

[0134] n s : s-th reflection element of the RIS panel

[0135] : Normalized total radiation pattern considered in received signal power

[0137] Meanwhile, the base station device (106) may be equipped with multiple (K) antennas that communicate with multiple remote user terminals (102) within a dedicated area. To improve the coverage range of sub-terahertz communication, the base station device (106) may support multiple RIS panels to the remote user terminals (102) and simultaneously transmit data streams from the K antennas. When the RIS panels are activated, each data stream is beam-formed to one remote user terminal (102).

[0138] Here, Defines as the channel matrix between the base station device and the RIS of the first satellite, and If defined as the channel matrix between RIS r and RIS r+1, the signal (y) received at the remote user terminal (u) u ) can be expressed by the following mathematical equation 8. In this case, the signal transmitted from the base station device (b) reaches the remote user terminal (u) before RIS hop R s (R s It can be assumed that movement occurs through (≤ S, where S is the total number of satellites). That is, R s can refer to the RIS panel that the signal passes through before reaching the remote user terminal.

[0139] (Mathematical Formula 8)

[0140]

[0141] L tot : Total loss until the signal is received by the remote user terminal

[0142] p u : Transmission power of base station device

[0143] g u : Channel from the last RIS to the remote user terminal

[0144] : Transmission vector of base station device

[0145] z uAWGN (Additive White Gaussian Noise) received from a remote user terminal

[0146] : Channel matrix between base station device and RIS r

[0147] : Phase shift of RIS r

[0148] Here, the phase shift of the r-th RIS panel ( ) can be expressed by the following mathematical formula 9.

[0149] (Mathematical Formula 9)

[0150]

[0151] A rn : Amplitude of the nth element in the r-th RIS panel

[0152] θ rn : Reflection coefficient of the nth element in the r-th RIS panel

[0153] In addition, the association matrix for all R RIS panels and all U remote user terminals is It can be represented as follows. Association matrix for each remote user terminal (u) The elements of can be represented by the following mathematical formula 10.

[0154] (Mathematical Formula 10)

[0155]

[0156] GBS: Base Station Device

[0157] RUE: Remote User Terminal

[0158] Also, transmission vector ( )Is It can be expressed as. Here, (K is the number of antennas in the base station) is the beamforming vector, and can be a Gaussian signal of each remote user terminal (u).

[0159] For a received signal given by Equation 8, the instantaneous SNR (Signal to Noise Ratio) of a remote user terminal (u) at each time slot t can be expressed by Equation 11 below.

[0160] (Mathematical Formula 11)

[0161]

[0162] p s,u : Power for transmitting signals from a satellite to a remote user terminal

[0163] N0: Noise spectral density

[0164] And, the executable data rate (R) of the remote user terminal (u) at time slot t. u,t ) can be expressed by the following mathematical formula 12.

[0165] (Mathematical Formula 12)

[0166]

[0167] B u : Total bandwidth available on the remote user terminal

[0168] Therefore, increasing the RIS panel and SNR level can improve the data rate.

[0170] In the communication system (100), the correlation, phase shift, satellite power, and elevation angle between the RIS panel and the remote user terminal can be optimized under preset constraints to maximize the data rate of the remote user terminal (102). Accordingly, the objective function of the communication system (100) can be expressed as shown in Equation 13 below.

[0171] (Mathematical Formula 13)

[0172]

[0173] p : Power for transmitting a signal from a satellite to a remote user terminal

[0174] : Association matrix between the satellite RIS panel and the remote user terminal

[0175] : Phase shift of the satellite's RIS panel

[0176] : Satellite elevation angle

[0177] FIG. 3 is a flowchart illustrating a process for maximizing the data rate of a remote user terminal in an embodiment of the present invention. Referring to FIG. 3, the problem of maximizing the data rate of a remote user terminal can be broadly divided into two parts. First, the satellite power is optimized by initializing the associativity matrix, phase shift, and elevation angle (S 101), and then the associativity matrix, phase shift, and elevation angle can be optimized based on the optimized satellite power (S 103). At this time, it is checked whether the data rate of the remote user terminal is maximized (S 105), and if the data rate is maximized, the optimized values ​​of the satellite power, associativity matrix, phase shift, and elevation angle can be output (S 107). If the data rate is not maximized, the above process can be repeated.

[0178] Specifically, the problem of optimizing the satellite's transmission power can be solved using WOA (Whale Optimization Algorithm). The association matrix between the RIS panel and the remote user terminal ( ), phase shift of the RIS panel ( ), and the elevation angle of the satellite ( When initializing ), the problem of optimizing the satellite's transmission power can be expressed as Equation 14 below.

[0179] (Mathematical Formula 14)

[0180]

[0181] Here, the Whale Optimization Algorithm (WOA) is a meta-heuristic algorithm that mimics the hunting characteristics of humpback whales. After spotting prey, humpback whales dive deep beneath the prey and create a spiral bubble, which serves to surround the prey within a smaller area.

[0182] In the Whale Optimization algorithm, individual whales (agents) represent different control variables, and the distance between the whale and the prey represents the cost of the objective function. At this time, the position of each individual whale over time is calculated through the processes of 1) Encircling Prey, 2) Bubble-Net Attacking, and 3) Search for Prey. Since the Whale Optimization algorithm (WOA) is a previously disclosed algorithm, a detailed explanation thereof will be omitted.

[0183] In the disclosed embodiment, the satellite power (p) can be the whale's position. The optimal whale position (i.e., the satellite power value) can be calculated through the fitness function according to the following Equation 15.

[0184] (Mathematical Formula 15)

[0185]

[0186] : Inequality function

[0187] : Pre-set constant

[0188] : Index function

[0189] Here, the index function is an inequality function If is greater than or equal to 0, it becomes 0, and the inequality function If is less than 0, it can have a value of 1. That is, the relationship between the index function and the inequality function can be expressed by the following mathematical equation 16.

[0190] (Mathematical Formula 16)

[0191]

[0192]

[0193] In mathematical equation 15, placing a negative sign on the right-hand side of the fitness function is intended to transform the maximization problem into a minimization problem.

[0194] In this way, when the satellite power (p) is optimized, the communication system (100) obtains an association matrix between the RIS panel and the remote user terminal through Multi Agent Reinforcement Learning ( ), phase shift of the RIS panel ( ), and the elevation angle of the satellite ( ) can be optimized. This can be expressed as in Equation 17 below.

[0195] (Mathematical Formula 17)

[0196]

[0197] FIG. 4 illustrates a schematic diagram for multi-agent reinforcement learning in one embodiment of the present invention. Referring to FIG. 4, the RIS-LEO network environment may include an entire network environment that includes all information regarding low-orbit satellites, power usage, input and output signals of the RIS, and the altitude and angle of the satellites. In this case, the information within the network may be used to jointly learn each low-orbit satellite (agent). Each agent may receive observations regarding its state from the network environment. Here, each satellite acts as an agent. The multi-agent reinforcement learning model may be installed in a base station device, but is not limited thereto and may also be installed in other server computing devices.

[0198] Multi-agent reinforcement learning models are policy networks and value network It can have. Here, the policy network and the value network can be artificial neural networks for multi-agent reinforcement learning. The value network can act as a local critic for each agent. The value network can provide immediate feedback on the accuracy of a new policy when each agent (satellite) learns it. The policy network can serve as the learning unit for the main policy. The policy can be used by each agent to make decisions (i.e., actions).

[0199] The actions taken by each agent in a distributed manner can be used for evaluation by a centralized critic. The centralized critic ensures cooperation among all agents, calculates the global loss of all agents, and enables the system's performance to converge globally. Current rewards can be sent to the centralized critic to analyze the results of each distributed agent, and each reward and state can be stored in an Experience Pool.

[0200] The policy network can be referred to as an actor, and the value network as a critic. The policy network (i.e., the actor) looks only at local information (i.e., the information of the agent in question) and generates actions based on it, whereas the value network (i.e., the critic) can view the entire information of other agents (i.e., other satellites) to better optimize itself.

[0201] Here, let s' be the global state, and let s' be the local state of agent s (i.e., satellite s). s If so, this can be expressed as shown in the following mathematical equation 18. The local states (i.e., global states) of each agent are used as inputs to the reinforcement learning model.

[0202] (Mathematical Formula 18)

[0203]

[0204]

[0205] : Position of satellite s

[0206] : Association matrix between the RIS panel of satellite s and the user terminal

[0207] : Phase shift of the RIS panel of satellite s

[0208] : Optimized power of satellite s

[0209] i s : The angle of intersection between the orbital plane of satellite s and the equator

[0210] ω s : The angle between the vernal equinox of satellite s and the intersection of the orbital plane and the equatorial plane

[0211] χ s : The angle formed by the direction of satellite s and the intersection of the orbital plane and the equatorial plane

[0212] Agent s is the optimal power in the assigned area After obtaining, local state s' s It can be passed to the actor (policy network). The current local state s' s In this, the actor can calculate a method to optimize the association between the RIS panel and the remote user terminal, the phase shift of the RIS panel, and the elevation angle of the phase. At this time, the action performed by the agent's actor (a s ) can be expressed by the following mathematical formula 19. That is, the agent (satellite) can perform actions to change one or more of the association between the RIS panel and the remote user terminal, the phase shift of the RIS panel, and the elevation angle of the phase.

[0213] (Mathematical Formula 19)

[0214]

[0215] After generating action probabilities, the actor samples the action distribution and interacts with the local environment to determine the next state ( Before moving to ), the reward(R s You can obtain ). And, in the experience pool D, experience This is stored. At this time, data throughput and QoS fairness to maximize Equation 17 can be used as immediate rewards. Additionally, to strengthen cooperation among agents, a global reward expressed by Equation 20 can be received.

[0216] (Mathematical Formula 20)

[0217]

[0218] : Penalty value

[0219] Here, global rewards is the immediate reward of agent s' at time slot t. All agents (all satellites) can share the global reward. A reinforcement learning model can be trained to maximize the global reward.

[0220] Cumulative discounted reward is the discount rate It can be calculated using mathematical formula 21 with (a value between 0 and 1).

[0221] (Mathematical Formula 21)

[0222]

[0223] The combined policy of multi-agents If so, the state-value function ) can be expressed by mathematical formula 22.

[0224] (Mathematical Formula 22)

[0225]

[0226] Here, the optimization problem can be applied as determining the best method to maximize the expected cumulative discount reward for an initial state s'(0), and thus the optimization problem of Equation 17 through multi-agent reinforcement learning can be expressed by Equation 23 below. That is, the optimization problem of Equation 17 can be expressed as searching for an optimal policy that maximizes the expected cumulative reward over time.

[0227] (Mathematical Formula 23)

[0228]

[0229] Meanwhile, given with Maximizing is equivalent to maximizing the following mathematical formula 24.

[0230] (Mathematical Formula 24)

[0231]

[0232] : Advantage function

[0233] Here, the advantage function is It is the same as. And, is a state-action value function, and this is It can be represented as. Also, using the clip function By approximating it, the optimization problem of Equation 23 can be transformed into Equation 25 below.

[0234] (Mathematical Formula 25)

[0235]

[0236] joint probability ratio

[0237] Here, the joint probability ratio It can be expressed as follows. The value of the joint probability ratio is obtained using a clip function that keeps policy updates within a small range. It can be maintained within the range of.

[0238] As the combined action of all agents excluding agent s Use , and as an integrated policy for all agents excluding agent s You can use it. Because policies are independent It is safe to believe that. Here, is an action-observation history. From this factorization, a goal such as the following mathematical equation 26 can be generated.

[0239] (Mathematical Formula 26)

[0240]

[0241] : Agent s's policy parameter

[0242] Here, is. This point function Although it is defined, it is difficult to determine the individual contribution of each agent. To facilitate credit allocation, the joint advantage function can be decomposed into specific local agents of the agent as follows.

[0243] Here, represents a counterfactual advantage, and c s represents a non-negative weight.

[0244] Due to the non-negative decomposition of, there is a monotonic connection between the global optimal and the local optimal, and this at It means a transformation to. The optimization of Equation 26 can be modified as shown in Equation 27 below to maximize the individual goals of each agent.

[0245] (Mathematical Formula 27)

[0246]

[0247] However, if each agent is completely independent throughout the execution, it increases the possibility of excessive variance, so the variance can increase exponentially with the number of agents. Accordingly, Equation 27 can be modified as shown in Equation 28 below through a double clipping approach.

[0248] (Mathematical Formula 28)

[0249]

[0250] Here, is. In mathematical formula 27 The existence of The weight of affects the agent's goal. As a result of clipping for , the effect of other agents' updates on agent s It is limited to restrict variations generated by other agents.

[0251] When the parameters of the actor network are modified, the sampled action is passed to the critic network, which estimates the estimated value of the action. Based on the predicted value, the actor can be modified using a policy gradient approach. The benefit function in the training phase uses preserved samples. The expected value of can be estimated. That is, the policy can be updated using the gradient as shown in the following mathematical equation 29.

[0252] (Mathematical Formula 29)

[0253]

[0254] Here, is the Generalized Advantage Estimation (GAE), which can be defined by the following mathematical formula 30.

[0255] (Mathematical Formula 30)

[0256]

[0257] Here, can be estimated using the k-step bootstrapping linear combination as shown in the following Equation 31.

[0258] (Mathematical Formula 31)

[0259]

[0260] Here, is the temporal difference, which can be defined by mathematical equation 32.

[0261] (Mathematical Formula 32)

[0262]

[0263] Since each satellite s estimates the same combined action-state value function, the loss function of the value network can be expressed as in the following Equation 33, and the value network can be trained to minimize this loss function.

[0264] (Mathematical Formula 33)

[0265]

[0266] To this end, a gradient descent method can be used, and the gradient value of each satellite (agent) can be calculated through the following mathematical formula 34.

[0267] (Mathematical Formula 34)

[0268]

[0269] Here, the parameter and can be changed until the loss functions of the policy network and the value network converge, according to Equation 29 and Equation 34, respectively.

[0270] According to the disclosed embodiment, by optimizing the power of the satellite, the association matrix between the satellite's RIS panel and the remote user terminal, the phase shift of the satellite's RIS panel, and the elevation angle of the satellite, the data rate according to the received signal of the remote user terminal can be maximized and the coverage of each low-orbit satellite can be optimized.

[0272] FIG. 5 is a block diagram illustrating a computing environment (10) including a computing device suitable for use in exemplary embodiments. In the illustrated embodiments, each component may have different functions and capabilities in addition to those described below, and may include additional components in addition to those described below.

[0273] The illustrated computing environment (10) includes a computing device (12). In one embodiment, the computing device (12) may be a satellite (102). Additionally, the computing device (12) may be a remote user terminal (104). Additionally, the computing device (12) may be a base station device (106). Additionally, the computing device (12) may be a server computing device with a multi-agent reinforcement learning model installed.

[0274] The computing device (12) includes at least one processor (14), a computer-readable storage medium (16), and a communication bus (18). The processor (14) can cause the computing device (12) to operate according to the exemplary embodiment described above. For example, the processor (14) can execute one or more programs stored in the computer-readable storage medium (16). The one or more programs may include one or more computer-executable instructions, and the computer-executable instructions may be configured to cause the computing device (12) to perform operations according to the exemplary embodiment when executed by the processor (14).

[0275] A computer-readable storage medium (16) is configured to store computer-executable instructions or program code, program data and / or other suitable forms of information. A program (20) stored in the computer-readable storage medium (16) includes a set of instructions executable by a processor (14). In one embodiment, the computer-readable storage medium (16) may be memory (volatile memory such as random access memory, non-volatile memory, or a suitable combination thereof), one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other forms of storage media that are accessed by a computing device (12) and capable of storing desired information, or a suitable combination thereof.

[0276] The communication bus (18) interconnects various other components of the computing device (12), including the processor (14) and the computer-readable storage medium (16).

[0277] The computing device (12) may also include one or more input / output interfaces (22) and one or more network communication interfaces (26) that provide interfaces for one or more input / output devices (24). The input / output interfaces (22) and network communication interfaces (26) are connected to a communication bus (18). The input / output devices (24) may be connected to other components of the computing device (12) through the input / output interfaces (22). An exemplary input / output device (24) may include an input device such as a pointing device (such as a mouse or trackpad), a keyboard, a touch input device (such as a touchpad or touchscreen), a voice or sound input device, various types of sensor devices and / or imaging devices, and / or an output device such as a display device, a printer, a speaker and / or a network card. An exemplary input / output device (24) may be included inside the computing device (12) as a component constituting the computing device (12), or it may be connected to the computing device (12) as a separate device distinct from the computing device (12).

[0279] Although representative embodiments of the present invention have been described in detail above, those skilled in the art will understand that various modifications can be made to the above-described embodiments without departing from the scope of the present invention. Therefore, the scope of the present invention should not be limited to the described embodiments, but should be defined by the claims set forth below as well as equivalents thereof. Explanation of the symbols

[0281] 10: Computing Environment 12: Computing device 14 : Processor 16: Computer-readable storage media 18: Communication bus 20 : Program 22 : Input / Output Interface 24 : Input / Output Devices 26: Network communication interface 100 : Communication System 102 : Satellite 104 : Remote user terminal 106: Base station device

Claims

Claim 1 A communication method of a communication system comprising a plurality of satellites equipped with RIS (Reconfigurable Intelligent Surface) panels, a plurality of remote user terminals, and one or more base station devices, and a communication method performed on a computing device equipped with one or more processors and a memory storing one or more programs executed by said one or more processors, wherein when a remote user terminal receives a signal transmitted from a satellite, the method comprises: a step of calculating a received signal of the remote user terminal; a step of calculating a data rate of said remote user terminal using the SNR (Signal to Noise Ratio) of said remote user terminal with respect to said received signal; and a step of optimizing preset parameters to maximize the data rate of said remote user terminal, wherein the step of optimizing preset parameters comprises power (p) for transmitting a signal from a satellite to a remote user terminal to satisfy the objective function of the following mathematical formula, and an association matrix between the RIS panel of the satellite and the remote user terminal ( ), phase shift of the satellite's RIS panel ( ), and the elevation angle of the satellite ( A communication method that optimizes ). (Mathematical formula) R u,t : Data rate T of remote user terminal u in time slot t : Total number of time slots : Total number of remote user terminals Claim 2 In claim 1, the received signal (y u ) is a communication method calculated through the following mathematical formula. (Mathematical formula) L tot : Total loss p until the signal is received at the remote user terminal u : Transmission power g of the base station device u : Channel from the last RIS panel to the remote user terminal : Transmission vector R of the base station device s : Total number of RIS panels z u : AWGN (Additive White Gaussian Noise) received from the remote user terminal : RIS index : Channel matrix between base station device and RIS panel r : Phase shift of RIS panel r Claim 3 In claim 2, the SNR (Signal to Noise Ratio) at time slot t of the remote user terminal ( ) is a communication method calculated by the following mathematical formula. (Mathematical formula) p s,u : Power for transmitting signals from a satellite to a remote user terminal : Element N0 of the association matrix for the remote user terminal: Noise spectral density : Beamforming vector of base station device Claim 4 In claim 3, the data rate (R) of the remote user terminal u,t ) is a communication method calculated by the following mathematical formula. (Mathematical formula) B u : Total bandwidth available on the remote user terminal Claim 5 delete Claim 6 In claim 1, the step of optimizing preset parameters comprises an association matrix between the RIS panel and the remote user terminal ( ), phase shift of the RIS panel ( ), and the elevation angle of the satellite ( A first optimization step of optimizing the power (p) using a preset first algorithm after initializing ); and an association matrix between the RIS panel and the remote user terminal ( ), phase shift of the RIS panel ( ), and the elevation angle of the satellite ( A communication method comprising a second optimization step for optimizing ). Claim 7 A communication method according to claim 6, wherein the first algorithm is WOA (Whale Optimization Algorithm), and the first optimization step calculates the optimal value of the power (p) through a fitness function according to the following mathematical formula. (Mathematical formula) : Inequality function : Pre-set constant : Index function. The index function becomes 0 if the inequality function is 0, and has a value of 1 if the inequality function is less than 0. Claim 8 In claim 6, the second algorithm is an algorithm using a Multi-Agent Reinforcement Learning model, and an association matrix between the RIS panel and the remote user terminal ( ), phase shift of the RIS panel ( ), and the elevation angle of the satellite ( A communication method that optimizes ). (Mathematical formula) Claim 9 In claim 8, the second optimization step comprises the global state (s') of the plurality of satellites and the local state (s') of each satellite according to the following mathematical formula. s A communication method comprising the step of inputting ) into the above multi-agent reinforcement learning model. (Mathematical formula) : Position of satellite s : Association matrix between the RIS panel of satellite s and the user terminal : Phase shift of the RIS panel of satellite s : Optimized power i of satellite s s : Angle of intersection ω between the orbital plane of satellite s and the equator s : Angle χ between the vernal equinox of satellite s and the intersection of the orbital plane and the equatorial plane s : The angle formed by the direction of satellite s and the intersection of the orbital plane and the equatorial plane Claim 10 A communication method according to claim 9, wherein the second optimization step comprises: calculating an action regarding the association between the RIS panel and the remote user terminal, the phase shift of the RIS panel, and the elevation angle of the phase based on the local state of each satellite in the multi-agent reinforcement learning model; and granting a global reward for the calculated action to the plurality of satellites in the multi-agent reinforcement learning model. Claim 11 In claim 10, the second optimization step comprises the global compensation (R) according to the following mathematical formula. s',t A communication method further comprising the step of training the multi-agent reinforcement learning model such that ) is maximized. (Mathematical formula) s t ' : Local state of the satellite a at time slot t t : Satellite action RUE at time slot t : Remote user terminal : Penalty value Claim 12 A computing device for performing a communication method of a communication system comprising one or more processors; memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and wherein the communication system comprises a plurality of satellites having a Reconfigurable Intelligent Surface (RIS) panel, a plurality of remote user terminals, and one or more base station devices, wherein the one or more programs include: a command for calculating a received signal of a remote user terminal when the remote user terminal receives a signal transmitted from a satellite; a command for calculating a data rate of the remote user terminal using the Signal to Noise Ratio (SNR) of the remote user terminal for the received signal; and a command for optimizing preset parameters to maximize the data rate of the remote user terminal, wherein the command for optimizing preset parameters includes power (p) for transmitting a signal from a satellite to a remote user terminal to satisfy the objective function of the following mathematical formula, and an association matrix between the RIS panel of the satellite and the remote user terminal ( ), phase shift of the satellite's RIS panel ( ), and the elevation angle of the satellite ( A computing device that optimizes ). (Mathematical formula) R u,t : Data rate T of remote user terminal u in time slot t : Total number of time slots : Total number of remote user terminals Claim 13 delete Claim 14 In claim 12, the command for optimizing preset parameters is an association matrix between the RIS panel and the remote user terminal ( ), phase shift of the RIS panel ( ), and the elevation angle of the satellite ( A first optimization command that optimizes the power (p) using a preset first algorithm after initializing ); and an association matrix between the RIS panel and the remote user terminal ( ), phase shift of the RIS panel ( ), and the elevation angle of the satellite ( A computing device comprising a second optimization instruction that optimizes ). Claim 15 A computing device according to claim 14, wherein the first algorithm is a WOA (Whale Optimization Algorithm), and the first optimization command calculates the optimal value of the power (p) through a fitness function according to the following mathematical formula. (Mathematical formula) : Inequality function : Pre-set constant : Index function. The index function becomes 0 if the inequality function is 0, and has a value of 1 if the inequality function is less than 0. Claim 16 In claim 14, the second algorithm is an algorithm using a Multi-Agent Reinforcement Learning model, and an association matrix between the RIS panel and the remote user terminal ( ), phase shift of the RIS panel ( ), and the elevation angle of the satellite ( A computing device that optimizes ). (Mathematical formula) Claim 17 In claim 16, the second optimization command comprises the global state (s') of the plurality of satellites and the local state (s') of each satellite according to the following mathematical formula. s A computing device comprising a command for inputting ) into the above multi-agent reinforcement learning model. (Mathematical formula) : Position of satellite s : Association matrix between the RIS panel of satellite s and the user terminal : Phase shift of the RIS panel of satellite s : Optimized power i of satellite s s : Angle of intersection ω between the orbital plane of satellite s and the equator s : Angle χ between the vernal equinox of satellite s and the intersection of the orbital plane and the equatorial plane s : The angle formed by the direction of satellite s and the intersection of the orbital plane and the equatorial plane Claim 18 A computing device according to claim 17, wherein the second optimization command comprises: a command for calculating an action regarding the association between the RIS panel and the remote user terminal, the phase shift of the RIS panel, and the elevation angle of the phase based on the local state of each satellite in the multi-agent reinforcement learning model; and a command for granting a global reward for the calculated action to the plurality of satellites in the multi-agent reinforcement learning model. Claim 19 In claim 18, the second optimization command is the global compensation (R) according to the following mathematical formula s',t A computing device further comprising instructions for training the multi-agent reinforcement learning model such that ) is maximized. (Mathematical formula) s t ' : Local state of the satellite a at time slot t t : Satellite action RUE at time slot t : Remote user terminal : Penalty value Claim 20 A computer program stored on a non-transitory computer-readable storage medium, comprising instructions for performing a communication method of a communication system including a plurality of satellites equipped with RIS (Reconfigurable Intelligent Surface) panels, a plurality of remote user terminals, and one or more base station devices, wherein when the instructions are executed by a computing device having one or more processors, the computing device performs the steps of: calculating a received signal of a remote user terminal when the remote user terminal receives a signal transmitted from a satellite; calculating a data rate of the remote user terminal using the SNR (Signal to Noise Ratio) of the remote user terminal for the received signal; and optimizing preset parameters to maximize the data rate of the remote user terminal, wherein the step of optimizing preset parameters includes power (p) for transmitting a signal from a satellite to a remote user terminal to satisfy the objective function of the following mathematical formula, and an association matrix between the RIS panel of the satellite and the remote user terminal ( ), phase shift of the satellite's RIS panel ( ), and the elevation angle of the satellite ( A computer program that optimizes ). (Mathematical formula) R u,t : Data rate T of remote user terminal u in time slot t : Total number of time slots : Total number of remote user terminals

Images