Ocean communication on-demand service coverage method and device based on low orbit satellite hopping beam

Abstract

The invention discloses an ocean communication on-demand service coverage method and device based on a low-orbit satellite hopping beam, and relates to the technical field of satellite communication, and the method comprises the steps: determining the position of each user terminal on the ocean through a low-orbit satellite; determining the number of beam positions and the position of each candidate beam position center point according to the position of each user terminal; based on the position of each user terminal and the position of the central point of each candidate beam position, taking the low-orbit satellites as relays for communication between each user terminal and a ground station, and constructing an optimization model of multi-low-orbit satellite joint hopping beam and user terminal transmitting power; and in each time slot, a layered heterogeneous multi-agent deep reinforcement learning method is adopted to solve optimization variables of the optimization model in a single time slot, the optimization variables comprise uplink transmitting power of the user terminal, a hopping beam pattern of a low-orbit satellite and sub-channel allocation, and the service coverage performance and the wireless resource utilization rate of ocean communication can be improved.

Description

Technical Field

[0001] The present application relates to the field of satellite communication technology, and in particular to a method and device for on-demand marine communication service coverage based on low-orbit satellite beam hopping. Background Art

[0002] The fifth generation of mobile communication technology (5G) provides powerful access capabilities for densely populated areas. However, in remote areas, polar regions, and oceans, there are problems such as difficulty in network deployment and high operation and maintenance costs. Satellite communication networks have the advantages of wide coverage, flexible beam directionality, and communication not restricted by geographical environment. They can provide effective services in remote mountainous areas, air, deserts, oceans, etc. In addition, satellite communications can also be used in special scenarios such as military applications and emergency communications.

[0003] At present, satellite communication has become an important research direction for the sixth generation mobile communication standard (6G). According to the application field, satellites can be divided into communication satellites, remote sensing satellites, navigation satellites, etc.; according to the height of the orbit, satellites can be divided into low earth orbit (LEO) satellites, medium earth orbit (MEO) satellites, and geostationary earth orbit (GEO) satellites. In recent years, the number of LEO satellite launches has increased rapidly, and they are gradually becoming the main component of satellite communications. They operate at an altitude of 500-2000 kilometers. Compared with GEO satellites, they have shorter propagation delays, higher data rates and lower launch costs, making them attractive.

[0004] For marine communication scenarios, since sea surface users are usually unevenly distributed and may be sparse, and the terminals in the coverage area may have fast-moving characteristics, the service demand is unevenly distributed in space and time. In the traditional static multi-beam system, the service position of the beam relative to the satellite does not change with time or periodically jumps in a specific pattern. There is still much room for improvement in network coverage performance and space-time and spectrum resource utilization. Summary of the invention

[0005] The purpose of this application is to provide a method and device for on-demand coverage of marine communications based on low-orbit satellite beam hopping, which can improve the coverage performance and resource utilization efficiency of marine communication networks.

[0006] To achieve the above objectives, this application provides the following solutions:

[0007] In a first aspect, the present application provides a method for on-demand service coverage of marine communications based on low-orbit satellite beam hopping, comprising:

[0008] Determine the location of each user terminal on the ocean through low-orbit satellites;

[0009] Determine the number of wave positions and the position of the center point of each candidate wave position according to the position of each user terminal;

[0010] Based on the location of each user terminal and the location of the center point of each candidate beam position, the low-orbit satellite is used as a relay for each user terminal to communicate with the ground station, and an optimization model of multiple low-orbit satellite joint beam hopping and user terminal transmission power is constructed;

[0011] In each time slot, a hierarchical heterogeneous multi-agent deep reinforcement learning method is used to solve the optimization variables of the optimization model in a single time slot. The optimization variables include the uplink transmission power of the corresponding user terminal output by the user terminal agent, the beam hopping pattern of the corresponding low-orbit satellite output by the low-orbit satellite agent, and the sub-channel allocation of the user terminal.

[0012] In a second aspect, the present application provides an on-demand service coverage device for marine communication based on low-orbit satellite beam hopping, and the on-demand service coverage device for marine communication based on low-orbit satellite beam hopping includes:

[0013] A user terminal location determination module is used to determine the location of each user terminal on the ocean through low-orbit satellites;

[0014] A beam position determination module, used to determine the number of beam positions and the position of the center point of each candidate beam position according to the position of each user terminal;

[0015] The optimization model building module is used to build an optimization model of multi-low-orbit satellite joint beam hopping and user terminal transmission power based on the location of each user terminal and the location of each candidate beam position center point, using the low-orbit satellite as a relay for each user terminal to communicate with the ground station;

[0016] A solution module is used to solve the optimization variables of the optimization model in a single time slot by adopting a hierarchical heterogeneous multi-agent deep reinforcement learning method in each time slot, wherein the optimization variables include the uplink transmission power of the corresponding user terminal output by the user terminal agent, the beam hopping pattern of the corresponding low-orbit satellite output by the low-orbit satellite agent, and the sub-channel allocation of the user terminal.

[0017] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0018] The present application provides a method and device for on-demand service coverage of marine communications based on low-orbit satellite beam hopping. The number of beam positions and the position of the center point of each candidate beam position are determined according to the position of each user terminal, so as to realize the allocation of coverage tasks of multiple low-orbit satellites, avoid excessive coverage overlap, and improve on-demand coverage capability and resource utilization efficiency. Based on the position of each user terminal and the position of the center point of each candidate beam position, the low-orbit satellite is used as a relay for each user terminal to communicate with the ground station, and an optimization model of joint beam hopping of multiple low-orbit satellites and user terminal transmission power is constructed. In each time slot, a hierarchical heterogeneous multi-agent deep reinforcement learning method is used to solve the optimization variables of the optimization model in a single time slot. The optimization variables include the uplink transmission power of the corresponding user terminal output by the user terminal agent, the beam hopping pattern of the corresponding low-orbit satellite output by the low-orbit satellite agent, and the sub-channel allocation of the user terminal, so as to realize the optimization of the low-orbit satellite and the user terminal according to the obtained beam pattern scheme and the uplink transmission power, realize on-demand coverage of user services, and further suppress beam interference and improve network resource utilization efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0020] Figure 1 A flowchart of a method for on-demand marine communication service coverage based on low-orbit satellite beam hopping provided in one embodiment of the present application;

[0021] Figure 2 A flowchart of determining a candidate beam center point provided in an embodiment of the present application;

[0022] Figure 3 A diagram of an on-demand coverage scenario for marine communications provided in an embodiment of the present application;

[0023] Figure 4 A framework diagram of a hierarchical heterogeneous multi-agent optimization algorithm provided in one embodiment of the present application;

[0024] Figure 5 A diagram of a local network and a hybrid network provided in an embodiment of the present application;

[0025] Figure 6 A schematic diagram of functional modules of a low-orbit satellite beam-hopping-based marine communication on-demand service coverage device provided in another embodiment of the present application. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0027] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0028] For marine communication scenarios, the use of traditional multi-beam technology will bring a series of shortcomings to the communication system: First, in terms of spectrum utilization, since different beams need to maintain a certain distance, that is, a certain spectrum interval is required between beams, part of the spectrum resources will be wasted; multi-beam technology is more difficult in interference management. When multiple beams are transmitted simultaneously on the same spectrum resources, the adjacent beams are too close to each other, which will cause inter-beam interference. Interference management strategies such as frequency reuse and interference elimination technology need to be adopted. In addition, traditional multi-beam technology requires complex antenna arrays and signal processing algorithms to achieve multi-beam transmission and reception, which will increase the complexity and cost of the system and may place higher requirements on hardware and software. On the other hand, in order to reasonably save resources and avoid interference between multiple users, the user's uplink transmission power should be reduced as much as possible to improve the system throughput of the communication system while making efficient use of system resources.

[0029] Based on this practical background, some people proposed the beam hopping (BH) technology. Beam hopping is a technology based on time slicing. In a certain time slot, driven by business needs, only part of the beams are activated, matching non-uniform and time-varying needs with satellite system resources. LEO satellites use beam hopping technology to achieve efficient use of onboard resources, provide services for sea surface user terminal systems in the coverage area, and transmit the information to be sent back to the ground station or data center. Beam hopping technology increases the flexibility of resource allocation in the spatial dimension of the system. By adjusting the number of time slots allocated at different wave positions, communication resources that match business needs are provided, and fewer hopping beams can be used to cover more communication areas. In terms of interference avoidance, interference can be effectively avoided by reasonably activating wave positions, while meeting user needs and improving network performance.

[0030] However, with the continuous development of satellite communication systems, beam hopping can meet the uneven spatial and temporal distribution of users to a certain extent through time slicing and on-demand activation of hopping, and improve the utilization rate of on-board resources. However, the application of beam hopping technology in satellite communication is still facing some challenges, mainly manifested in the following aspects: (1) Although beam hopping technology can meet the business needs of each beam position by changing the dwell time, the shape and relative position of the beam are usually fixed, and most of the existing studies consider the resource allocation of a single LEO satellite, that is, the resource allocation scheme of a single satellite serving a specific area on the ground. In practice, due to the low orbital altitude and limited coverage area of ​​LEO satellites, there are usually multiple satellites in each orbit, and there are multiple orbits around the earth. Therefore, LEO satellite coverage is coordinated by multiple satellites. For these satellites with collaborative coverage, the electromagnetic wave signals of their side lobes may interfere with the main lobes of the service beams of other satellites. Considering only the resource allocation of a single satellite will cause beam interference between satellites, resulting in reduced communication quality and system throughput; (2) To achieve user-centric on-demand access, LEO satellite networks need to adjust the beam hopping pattern according to dynamic changes in user distribution, service performance requirements, and network channel environment, so new beam hopping pattern design and optimization methods are needed. (3) Existing resource allocation algorithms are usually divided into two steps. First, the corresponding time slot is allocated to each beam, and then the hopping pattern is designed. However, this separate two-step design increases the complexity of resource allocation and is not suitable for scenarios where business demand changes rapidly and dynamically. When faced with real-time changing business needs, this method may not be able to adapt in time, resulting in reduced resource utilization efficiency. Therefore, it is necessary to study more flexible beam hopping and resource allocation algorithms to meet changing business needs.

[0031] The present application provides a method for on-demand service coverage of marine communications based on low-orbit satellite beam hopping, such as Figure 1 As shown, the method for on-demand service coverage of ocean communications based on low-orbit satellite beam hopping includes the following steps 101 to 104.

[0032] Step 101: Determine the location of each user terminal on the ocean through a low-orbit satellite.

[0033] Step 102: Determine the number of wave positions and the position of the center point of each candidate wave position according to the position of each user terminal.

[0034] Step 103: Based on the position of each user terminal and the position of the center point of each candidate beam position, the low-orbit satellite is used as a relay for each user terminal to communicate with the ground station, and an optimization model of multiple low-orbit satellite joint beam hopping and user terminal transmission power is constructed.

[0035] Step 104: In each time slot, a hierarchical heterogeneous multi-agent deep reinforcement learning method is used to solve the optimization variables of the optimization model in a single time slot, wherein the optimization variables include the uplink transmission power of the corresponding user terminal output by the user terminal agent, the beam hopping pattern of the corresponding low-orbit satellite output by the low-orbit satellite agent, and the sub-channel allocation of the user terminal.

[0036] In view of the communication characteristics and needs in ocean scenarios, this application designs an on-demand communication coverage method based on low-orbit satellite beam-hopping technology, that is, the communication method of this application improves the system efficiency of the LEO satellite network while meeting the user's business needs. Specifically, the number of wave positions and the position of the center point of each candidate wave position are determined according to the position of each user terminal, which can realize the allocation of multi-low-orbit satellite coverage tasks, avoid excessive coverage overlap, and improve on-demand coverage capabilities and resource utilization efficiency; based on the position of each user terminal and the position of the center point of each candidate wave position, the low-orbit satellite is used as the connection between each user terminal and the ground The station is used as a relay for communication, and an optimization model of joint beam hopping of multiple low-orbit satellites and transmission power of user terminals is constructed. In each time slot, a hierarchical heterogeneous multi-agent deep reinforcement learning method is used to solve the optimization variables of the optimization model in a single time slot. The optimization variables include the uplink transmission power of the corresponding user terminal output by the user terminal agent, the beam hopping pattern of the corresponding low-orbit satellite output by the low-orbit satellite agent, and the sub-channel allocation of the user terminal, so as to achieve the optimization of the low-orbit satellite and the user terminal according to the obtained beam pattern scheme and the uplink transmission power, further suppress beam interference and thus improve the efficiency of network resource utilization.

[0037] With the development of the future 6G integrated space-air-ground-sea communication system, the number of low-orbit Internet satellites will increase and the beam coverage area will shrink. Satellite communication terminals need to clarify their location information during registration, access and switching. Due to the particularity of sea users, especially IoT devices, it is impossible to calculate and solve on the user side, so the user terminal location can only be determined on the LEO satellite side.

[0038] Step 101 specifically includes: determining the position of each user terminal on the ocean by using a positioning method based on a time difference of arrival (TDOA) through a low-orbit satellite. At the same time, the ocean environment information of each user terminal can also be obtained through the low-orbit satellite.

[0039] The TDOA positioning algorithm is based on the time of arrival (TOA) and determines the position of the receiver through the intersection of three double-leaf hyperboloids on the same side. Its measurement value is obtained by the distance difference between the two transmitters and the receiver. Taking the satellite position as one focus of the hyperboloid and the other satellite position as the other focus, and substituting them into the distance difference expression of the TDOA measurement value, the analytical expression of the receiver position can be obtained. Due to the short communication time between satellites, real-time positioning can be achieved. In order to fully grasp the marine environmental information, LEO satellites must also be equipped with a variety of sensors to monitor parameters such as ocean temperature, salinity, ocean currents, and wave height. Through the real-time collection and analysis of these data, accurate environmental information support can be provided to marine end users.

[0040] Among them, the position of each user terminal on the ocean is determined by a low-orbit satellite and a positioning method based on arrival time difference is adopted, specifically including: obtaining the signal arrival time difference between at least 3 low-orbit satellites and the same user terminal respectively through at least 4 low-orbit satellites; based on at least 3 of the arrival time differences, a weighted least squares (Weighted Least Squares, WLS) algorithm is used to solve the coordinates of the user terminal, that is, the position of the user terminal.

[0041] The TDOA positioning method determines the position of each user terminal on the ocean. More specifically, it includes: in the spatial rectangular coordinate system, taking the center of mass of the earth as the origin, setting the coordinates of the user terminal P to be located as (x, y, z), setting the coordinates of the low-orbit satellite as (x i ,y i ,z i ), the receiving end transmits signals to the satellite and obtains the satellite time difference for observation and positioning. The TDOA of the signal can be obtained by the difference in signals received between satellites. The user terminal on the sea surface transmits a radio signal with a specific code and time mark. This signal contains a timestamp, marking the exact time when the signal leaves the ground station. After the satellite receives this signal, the receiver on the satellite records the exact time when the signal is received. In order to accurately locate, multiple satellites are required to perform TDOA measurements.

[0042] Each satellite provides an independent time difference data, which is combined and processed by a complex algorithm. The TDOA equation is: Among them, R i is the distance between the target point P and the i-th satellite. Square the above formula and set Rewrite the above formula as: R i,j =cΔt i,j =R i -R j , The corresponding TDOA value can be obtained from the signal transmitted by the satellite. c is the signal propagation speed, and the distance between one of the low-orbit satellites and the terminal is used as a reference, and this satellite is used as a reference station. i,j , Δt i,j are the distance between the i-th satellite and the j-th satellite and the transmission time difference respectively.

[0043] Substituting i=1 into the above formula, we get: R1 2 =K1-2x1x-2y1y-2z1z+x 2 +y 2 +z 2 、x i,1 =x i -x1;y i,1 =y i -y1;z i,1 =z i -z1,

[0044] When solving the above formula, at least 4 equations are required. When the number of satellites n≥4, at least 3 TDOA values ​​can be obtained. The TDOA algorithm usually uses the least squares algorithm to find the minimum sum of squares of the error and obtain the optimal solution of the system. The weighted least squares algorithm adds a weighted matrix to the least squares algorithm to solve the heteroscedasticity problem of the original algorithm. The TDOA measurement value contains errors. The WLS algorithm weights the TDOA measurement value of each satellite to obtain more accurate positioning accuracy.

[0045] The target equation is constructed according to the TDOA measurement equation: Y = AX, assuming that δ is the residual, δ = Y-AX, and the weight matrix is ​​H. The WLS algorithm derives the weighted value that minimizes the residual. That is the estimated value of X. The specific expression is: Let the partial derivative of function f(x) be 0, and we can get the coordinates of the user terminal: -2A T H(Y-AX)=0, X=(A T HA) -1 A T HY. Y is an observation value matrix, which contains the time difference information received by multiple receivers. A is a design matrix, which includes the location of each receiver, the location of the user terminal and other related parameters that may exist.

[0046] After determining the position of the user terminal on the sea surface, the satellite needs to optimize and divide the wave positions, including determining the number of wave positions and the position of the center point. The goal is to cover all user terminals with the least number of wave positions, ensure that there is a certain interval between the wave positions, and make the user terminal as close to the center point as possible to reduce interference between adjacent wave positions and reduce user transmission power. In this application, it is considered to use several circular wave positions with the same radius to cover all users. The system delay is positively correlated with the number of wave positions. The more wave positions the same beam serves, the greater the delay. Therefore, the goal is to cover all user terminals with the least number of wave positions. This problem is to solve the P-center problem in plane geometry. In this way, the wave position distribution can be optimized, the system delay and interference can be reduced, and the communication efficiency can be improved.

[0047] First, determine the number of beam positions. Use several circular beam positions with the same radius (preset value) to cover all user terminals. Figure 2 The figure shows a flow chart for determining the number of wave positions. Under the premise that the user terminal position on the sea surface is known, the coverage radius of the satellite beam is set to ensure that users within this radius can obtain satisfactory service quality, and the optimization goal is established: minimize the number of required wave positions while ensuring that each user is covered by at least one beam; constraint condition: the coverage radius of each beam is fixed and all users must be covered by at least one beam.

[0048] Wherein, step 102 specifically includes the following steps:

[0049] Step 1021: put the position of each user terminal into a position list, and initialize the number of wave positions to 1.

[0050] Step 1022: The central user position of all user terminals in the current position list is used as the initial wave position center point (x1, y1), and the initial wave position center point is placed in the initial wave position center point list, and the initial wave position center point placed in the initial wave position center point list is used as the current wave position center point, with the geometric center point (x1, y1) as the center and the preset value as the radius, to construct the first wave position (circular area).

[0051] Step 1023: Calculate the Euclidean distances of all user terminals to the initial wave position center point and determine which user terminals are covered, i.e., user terminals whose distances are less than or equal to the set radius. Remove user terminals in the current location list that are within the coverage range of a circle with a preset radius and centered at the current wave position center point, to obtain an updated location list.

[0052] Step 1024: Determine whether there are still user terminals in the updated location list, that is, determine whether there are still uncovered user terminals in the updated user location list.

[0053] If there is a user terminal, the number of wave positions is increased and step 1025 is executed.

[0054] Step 1025: update the user terminal position whose distance from the current wave position center point is greater than 2 times the preset value as the current wave position center point, and put the current wave position center point into the initial wave position center point list.

[0055] Step 1026: Use the updated location list as the current location list, return to the step of removing user terminals in the current location list that are within the coverage of a circle with a preset radius and centered at the current wave position center point, and obtain an updated location list.

[0056] If the user terminal does not exist, execute step 1027.

[0057] Step 1027: Output the initial wave position center point list, one wave position center point corresponds to one wave position, and all wave positions are numbered.

[0058] Step 1028: In order to obtain the best position of each wave position center point and ensure that the wave position can cover all user points in the best way, it is necessary to further optimize the wave position center point position. The Weiszfeld algorithm is used to update the position of each wave position center point in the initial wave position center point list, and each wave position center point after the update is used as a candidate wave position center point. The Weiszfeld algorithm is an iterative algorithm for solving the minimum maximum distance of a single facility, and is particularly suitable for processing the coordinates of points in continuous space, such as finding a point on a plane so that the sum of the distances from this point to a set of given points is minimized.

[0059] The specific implementation process of the Weiszfeld algorithm includes: Step 81: Select an initial point as the starting position of the algorithm. In this application, this starting point is the wave position center point of the initial wave position center point list obtained in step 1027.

[0060] Step 82: Use the following formula to iteratively update the position of the wave center point: Among them, d(x i ,x (k) ) represents the distance between the user terminal in the wave position and the current iteration point x (k) The weighted Euclidean distance between i is the weight, x (k+1) is the updated wave position center point, x (k) To update the center point of the previous wave position, x i is the location of the user terminal, and n is the number of user terminals.

[0061] Step 83: After each iteration, check whether the position difference between the new and old wave position center points is less than a predefined threshold ε, that is: ||x (k+1) -x(k) ||<ε. If this condition is met, the algorithm converges and the iteration can be stopped. Otherwise, use x (k+1) As the input point for the next iteration, step 82 is repeated.

[0062] After traversing all the wave positions in the initial wave position center point list, when the algorithm converges, the final iteration point x is output (k +1) As the final wave center point.

[0063] Considering that the user terminal cannot communicate directly with the ground station, in this case, by introducing the LEO satellite as a relay, an uplink connection between the sea terminal and the LEO satellite is established, and the process of satellite backhaul to the ground station is as follows: Figure 3 As shown. Under the premise of knowing the network information such as the user terminal location and channel status, a multi-LEO satellite joint beam hopping and terminal transmission power optimization model is established. The optimization goal is to maximize the network throughput, while considering multiple factors, including the number of users scheduled per time slot, power constraints, the association between the center point of the wave position and the satellite, the wave position activation status, the number of sub-channels occupied by the user, and binary variable constraints. These constraints ensure the effective allocation of resources and help solve the resource allocation problem in satellite communication networks. Improve resource utilization and avoid interference by optimizing beam hopping patterns and frequency reuse.

[0064] The signal model is described. In the uplink resource scheduling scenario of the LEO satellite communication system, it is assumed that there are M LEO satellites covering the entire area. The satellite is equipped with a planar phased array antenna, which divides the area covered by the above LEO satellite into N wave positions. In particular, one satellite can cover multiple wave positions, which is a major feature of the non-geostationary satellite orbit (NGSO) satellite communication system. Assume that there are K marine user equipment (MUE), And N<<K. The satellite position is represented by d m =(x m ,y m ,h m ), the MUE position can be expressed as d k =(x k ,y k ,h k ), the coordinate of the wave position center is d n =(x n ,y n ,h n). In order to make full use of spectrum resources and improve spectrum utilization, it is assumed that all beams occupy the entire satellite bandwidth B, that is, the multi-satellite beam hopping system adopts full frequency reuse (FFR), and there is co-channel interference between beams. It is assumed that users in each beam use orthogonal multiple access (OMA), which means that users in the beam use different sub-channels and there is no interference between users in the beam. Ideally, the network topology remains unchanged during the performance observation period.

[0065] In such a scenario, the communication link operates in the Ka-band (26.5-40 GHz), and the satellite m and user terminal k channels follow the following standard model: mk =ω mk +ρ mk 10log 10 d mk +ψ, where δ mk represents the path loss of all users associated with the satellite, ω mk represents the intercept parameter, i.e., the path loss at 1 meter distance (dB), ρ mk is the path loss exponent, d mk represents the distance between satellite m and user k, which can be expressed as: ψ represents the fitting deviation (dB), which is a Gaussian random variable with a mean of zero and a variance of Then, the channel gain experienced by the signal received by user terminal k at wave position n is: Where G(θ) represents the transmit antenna gain of user terminal m pointing to wave position n, β mk is the Rice fading channel gain coefficient, which represents the small-scale fading between satellite m and user terminal k. The receiving antenna gain can be expressed as: Where θ is the angle between the line connecting user k and satellite m and the center line of the beam pointing to the beam position, that is, the off-axis angle. The distance from satellite m to user terminal k, the distance from the satellite to the center of the beam, and the distance from user terminal k to the center of beam n are expressed as d m,k , d m,n , d k,n , then the off-axis angle can be given by the law of cosines: in, G0 is the beam peak gain, defined as η is the antenna efficiency, which is generally equal to 0.65, N′ represents the phased array antenna, θ 3dBis the 3 dB gain angle of the antenna, J1(·) and J3(·) represent the first and third Bessel functions respectively, u(θ)=2.07123sinθ / sinθ 3dB .

[0066] make is a set of time slots. In a certain time slot t, let represents the wave position matrix associated with the satellite, Indicates that wave position n can be covered by satellite m, is a binary variable, Indicates that beam n is activated, Indicates inactivation, activated by the binary variable The matrix is ​​defined as make represents the user terminal association matrix, Indicates that user terminal k resides in wave position n. Since there may be overlaps between wave positions, user terminal k may be covered by multiple wave positions. Considering that there are multiple users in each wave position, and there may be differentiated service requirements between users; in order to reduce the interference level and meet differentiated performance requirements, it is assumed that full spectrum is multiplexed between beams, the total bandwidth is B, and the bandwidth of each subchannel is ΔB; satisfying B = S × ΔB, that is, the number of subchannels is The noise power spectrum density is N0. Considering that the marine communication users are sparsely distributed and the user service requirements are differentiated, it is assumed that different numbers of sub-channels can be allocated to users within the beam for uplink transmission; define Indicates that in time slot t, user terminal k in wave position n covered by satellite m is allocated subchannel s for uplink transmission. The corresponding set can be expressed as Correspondingly, in time slot t, the uplink transmission power of user terminal k in wave position n covered by satellite m on subchannel s is For all users, the transmission power set in time slot t is defined as Correspondingly, the uplink received signal-to-noise ratio of user terminal k scheduled on subchannel s under beam n covered by satellite m is It can be expressed as:

[0067]

[0068] in, It represents the channel gain of user terminal k in wave position n covered by satellite m on subchannel s in time slot t.

[0069] Therefore, the corresponding transmission rate can be written as:

[0070] When user terminal k needs to allocate multiple sub-channels for business needs and network benefits, the uplink transmission rate It can be expressed as:

[0071] The optimization goal of the optimization model proposed in this application is to jointly optimize the satellite associated wave position, wave position activation and terminal transmission power to maximize the overall system throughput. Therefore, within T time slots, the optimization problem corresponding to the maximization of the system average throughput, that is, the optimization model can be expressed as:

[0072]

[0073] in, represents the average value of T time slots, K represents the number of user terminals, and w k represents the weight of user terminal k. The weight coefficient can be adjusted dynamically according to the performance requirements of marine users. represents the uplink transmission rate of multiple sub-channels allocated to user terminal k in time slot t; C1 represents the minimum number of users scheduled in each time slot, C2 represents the uplink transmission power constraint of the user terminal, C3 represents the constraint that the wave position is associated with at most one satellite in time slot t, C4 represents the constraint that the wave position can only be activated or deactivated by one satellite, C5 represents the constraint that the user terminal belongs to at most one wave position, M represents the number of low-orbit satellites, N is the number of wave positions, Indicates that the wave position n in time slot t is covered by the low-orbit satellite m, Indicates that beam n is activated in time slot t, Indicates that beam n in time slot t is not activated. Indicates that user terminal k resides in wave position n in time slot t, K min Indicates the minimum number of users; It means that in time slot t, user terminal k in wave position n covered by low-orbit satellite m is allocated subchannel s for uplink transmission; P represents the uplink transmission power of user terminal k in subchannel s in the coverage area n of low-orbit satellite m in time slot t; max,k Indicates the upper limit of the uplink transmit power of user terminal k; is the set of time slots, is the wave position set, A collection of user terminals.

[0074] The above optimization problem is a mixed integer nonlinear programming problem involving continuous variables. and discrete variables These variables are coupled with each other, making the solution space combinatorially huge. In addition, the non-convex nature of the objective function and constraints increases the complexity of the algorithm. To solve this problem, the problem is redefined as a collaborative distributed partially observable Markov decision process (Dec-POMDP) ​​and solved using a multi-agent deep reinforcement learning method.

[0075] Interference management in uplink transmission depends not only on user terminals but also on LEO satellites. In order to solve the optimization problem within a single time slot, a hierarchical heterogeneous multi-agent deep reinforcement learning method (HH-MADRL) is adopted. Considering the real-time and distributed requirements, the LEO satellite and the sea surface terminal are modeled as independent agents, and the problem is remodeled as Dec-POMDP. Specifically, the task of the LEO satellite is to optimize its associated wave position and activation problem. It should be noted that the same satellite can cover multiple wave positions, but the same wave position can only be activated by one satellite. The terminal agent is responsible for determining the appropriate transmission power level. The Deep Reinforcement Learning (DRL) method uses a deep neural network (DNN) for policy optimization. By continuously interacting with the environment, the agent can learn and optimize its strategy to maximize the reward.

[0076] Among them, step 104 specifically includes: in the process of heterogeneous multi-agent deep reinforcement learning, a distributed partially observable Markov decision process is used to model each of the user terminal agents and each of the low-orbit satellite agents, and through shared hybrid network and local network collaborative training and decision-making, the uplink transmission power of the user terminal and the beam hopping pattern of the low-orbit satellite are output respectively; the local network is a local Q network.

[0077] In order to learn the strategy of the intelligent agent, this application adopts a novel heterogeneous coordination algorithm to build a shared hybrid network to train a centralized Q function, which can be decomposed into individual Q functions of LEO satellite and terminal intelligent agents. After the trained local Q function, LEO satellite and terminal can make their own decisions.

[0078] In the process of heterogeneous multi-agent deep reinforcement learning, each of the user terminal agents and each of the low-orbit satellite agents is centrally trained, specifically including: each of the user terminal agents uses a local Q network to learn the Q value, each of the low-orbit satellite agents uses a local Q network to learn the Q value, the Q value output by each local Q network is input into a shared hybrid network, the shared hybrid network outputs a joint Q value, and the trained local Q networks are deployed to each corresponding user terminal or low-orbit satellite; the input of each local Q network is the observation value of the corresponding agent, and the output is the Q value of the action, and then the corresponding action decision is selected according to the Q value.

[0079] Dec-POMDP is an extension of Markov decision process, designed to handle decision-making problems of distributed multi-agents, where each agent can only observe part of the state of the environment. Dec-POMDP allows these agents to make decisions with limited, local information and collaborate to achieve a common goal or optimize the collective payoff. In each time slot, the environment information is represented by the state s t ∈S to describe each agent Receive a single observation It provides some information about the state t . Then, each agent in the strategy Choose an action under the guidance of The combined action of all agents Make the environment transition to the next state s t+1 , and all agents receive a shared reward. The reward here refers to the cumulative reward, that is: Where γ∈[0,1) is the discount factor. It should be noted that in this application, LEO satellites and sea surface user terminals are considered to be two types of heterogeneous agents, each with different observation and action spaces. All agents make decisions at the end of each time slot to ensure that operations related to user association or resource allocation can be performed in subsequent time slots. According to the optimization problem described in step 3, the basic components of Dec-POMDP are defined as follows.

[0080] Observation: Due to partial observability, each agent can only perceive environmental information within its specified perception range. Observations include the observations of user terminal agents and low-orbit satellite agents.

[0081] The observation values ​​of the user terminal agent include the current coordinates of the user terminal and the transmission power of the previous time slot, that is, in, is the observation value of user terminal k in time slot t, is the coordinate of user terminal k in time slot t, is the transmission power of user terminal k in time slot t-1.

[0082] The observation values ​​of the LEO satellite agent include the current (time slot t) coordinates of the LEO satellite, the location of each user terminal, the average throughput of multiple time slots before the current time slot, and the transmission power of all associated user terminals in the previous time slot. The set of wave positions associated with the previous time slot The set of wave positions activated in the current time slot And the sub-channel allocation of the wave position activated in the current time slot in, is the observation value of low-orbit satellite m in time slot t, is the coordinate of the low-orbit satellite m in time slot t, is the estimated coordinates of user terminal k in time slot t, is the average throughput of multiple time slots before time slot t. and are the wave positions activated by LEO satellite m multi-agent in time slot (t-1), and Indicates that all wave positions are inactive.

[0083] The status includes the position of each low-orbit satellite, the position of each user terminal, the set of wave positions associated with the previous time slot, the set of wave positions activated in the current time slot, the transmission power of each user terminal, the average throughput of multiple time slots before the current time slot, and the sub-channel allocation in the wave positions activated in the current time slot.

[0084] The status is:

[0085] The actions include the actions of the user terminal agent and the actions of the low-orbit satellite agent.

[0086] Since the trajectory of LEO satellite is fixed and the operating speed is basically stable, in each time slot, LEO satellite must adjust the antenna pointing to cover the wave position to provide fair service. The actions of the low-orbit satellite agent include the set of associated wave positions in the current time slot, the set of activated wave positions in the current time slot, and the allocation of sub-channels within the activated wave positions in the current time slot; the actions of the user terminal agent include the transmission power of the current time slot.

[0087] The actions of the LEO satellite agent are: in Indicates the wave position indicator vector selected for association in time slot t, is a binary vector whose element value is 0 or 1, indicating whether the LEO satellite activates its associated wave position in time slot t. Indicates the sub-channel allocation in the wave numbered n activated by the LEO satellite.

[0088] The actions of the user terminal agent are:

[0089] The action space of the satellite agent is discrete, while the action space of the user terminal agent is continuous. Since searching for the optimal action in the continuous space is still challenging for MADRL configuration, all continuous actions are considered to be discretized. Specifically, the terminal transmit power is processed as follows: the transmit power [0,P max ] is quantized into b power levels, denoted as: Such discrete power levels constitute the action space of the user terminal agent.

[0090] The goal of the reward optimization problem is to maximize the average system throughput during the entire service period. To achieve this, a greedy strategy is adopted and the reward is defined as a function of the average system throughput for each time slot. The goal of MADRL is to find an optimal joint strategy Expected return can be maximized, where R0 is the reward at the initial moment. Usually, the optimal Q function is used to describe the t Start, take action t , and then follow the optimal strategy π * Expected return at time: Where R t is the reward for time slot t. Given Q * (s t ,a t ), the optimal strategy π * We can get this by choosing a greedy strategy:

[0091] Learning optimal policies in a multi-agent setting is much more complex than in the single-agent case. One approach is to train a centralized Q function that depends on global states and joint actions by extending DQN to a multi-agent setting. However, as the number of agents increases, the training of this function becomes challenging, and it lacks the ability to generate local policies for distributed execution. Another approach is to train each agent independently, treating other agents as part of the environment. Due to non-stationarity issues, this approach often fails to converge because the environment of each agent evolves as the policies of other agents are updated. This problem is particularly challenging in the marine communication scenario of this application due to the different capabilities, roles, and contributions of LEO satellites and surface terminal agents. This application adopts a heterogeneous MADRL framework to address the above challenges.

[0092] This application adopts the Heterogeneous Coordination QMIX (HC-QMIX) algorithm, which is an extension of the QMIX framework and can coordinately learn local strategies for LEO satellites and terminal agents. The basic concept of HC-QMIX includes estimating the joint Q-value as a complex nonlinear combination of individual Q-values ​​of heterogeneous agents conditioned only on local observations. This process is called heterogeneous value decomposition. By ensuring the monotonicity of this combination function, each agent chooses greedy behavior based on the local Q-value to achieve global optimality. To implement this method, HC-QMIX includes a set of local Q-networks for learning individual Q-values ​​for LEO satellites and maritime terminal agents, and a shared hybrid network that combines these Q-values ​​into a joint Q-value, such as Figure 4 shown.

[0093] Each of the local Q networks includes a first fully connected layer, a gated recurrent unit, and a second fully connected layer, such as Figure 5 As shown in (a), a single local network (local Q network) represents a single Q function for each agent, which takes the agent's observation as input and generates Q values ​​for all actions in the action space. Based on these Q values, an arbitrary strategy is used to determine the optimal behavior. t Processed into embedded feature z t ; The gated recurrent unit maintains a hidden state h t , which is composed of the current embedding feature z t and the hidden state h of the previous time slot t-1 The second fully connected layer uses the current hidden state h t As input, generate the Q value of all actions, that is Finally, use these Q values ​​to determine the optimal action a t Its Q t .

[0094] Shared hybrid network represents joint Q tot (s t ,a t ) value and a single Q t Specifically, it takes the outputs of all local Q networks as input and generates a joint Q tot (s t ,a t ) value. The shared hybrid network includes an input layer and an output layer, such as Figure 5 As shown in (b), the input layer is expressed as: The output layer is expressed as:

[0095] Among them, Q t is the set of Q values ​​output by each local Q network, Win and b in are the weight matrix and bias vector of the input layer, respectively. is the embedding feature output by the input layer, σ in and σ out are activation functions, W out and b out are the weight matrix and bias vector of the output layer, Q tot (s t ,a t ) is the joint Q value output by the output layer, s t is the state in time slot t during deep reinforcement learning, a t is the action in time slot t during deep reinforcement learning.

[0096] This application adopts a framework of centralized training and distributed execution, which ensures that global information is utilized during training while allowing independent decision-making during execution. Specifically, during centralized training, the agent interacts with the environment using a greedy strategy derived from a local Q network to collect experience. These interactions are organized into time periods of uniform length. At the beginning of each time period, the state of the environment is reset, and the hidden state of the local Q network of each agent is initialized to an all-zero vector. The transfer tuple {(s t ,{o t},{a t},s t+1 ,{o t+1},r t ,{h t})} t＝1:T Stored in the replay buffer. Periodically extract a batch of time periods from the buffer and use stochastic gradient ascent to train the network parameters. Specifically, the parameter θ = {θ LEO ,θ MUE ,θ hyper}Update by minimizing the loss function: in θ - Indicates the target network parameters that are updated periodically.

[0097] After reaching convergence, the trained local Q networks are deployed to the corresponding user terminals or low-orbit satellites. The user terminal agents and the low-orbit satellite agents are distributed in a distributed manner through the formula Choose greedy actions to output the optimal policy.

[0098] in, is the optimal strategy output by agent i in time slot t. If ξ=LEO, agent i is a low-orbit satellite agent. If ξ=MUE, agent i is a user terminal agent. is a collection of user terminals, For low-orbit satellite collection, Q ξ,i (o ξ,i ,a ξ,i ) is the sum of the Q values ​​of the actions, o ξ,i is the observed value of agent i, a ξ,i is the action of agent i.

[0099] The design of the communication process between the LEO satellite network system and the marine user terminal in each time slot follows the beam pattern scheme, user scheduling strategy and uplink transmission power configuration determined in step 104, aiming to optimize communication efficiency and resource utilization. Specifically, first, the LEO satellite adjusts its antenna array according to the optimal beam pattern calculated in step 104 to form a high-precision directional beam to ensure that the signal can cover the corresponding wave position. At the same time, based on the user scheduling strategy, the system dynamically allocates communication resources and determines which marine terminals can transmit data in the current time slot, thereby avoiding unnecessary competition and interference. The marine terminal user adjusts the power level of its transmitter according to the uplink transmission power setting indicated in step 104 to ensure that the signal is neither too strong to cause waste nor too weak to be drowned by noise when it reaches the LEO satellite. This power control helps to extend the battery life of the marine terminal and maintain good communication quality. The communication process of the current time slot ends, and the system automatically transitions to the next time interval. During this process, the LEO satellite network system will recalculate and update the beam pattern, user scheduling, and uplink power parameters based on the communication effect and environmental changes of the previous time slot. This closed-loop dynamic adjustment mechanism ensures that the system can continue to adapt to changing communication needs and external conditions. The entire process constitutes a cycle, from the initial parameter setting to the final strategy optimization, and the communication between the LEO satellite and the ocean terminal user continues until the entire operation cycle ends or a specific stop condition is met.

[0100] The present application has the following technical effects: (1) Estimate the approximate location and marine environment information of ocean terminal users based on the TDOA positioning method on LEO satellites; be able to estimate the location information and network information of ocean user terminals, which is convenient for beam hopping design and on-demand coverage. (2) A method for determining the number of wave positions is proposed based on the location information of sea surface users, and the location of the center point of the candidate wave position is further determined; it is possible to realize the allocation of coverage tasks of multiple LEO satellites while avoiding excessive coverage overlap, thereby improving on-demand coverage capabilities and resource utilization efficiency. (3) Based on the acquired location and network information, a joint optimization model of multi-LEO satellite joint beam hopping and terminal transmission power is established; by optimizing beam hopping and uplink transmission power, interference can be further suppressed to improve network resource utilization efficiency. (4) Based on the hierarchical heterogeneous multi-agent deep reinforcement learning method to solve the optimization problem within a single time slot, the marine user terminal and LEO satellite decide their own uplink transmission power and beam hopping pattern scheme according to the algorithm output; centralized training and distributed execution online deployment can be realized; LEO satellite and marine terminal users complete the communication process of the corresponding time slot according to the obtained beam pattern scheme, user multi-degree strategy and uplink transmission power. LEO satellite and marine terminal users complete the communication process of the corresponding time slot according to the beam pattern scheme, user multi-degree strategy and uplink transmission power obtained in the previous time slot, and the network system enters the next time slot and repeats the above steps in sequence to achieve resource optimization of the entire communication process.

[0101] This application can meet the communication coverage requirements of unevenly distributed sea users and differentiated service performance, and realize the joint optimization of beam hopping and terminal transmission power.

[0102] Based on the same inventive concept, the embodiment of the present application also provides a device for on-demand service coverage of marine communications based on low-orbit satellite beam hopping for implementing the above-mentioned method for on-demand service coverage of marine communications based on low-orbit satellite beam hopping. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above-mentioned method, so the specific limitations in the embodiments of one or more devices for on-demand service coverage of marine communications based on low-orbit satellite beam hopping provided below can be found in the above-mentioned limitations on the method for on-demand service coverage of marine communications based on low-orbit satellite beam hopping, and will not be repeated here.

[0103] In an exemplary embodiment, Figure 6 As shown, a marine communication on-demand service coverage device based on low-orbit satellite beam hopping is provided, comprising:

[0104] The user terminal location determination module is used to determine the location of each user terminal on the ocean through low-orbit satellites.

[0105] The beam position determination module is used to determine the number of beam positions and the position of the center point of each candidate beam position according to the position of each user terminal.

[0106] The optimization model building module is used to build an optimization model of multiple low-orbit satellite joint beam hopping and user terminal transmission power based on the position of each user terminal and the position of the center point of each candidate wave position, using the low-orbit satellite as a relay for communication between each user terminal and the ground station.

[0107] A solution module is used to solve the optimization variables of the optimization model in a single time slot by adopting a hierarchical heterogeneous multi-agent deep reinforcement learning method in each time slot, wherein the optimization variables include the uplink transmission power of the corresponding user terminal output by the user terminal agent, the beam hopping pattern of the corresponding low-orbit satellite output by the low-orbit satellite agent, and the sub-channel allocation of the user terminal.

[0108] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0109] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A method for on-demand service coverage of marine communications based on low-orbit satellite beam hopping, characterized in that: The method for on-demand service coverage of ocean communication based on low-orbit satellite beam hopping includes: Determine the location of each user terminal on the ocean through low-orbit satellites; Determine the number of wave positions and the position of the center point of each candidate wave position according to the position of each user terminal; Based on the location of each user terminal and the location of the center point of each candidate beam position, the low-orbit satellite is used as a relay for each user terminal to communicate with the ground station, and an optimization model of multiple low-orbit satellite joint beam hopping and user terminal transmission power is constructed; In each time slot, a hierarchical heterogeneous multi-agent deep reinforcement learning method is used to solve the optimization variables of the optimization model in a single time slot. The optimization variables include the uplink transmission power of the corresponding user terminal output by the user terminal agent, the beam hopping pattern of the corresponding low-orbit satellite output by the low-orbit satellite agent, and the sub-channel allocation of the user terminal.

2. The method for on-demand service coverage of marine communication based on low-orbit satellite beam hopping according to claim 1, characterized in that: The location of each user terminal on the ocean is determined by low-orbit satellites, including: Through low-orbit satellites, the location of each user terminal on the ocean is determined by using the arrival time difference positioning method.

3. The method for on-demand marine communication service coverage based on low-orbit satellite beam hopping according to claim 2, characterized in that: Through low-orbit satellites, the location of each user terminal on the ocean is determined by using a positioning method based on arrival time difference, including: Obtaining, by using at least four low-orbit satellites, signal arrival time differences between at least three low-orbit satellites and the same user terminal; Based on at least three of the arrival time differences, a weighted least squares algorithm is used to solve the coordinates of the user terminal.

4. The method for on-demand service coverage of ocean communication based on low-orbit satellite beam hopping according to claim 1, characterized in that: The number of wave positions and the position of the center point of each candidate wave position are determined according to the position of each user terminal, specifically including: Put the location of each user terminal into a location list; The central user position of all user terminals in the current position list is used as the initial wave position center point, and the initial wave position center point is added to the initial wave position center point list, and the initial wave position center point added to the initial wave position center point list is used as the current wave position center point; Remove user terminals in the current location list that are within the coverage area of ​​a circle with a preset radius and a current wave position center point as the center, to obtain an updated location list; Determine whether the user terminal still exists in the updated location list; If there is a user terminal, the user terminal position whose distance from the current wave position center point is greater than 2 times the preset value is updated as the current wave position center point, and the current wave position center point is added to the initial wave position center point list; The updated location list is used as the current location list, and the user terminals in the current location list that are within the coverage area of ​​a circle with a preset radius and a center point of the current wave position as the center are removed to obtain an updated location list; If there is no user terminal, output the initial wave position center point list; The Weiszfeld algorithm is used to update the position of each wave position center point in the initial wave position center point list, and each wave position center point after the update is used as a candidate wave position center point.

5. The method for on-demand service coverage of marine communication based on low-orbit satellite beam hopping according to claim 1, characterized in that: The optimization goal of the optimization model is to maximize the weighted average rate of all users in T time slots. The optimization model can be expressed as: in, represents the average value of T time slots, K represents the number of user terminals, and w k represents the weight of user terminal k, represents the uplink transmission rate of multiple sub-channels allocated to user terminal k in time slot t; C1 represents the minimum number of users scheduled in each time slot, C2 represents the uplink transmission power constraint of the user terminal, C3 represents the constraint that the wave position is associated with at most one satellite in time slot t, C4 represents the constraint that the wave position can only be activated or deactivated by one satellite, C5 represents the constraint that the user terminal belongs to at most one wave position, M represents the number of low-orbit satellites, N is the number of wave positions, Indicates that the wave position n in time slot t is covered by the low-orbit satellite m, Indicates that beam n is activated in time slot t, Indicates that beam n in time slot t is not activated. Indicates that user terminal k resides in wave position n in time slot t, K min Indicates the minimum number of users; It means that in time slot t, user terminal k in wave position n covered by low-orbit satellite m is allocated subchannel s for uplink transmission; P represents the uplink transmission power of user terminal k in subchannel s in the coverage area n of low-orbit satellite m in time slot t; max,k Indicates the upper limit of the uplink transmit power of user terminal k; is the set of time slots, is the wave position set, A collection of user terminals.

6. The method for on-demand service coverage of ocean communication based on low-orbit satellite beam hopping according to claim 1, characterized in that: In each time slot, a hierarchical heterogeneous multi-agent deep reinforcement learning method is used to solve the optimization variables of the optimization model in a single time slot, specifically including: In the process of heterogeneous multi-agent deep reinforcement learning, a distributed partially observable Markov decision process is used to model each of the user terminal agents and each of the low-orbit satellite agents, and the uplink transmission power of the user terminal and the beam hopping pattern of the low-orbit satellite are output respectively through collaborative training and decision-making of a shared hybrid network and a local network; the local network is a local Q network; In the process of heterogeneous multi-agent deep reinforcement learning, each of the user terminal agents and each of the low-orbit satellite agents is centrally trained, specifically including: each of the user terminal agents uses a local Q network to learn the Q value, each of the low-orbit satellite agents uses a local Q network to learn the Q value, the Q value output by each local Q network is input into a shared hybrid network, the shared hybrid network outputs a joint Q value, and the trained local Q networks are deployed to each corresponding user terminal or low-orbit satellite; the input of each local Q network is the observation value of the corresponding agent, and the output is the Q value of the action, and then the corresponding action decision is selected according to the Q value.

7. The method for on-demand service coverage of ocean communication based on low-orbit satellite beam hopping according to claim 6, characterized in that: In the process of heterogeneous multi-agent deep reinforcement learning, a distributed partially observable Markov decision process is used for modeling, where observations include the observations of the user terminal agent and the observations of the low-orbit satellite agent. The observation values ​​of the user terminal agent include the current coordinates of the user terminal and the transmission power of the previous time slot; The observation values ​​of the low-orbit satellite agent include the current coordinates of the low-orbit satellite, the position of each user terminal, the average throughput of multiple time slots before the current time slot, the transmission power of all associated user terminals in the previous time slot, the set of wave positions associated with the previous time slot, the set of wave positions activated in the current time slot, and the subchannel allocation of the wave positions activated in the current time slot; The status includes the position of each low-orbit satellite, the position of each user terminal, the set of wave positions associated with the previous time slot, the set of wave positions activated in the current time slot, the transmit power of each user terminal, the average throughput of multiple time slots before the current time slot, and the subchannel allocation in the wave positions activated in the current time slot; The actions include the actions of the user terminal agent and the actions of the low-orbit satellite agent; The actions of the user terminal agent include the transmit power of the current time slot; The actions of the low-orbit satellite agent include the current time slot associated wave position set, the current time slot activated wave position set, and the sub-channel allocation within the current time slot activated wave position; The reward is a function of the average system throughput in the current time slot.

8. The method for on-demand marine communication service coverage based on low-orbit satellite beam hopping according to claim 6, characterized in that: Each of the local Q networks comprises a first fully connected layer, a gated recurrent unit, and a second fully connected layer; The shared hybrid network includes an input layer and an output layer, and the input layer is expressed as: The output layer is expressed as: Among them, Q t is the set of Q values ​​output by each local Q network, W in and b in are the weight matrix and bias vector of the input layer, respectively. is the embedding feature output by the input layer, σ in and σ out are activation functions, W out and b out are the weight matrix and bias vector of the output layer, Q tot (s t ,a t ) is the joint Q value output by the output layer, s t is the state in time slot t during deep reinforcement learning, a t is the action in time slot t during deep reinforcement learning.

9. The method for on-demand service coverage of ocean communication based on low-orbit satellite beam hopping according to claim 6, characterized in that: After deploying the trained local Q networks to the corresponding user terminals or low-orbit satellites, the user terminal agents and the low-orbit satellite agents are distributed in a distributed manner through the formula Choose greedy actions to output the optimal strategy; in, is the optimal strategy output by agent i in time slot t. If ξ=LEO, agent i is a low-orbit satellite agent. If ξ=MUE, agent i is a user terminal agent. is a collection of user terminals, For low-orbit satellite collection, Q ξ,i (o ξ,i ,a ξ,i ) is the sum of the Q values ​​of the actions, o ξ,i is the observation value of agent i, a ξ,i is the action of agent i.

10. A marine communication on-demand service coverage device based on low-orbit satellite beam hopping, characterized in that: The on-demand service coverage device for ocean communication based on low-orbit satellite beam hopping includes: A user terminal location determination module is used to determine the location of each user terminal on the ocean through low-orbit satellites; A beam position determination module, used to determine the number of beam positions and the position of the center point of each candidate beam position according to the position of each user terminal; The optimization model building module is used to build an optimization model of multi-low-orbit satellite joint beam hopping and user terminal transmission power based on the location of each user terminal and the location of each candidate beam position center point, using the low-orbit satellite as a relay for each user terminal to communicate with the ground station; A solution module is used to solve the optimization variables of the optimization model in a single time slot by adopting a hierarchical heterogeneous multi-agent deep reinforcement learning method in each time slot, wherein the optimization variables include the uplink transmission power of the corresponding user terminal output by the user terminal agent, the beam hopping pattern of the corresponding low-orbit satellite output by the low-orbit satellite agent, and the sub-channel allocation of the user terminal.

Images