Satellite random access method based on decoupling reinforcement learning

Abstract

A satellite random access method based on decoupling reinforcement learning belongs to the technical field of communication, and comprises the following steps: constructing a multi-user uplink satellite random access transmission scene; establishing a code domain and power domain combined non-orthogonal multiple access random access transmission model; designing a random access strategy space of the codebook-power-time slot joint domain non-orthogonal multiple access random access transmission model, and performing decoupling dimension reduction on the strategy space; constructing an optimization problem with system throughput maximization as a target; according to the method, the optimal access strategy of each piece of user equipment is obtained through optimization solution, and the optimal random access scheme is designed according to the optimal access strategy of each piece of user equipment. Establishing a code domain and power domain combined non-orthogonal multiple access random access data packet detection model, and performing data packet detection; and establishing a code domain and power domain combined non-orthogonal multiple access random access method evaluation index and carrying out performance analysis. According to the invention, the system can continuously maintain high throughput after overload, and the system performance after overload is improved.

Description

Technical Field

[0001] The invention relates to a satellite random access method based on decoupled reinforcement learning, belonging to the technical field of communication. Background Art

[0002] Low-orbit satellite Internet has become a beneficial extension and supplement to the ground network with its all-weather, wide coverage and large capacity. With the explosive growth of various machine-type communication user devices, their access needs have increased dramatically. In the random access method, each user device accesses the channel in a random competitive manner, without the need to negotiate with the system or other devices, without the need to apply for authorization from the access point, and without the need for signaling transmission. It can be sent as soon as it arrives, without waiting, and improves the transmission efficiency of short data packets of user devices. Due to its high spectrum efficiency, high flexibility, low signaling overhead and low access delay, the random access method has become an important access solution for low-orbit satellite Internet that carries large-scale machine-type communication services.

[0003] At present, researchers usually apply power-time slot or codebook-time slot non-orthogonal multiple access schemes to random access systems, making multi-user transmission within the same channel possible. Random access schemes based on sparse code access use non-orthogonal codebooks and advanced multi-user detection algorithms to enable communication systems to operate under overload conditions. Non-orthogonal random multiple access schemes based on power domain use signal power differences and continuous interference cancellation technology to enable multiple user devices to share the same spectrum resources. However, due to the non-cooperative nature of the random access process, some satellite user devices may choose the same resources to transmit data packets, resulting in collisions and conflicts, and a decrease in system throughput. Summary of the invention

[0004] The technical problem solved by the present invention is: to overcome the shortcomings of the prior art, provide a satellite random access method based on decoupled reinforcement learning, improve the problem of a cliff-like drop in throughput after a critical load point in a time-slot ALOHA-type method, design a reinforcement learning reward mechanism based on congestion control, so that the system can continue to maintain a high throughput after an overload, thereby improving the throughput of the system after an overload.

[0005] The technical solution of the present invention is: a satellite random access method based on decoupled reinforcement learning, comprising:

[0006] Construct a transmission scenario of multiple user uplink random access based on a single satellite;

[0007] According to the random access transmission scenario, a non-orthogonal multiple access random access model of a codebook-power-time slot joint domain is established;

[0008] Designing a random access strategy space for the non-orthogonal multiple access random access model, and performing decoupling and dimensionality reduction optimization on the strategy space;

[0009] According to the optimized random access strategy space, the optimization function of the non-orthogonal multiple access random access model is designed, and the optimization problem with the goal of maximizing the system throughput is established;

[0010] Solve the optimization problem to obtain the optimal codebook, power level, and time slot optimal access solution for each user equipment to achieve multi-user access;

[0011] A non-orthogonal multiple access random access data packet detection model is established in the codebook-power-time slot joint domain to detect and count the accessed multi-user data packets;

[0012] According to the detection and statistical results, the evaluation index of the non-orthogonal multiple access random access method in the codebook-power-time slot joint domain is established to perform multi-user access performance analysis.

[0013] Furthermore, the non-orthogonal multiple access random access model of the codebook-power-time slot joint domain is a multi-dimensional joint optimization access model based on random precoding and power multiplexing, and the multi-dimensional resources include time domain, code domain, and power domain.

[0014] Furthermore, the random access strategy space includes: the user equipment has L power levels to choose from, and the power set is represented by P = [P1, ..., P l ,...,P L ], where P l =Γ(Γ+1) L-l , represents the lth power level, Γ=2 r -1 represents the target signal to noise ratio, r is the transmission rate; there are V codebooks in total, and the sparse codebook set is represented by C = [C1,...,C v ,...,C V ], where codebook C v The length of a frame is M time slots; a frame contains N subframes, and the nth subframe is represented by T n , 1≤n≤N, each user equipment randomly selects a power level and codebook, and randomly accesses in each subframe; the maximum overload factor is ρ=VL / M, that is, the ratio of the maximum number of decodable satellite user equipment to the number of time slots in a subframe, and the maximum normalized throughput of the system is ρ; the strategy space based on the random access model is {P,C,T}, and the dimension size of the strategy space is L×V×N; the strategy space is decoupled into two domains: subframe {T} and power-codebook {P,C}. All users first make random selections in subframe {T}. If there is no conflict, they enter the power-codebook {P,C} for random selection. If there is a conflict, the subframe selection is re-performed.

[0015] Furthermore, the optimization goal of the optimization problem is to maximize the throughput, and the objective function is the total system throughput, that is, the ratio of the number of successfully decoded users without subframe, codebook and power collision to the number of time domain time slots of the system, and the constraints are the number of users, the number of time subframes, the number of codebooks, and the number of power levels.

[0016] Further, solving the optimization problem includes:

[0017] A decoupled distributed reinforcement learning algorithm is constructed, and the access process of each device in the satellite network is modeled as two decoupled Markov decision processes. Each device learns the access time slot, power level, and codebook in a distributed manner according to the feedback information received from the satellite, and updates its two Q tables using the corresponding Q-value function and reward function respectively. After selecting the subframe and codebook-power pair with the highest Q value according to the two Q tables, the device starts to transmit information. This process continues until convergence, until each device can find the optimal subframe, codebook, and power level for its transmission, and finally randomly accesses the satellite in the uplink.

[0018] Furthermore, the random access data packet detection model is established according to the message transmission joint continuous interference elimination algorithm, and the satellite transparently forwards the IoT device data to the ground gateway station to realize the multi-slot multi-user joint detection algorithm for the code domain and power domain.

[0019] Furthermore, in the time domain uplink, multiple user equipments are allowed to share the same time-frequency resources by using different transmit powers and codebooks; the satellite receiving end uses message passing combined with serial interference cancellation method for multi-user detection;

[0020] First, the message passing algorithm detector is used to detect Y n The codebook of the user in the received packet is detected. When there is no codebook conflict, the code domain decoding is successful and removed from the received signal. On the contrary, when a codebook conflict is detected, the power of the collision packet is detected using the continuous interference elimination method. When the target signal-to-interference-noise ratio is greater than the preset threshold, the user device data packet message is successfully decoded and deleted from the received data packet. This process will be repeated until all data packets are decoded.

[0021] Furthermore, the evaluation index of the random access method includes the normalized throughput of multi-user access, the algorithm convergence time and the normalized throughput of overload;

[0022] Furthermore, by comparing and analyzing the performance of a method using codebook and time slot as policy space, a method using power and time slot as policy space, a Q learning method using power domain and code domain as policy space but not using policy space decoupling, and a method using power domain and code domain as policy space but using reinforcement learning for action selection optimization, the Q learning method is a decoupled distributed reinforcement learning algorithm.

[0023] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of a satellite random access method based on decoupled reinforcement learning are implemented.

[0024] A satellite random access device based on decoupled reinforcement learning comprises a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the satellite random access method based on decoupled reinforcement learning are implemented.

[0025] The advantages of the present invention compared with the prior art are:

[0026] (1) The present invention effectively solves the problem of low access success rate of large-scale users in burst access scenarios. By proposing a multi-dimensional joint SCMA-NOMA-SA random access protocol, users can improve the access dimension and increase the competition space by randomly multiplexing time slots, codebooks, and power.

[0027] (2) In order to solve the collision problem of power, codebook and time slot in SCMA-NOMA-SA, the present invention proposes an optimization method based on reinforcement learning, which effectively solves the collision problem of power, codebook and time slot and improves the access efficiency.

[0028] (3) The present invention overcomes the problem of slow convergence of traditional reinforcement learning algorithms when the strategy space and action set are large. By proposing a decoupling-based reinforcement learning algorithm to improve the convergence speed, a Q learning mechanism on the user side learns the access subframe of the satellite Internet user, while another two-dimensional Q learning mechanism jointly learns the sparse codebook and power level selected by the user, reducing the number of action sets from the multiplier level to the adder level, thereby improving the algorithm convergence speed and overcoming the slow convergence problem of traditional single-dimensional non-decoupled reinforcement learning algorithms under large-scale strategy sets.

[0029] (4) The present invention further improves the throughput threshold effect of the time-slotted ALOHA-type protocol after passing the critical load point. By adopting a distributed intelligent global congestion control optimization solution, the network congestion problem is effectively solved, so that the system throughput can be slowly reduced after overload, thereby improving the throughput performance of the system after overload. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0031] Figure 1 It is a schematic diagram of the system model of the present invention;

[0032] Figure 2 Schematic diagram of two Q tables in the DDQL algorithm of the present invention;

[0033] Figure 3 The simulation diagram of the relationship between the throughput and normalized load of different random access methods;

[0034] Figure 4 It is a convergence simulation diagram of the present invention at different normalized load G values;

[0035] Figure 5 This is a load control simulation diagram of the present invention. DETAILED DESCRIPTION

[0036] In order to better understand the above technical scheme, the technical scheme of the present invention is described in detail below through the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present invention and the specific features in the embodiments are detailed descriptions of the technical scheme of the present invention, rather than limitations on the technical scheme of the present invention. In the absence of conflict, the embodiments of the present invention and the technical features in the embodiments can be combined with each other.

[0037] The following is a further detailed description of a satellite random access method based on decoupled reinforcement learning provided by an embodiment of the present invention in conjunction with the accompanying drawings of the specification. The specific implementation method may include:

[0038] Step S1: construct a transmission scenario of uplink random access for multiple users based on a single satellite. A typical satellite random access consists of K user devices and a single satellite system. The uplink adopts a time slot ALOHA access scheme based on TDMA. A frame of the system is divided into N subframes, and each subframe consists of M time slots.

[0039] Step S2: According to the constructed random access transmission scenario, a non-orthogonal multiple access random access model of the codebook-power-time slot joint domain is established. The access model adopts the non-orthogonal multiple access NOMA-SCMA-SA access scheme of the power domain and the code domain. Each user device selects a power P and a codebook C to transmit in a time slot, and the sum of multiple user signals selected in the nth time slot is recorded as X. n ;

[0040] Step S3: Design the access strategy space of the codebook-power-time slot joint domain random access model, and perform decoupling and dimensionality reduction optimization on the strategy space. The user equipment has L power levels to choose from, and the power set is represented by P = [P1, ..., P l ,...,P L ], where P l=Γ(Γ+1) L-l , represents the lth power level, Γ=2 r -1 represents the target signal to interference plus noise ratio (SINR), and r is the transmission rate. There are V codebooks in total, and the sparse codebook set is represented by C = [C1,...,C v ,...,C V ], where codebook C v The length of a frame is M time slots. A frame contains N subframes, and the nth subframe is represented by T n (1≤n≤N), each user equipment randomly selects a power level and codebook, and randomly accesses in each subframe. The maximum overload factor of the system is ρ = VL / M, that is, the ratio of the maximum number of satellite user equipment that can be decoded by the system to the number of time slots in the subframe, and the maximum normalized throughput of the system is ρ. The policy space based on SCMA-NOMA-SA is {P,C,T}, and the dimension of the policy space is L×V×N. In order to speed up the policy search and algorithm convergence time, the policy space is decoupled into two domains: subframe {T} and power-codebook {P,C}. All users first make a random selection in the subframe {T}. If there is no conflict, they enter the power-codebook {P,C} for random selection. If there is a conflict, the subframe selection is re-performed. The dimension of the policy space is reduced from L×V×N to L×V+N, which greatly improves the search efficiency of the policy space.

[0041] Step S4: Based on the optimized strategy space, an optimization problem with the goal of maximizing system throughput is constructed. Multi-user random access can be constructed as an optimization problem with the maximum system throughput as the optimization goal. The optimization problem is expressed as:

[0042]

[0043] in It is used to indicate whether the data of the kth user equipment is successfully demodulated by selecting the nth subframe, the vth codebook and the lth power level. Indicates that the user has selected a unique codebook and power combination (not selected by other user devices) in the nth subframe and successfully demodulated the data packet, otherwise it is equal to 0. The multi-user random access problem is a non-convex problem, and its optimal solution is difficult to solve directly. In order to minimize the number of codebook and power collisions, the system is operated in a maximum overload state. The present invention proposes a decoupled reinforcement learning method (Dual Decoupled Q-Learning Algorithm, DDQL), which constructs each user device as an intelligent agent, and multiple intelligent agents collaborate to automatically select the optimal subframe, codebook and power configuration of each device, thereby obtaining the optimal solution to the optimization problem.

[0044] Step S5: For the optimization problem, a decoupled reinforcement learning algorithm is proposed to optimize and solve the problem, and the optimal codebook, power level, and time slot access scheme for each user device are obtained to achieve successful access of multiple users. To solve the optimization problem (1), the random selection process of the user terminal in the subframe and power-codebook domains is modeled as two Markov decision processes, and the selection occupancy in the subframe and codebook-power strategy space is regarded as the state information of the two Markov decision processes. The selection behavior of multiple user terminals in the strategy space is regarded as an action. Each user device is an intelligent agent, which selects an action according to the state information. The state-action score value is represented by a Q function, which is used to guide the selection of the best action in the current state. The satellite access system rewards the current action of the user terminal according to the reward mechanism, and the reward value is fed back to the Q function to enter the next iterative update of the state-action Q function. After multiple rounds of iterative learning, each intelligent agent obtains an optimal action.

[0045] Let Q k (n) is the Q value when the kth device sends a data packet in the nth subframe, Q k (v,l) is the Q value when the kth device selects the vth codebook and the lth power level. In each time step t, a user device selects an action to optimize the two Q value functions, and the Q table update rule is as follows:

[0046]

[0047] Among them, α∈[0,1] is the learning rate, R is the reward function, and γ∈[0,1] is the discount factor, which is used to represent the impact of future rewards on Q value updates. Let CL=[CL1,...,CL n ,...,CL N ] is the number of user equipments that select the same subframe among N subframes, CL n The larger the value, the more serious the system congestion. The multi-user MPA-SIC detection method can determine the codebook and power collision status of the device in each subframe. In order to control the cliff-like drop in throughput at the overload critical point, an overload critical point congestion control optimization learning function is designed to improve the throughput of the system after overload. The reward function under the two Q learning mechanisms is defined as,

[0048]

[0049] In terms of subframe learning rewards, when the congestion level of the nth subframe exceeds V×L, R k(n) is equal to -CL(n) / N, and a penalty reward is given to it so that the user terminal does not select subframes exceeding the maximum overload number. In terms of power-codebook learning reward, if the power-codebook pair selected by the user equipment is also selected by other users, that is, a power-codebook pair collision occurs, then R k (v,l) is equal to -1, thus suppressing this type of selection. Each user device performs its own actions according to the Q table, that is,

[0050] The algorithm model process is as follows: First, each user device is initialized and all Q values ​​in the Q table are set to zero. Second, the user device uses the randomly selected power and codebook to send data packets in the randomly selected subframe. Finally, each device receives status feedback from the satellite access system and updates its two Q tables using formulas (2)-(5). After selecting the subframe and power-codebook pair with the highest Q value according to the two Q tables, this process continues until the algorithm model converges and each user device can find the best subframe, power and codebook level.

[0051] Step S6: Establish a non-orthogonal multiple access random access data packet detection model in the codebook-power-time slot joint domain to detect and count the successfully accessed multi-user data packets. In the TDMA uplink, NOMA-SCMA technology allows multiple user devices to share the same time-frequency resources by using different transmission powers and codebooks. n After reaching the satellite receiving end, it is Y n The satellite receiving end uses the message passing combined with serial interference cancellation MPA-SIC method to perform multi-user detection. First, the MPA detector is used to detect Y n The codebook of the user in is used for detection. When there is no codebook conflict, SCMA decoding is successful and it is removed from the received signal, which is recorded as Conversely, when a codebook conflict is detected, the SIC is used to detect the power of the collision packet. When the SINR is greater than the threshold Γ, the user equipment data packet message is successfully decoded and deleted from the received data packet. This process will be repeated until all data packets are decoded;

[0052] Step S7, according to the statistical results, establish the evaluation index of the non-orthogonal multiple access random access method in the codebook-power-time slot joint domain, and perform multi-user access performance analysis. Establish three indicators of normalized throughput of multi-user access, algorithm convergence time and normalized throughput of overload. Compare and analyze the performance by comparing with DQL-SCMA-SA (a method using codebook and time slot as policy space), DQL-NOMA-SA (a method using power and time slot as policy space), Q-NOMA-SCMA (a Q learning method using NOMA and SCMA as policy space but not using policy space decoupling), and NOMA-SCMA-SA, ρ=300% / ρ=400% (using NOMA and SCMA as policy space but using reinforcement learning for action selection optimization).

[0053] The solution provided in the embodiment of the present invention specifically includes:

[0054] Step S1: construct a random access transmission scenario for multiple user uplinks based on a single satellite. Figure 1 As shown in Figure 1, a typical satellite random access consists of K user equipment and a satellite system. The uplink adopts a TDMA-based slotted ALOHA access scheme. A frame of the system is divided into N subframes, and each subframe consists of M time slots.

[0055] Step S2: According to the constructed random access transmission scenario, a non-orthogonal multiple access random access model of the codebook-power-time slot joint domain is established. The transmission model adopts the non-orthogonal multiple access NOMA-SCMA-SA access scheme of the power domain and the code domain. Each user equipment selects a power P and a codebook C to transmit in a time slot, which is denoted as S k (n,l,v), where 1≤n≤N,1≤l≤L,1≤v≤V, means that the kth user selects lth power and vth codebook to access in the nth subframe. Let the sum of multiple user signals selecting the nth time slot be X n Set the number of system user equipment K = [10,500], the number of subframes N to 25, the codebook dimension M to 4, the power level L to 2, the number of codebooks V to [6,8], corresponding to the overload factor ρ = [300%, 400%].

[0056] Step S3: Design the access strategy space of the codebook-power-time slot joint domain random access model, and perform decoupling and dimensionality reduction optimization on the strategy space. The user equipment has L power levels to choose from, and the power set is represented by P = [P1, ..., P l ,...,P L ], where P l =Γ(Γ+1) L-l, represents the lth power level, Γ represents the target signal to interference plus noise ratio (SINR). There are V codebooks in total, and the sparse codebook set is represented by C = [C1,...,C v ,...,C V ], where codebook C v The length of a frame is M time slots. A frame contains N subframes, and the nth subframe is represented by T n (1≤n≤N), each user equipment randomly selects a power level and codebook, and randomly accesses in each subframe. The maximum overload factor of the system is ρ=VL / M, that is, the ratio of the maximum number of satellite user equipment that can be decoded by the system to the number of time slots in the subframe, and the maximum normalized throughput of the system is ρ. Set the number of user equipment to 10≤K≤500, L=2, P=[3.01dB,7.78dB], V=[6,8], N=25, M=4, then the corresponding maximum overload factor ρ=[300%,400%], the normalized load 0.1≤G≤5 (that is, the average number of users in a unit time slot). The strategy space based on SCMA-NOMA-SA is {P,C,T}={[P1,P2],[C1,...,C 6 / 8 ],[T1,...,T 25 ]}, the dimension of the policy space is L×V×N=300 / 400. In order to speed up the policy search and algorithm convergence time, the policy space is decoupled into two domains: subframe {T} and power-codebook {P,C}. All users first make a random selection in subframe {T}. If there is no conflict, they enter the power-codebook {P,C} for random selection. If there is a conflict, the subframe selection is re-performed. The dimension of the policy space is reduced from 300 and 400 to 37 and 41 dimensions, which greatly improves the search efficiency of the policy space.

[0057] Step S4: Based on the optimized strategy space, an optimization problem with the goal of maximizing system throughput is constructed. Multi-user random access can be constructed as an optimization problem with the maximum system throughput as the optimization goal. The optimization problem is expressed as:

[0058]

[0059] in It is used to indicate whether the data of the kth user equipment is successfully demodulated by selecting the nth subframe, the vth codebook and the lth power level. Indicates that the user has selected a unique codebook and power combination (not selected by other user devices) in the nth subframe and successfully demodulated the data packet, otherwise it is equal to 0. The multi-user random access problem is a non-convex problem, and its optimal solution is difficult to solve directly. In order to minimize the number of codebook and power collisions, the system is operated in a maximum overload state. The present invention proposes a decoupled reinforcement learning method (Dual Decoupled Q-Learning Algorithm, DDQL), which constructs each user device as an intelligent agent, and multiple intelligent agents collaborate to automatically select the optimal subframe, codebook and power configuration for each device, thereby obtaining a solution to the optimization problem.

[0060] Step S5: For the optimization problem, a decoupled reinforcement learning algorithm is proposed to optimize and solve the problem, and the optimal codebook, power level, and time slot access scheme for each user device are obtained to achieve successful access of multiple users. To solve the optimization problem (6), the random selection process of the user terminal in the subframe and power-codebook domains is modeled as two Markov decision processes, and the selection occupancy in the subframe and codebook-power strategy space is regarded as the state information of the two Markov decision processes. The selection behavior of multiple user terminals in the strategy space is regarded as an action. Each user device is an intelligent agent, which selects an action according to the state information. The state-action score value is represented by a Q function, which is used to guide the selection of the best action in the current state. The satellite access system rewards the current action of the user terminal according to the reward mechanism, and the reward value is fed back to the Q function to enter the next state-action Q function iterative update. After multiple rounds of iterative learning, each intelligent agent obtains an optimal action.

[0061] Let Q k (n) is the Q value when the kth device sends a data packet in the nth subframe, Q k (v,l) is the Q value when the kth device selects the vth codebook and the lth power level. In each time step t, a user device selects an action to optimize the two Q value functions, and the Q table update rule is as follows:

[0062]

[0063] Among them, α is set to 0.1 as the learning rate, γ is 0.8 as the discount factor, and R is the reward function, which is used to represent the impact of future rewards on the Q value update. Let CL = [CL1,...,CL n ,...,CL N ] is the number of user equipments that select the same subframe among N subframes, CL n is the intermediate state of the iterative learning process, CL nThe larger the value, the more serious the system congestion. The multi-user MPA-SIC detection method can determine the codebook and power collision status of the device in each subframe. In order to control the cliff-like drop in throughput at the overload critical point, an overload critical point congestion control optimization learning function is designed to improve the throughput of the system after overload. The reward function under the two Q learning mechanisms is defined as,

[0064]

[0065] In terms of subframe learning rewards, when the congestion level of the nth subframe exceeds [12,16], R k (n) is equal to -CL(n) / 25, and a penalty reward is given to it so that the user terminal does not select subframes exceeding the maximum overload number. In terms of power-codebook learning reward, if the power-codebook pair selected by the user equipment is also selected by other users, that is, a power-codebook pair collision occurs, then R k (v,l) is equal to -1, thus suppressing this type of selection. Each user device performs its own actions according to the Q table, that is,

[0066] The algorithm model process is as follows: First, each user device is initialized and all Q values ​​in the Q table are set to zero. Second, the user device uses the randomly selected power and codebook to send data packets in the randomly selected subframe. Finally, each device receives status feedback from the satellite access system and updates its two Q tables using formulas (2)-(5). After selecting the subframe and power-codebook pair with the highest Q value according to the two Q tables, this process continues until the algorithm model converges and each user device can find the best subframe, power and codebook level.

[0067] like Figure 2 As shown in Figure 1, the update of the Q table during the learning and updating process of the algorithm model is given. Each user device has a one-dimensional Q table and a two-dimensional Q table to store the Q values ​​associated with a specific state-action pair. The vertical axis of the one-dimensional Q table represents the subframe index, while the horizontal and vertical axes of the two-dimensional Q table correspond to the power level index and codebook index, respectively. Figure 2 It can be seen that the user equipment uses the second power level and the Vth codebook after iterative learning, and transmits its data packet in the first subframe, thus resolving the codebook and power collision.

[0068] Step S6: Establish a non-orthogonal multiple access random access data packet detection model in the codebook-power-time slot joint domain to detect and count the successfully accessed multi-user data packets. n After reaching the satellite receiving end, it is Y n The satellite receiving end uses the message passing combined with serial interference cancellation MPA-SIC method to perform multi-user detection. First, the MPA detector is used to detect Yn The codebook of the user in is used for detection. When there is no codebook conflict, SCMA decoding is successful and it is removed from the received signal, which is recorded as On the contrary, when a codebook conflict is detected, the SIC is used to detect the power of the collision packet. When the SINR is greater than the threshold Γ = 3.01dB, the user equipment data packet message is successfully decoded and deleted from the received data packet. This process will be repeated until all data packets are decoded.

[0069] Step S7, based on the statistical results, establish the evaluation index of the non-orthogonal multiple access random access method in the codebook-power-time slot joint domain, and perform multi-user access performance analysis. Establish three indicators of normalized throughput of multi-user access, algorithm convergence time, and normalized throughput of overload. DQL-SCMA-SA represents a method using codebook and time slot as a policy space; DQL-NOMA-SA represents a method using power and time slot as a policy space; Q-NOMA-SCMA represents a Q learning method using NOMA and SCMA as a policy space, but not using policy space decoupling; NOMA-SCMA-SA, ρ=300% / ρ=400% represents the use of NOMA and SCMA as a policy space, but using reinforcement learning for action selection optimization.

[0070] Figure 3 The performance comparison with other methods is shown. DQL-SCMA-SA represents the method using codebook and time slot as the policy space; DQL-NOMA-SA represents the method using power and time slot as the policy space; Q-NOMA-SCMA represents the Q-learning method using NOMA and SCMA as the policy space but not using the policy space decoupling; NOMA-SCMA-SA, ρ = 300% / ρ = 400% represents the method using NOMA and SCMA as the policy space but using reinforcement learning for action selection optimization.

[0071] The throughput of this method has been substantially improved. When the overload factor ρ = 300%, the peak throughput of the proposed DDQL method is 150% higher than that of the DQL-NOMA-SA scheme and 200% higher than that of the DQL-SCMA-SA scheme. This is mainly due to the additional access freedom of the present invention. In addition, the throughput gain of this scheme is significantly improved compared to the NOMA-SCMA-SA scheme. For example, the throughput gain is 160% when ρ = 400%. From Figure 3 It can also be seen that the peak throughput of the new method is 250% higher than that of the Q-NOMA-SCMA scheme based on traditional Q-Learning. At the same time, the proposed DDQL scheme enables the system to converge to the optimal state.

[0072] In order to evaluate the convergence performance of the DDQL method, the proposed scheme is compared with the traditional Q-Learning based scheme in Figure 4 The results are compared with different normalized loads G. Figure 4 It can be seen that the DDQL scheme proposed in the present invention can achieve the optimal solution under different G and has a faster convergence speed than the traditional Q-Learning algorithm. In addition, the convergence time of the method increases with the increase of normalized load. This is because the increase in load level causes each MTD to change its state and action more frequently, so the system needs a longer convergence time to reach the optimal solution.

[0073] In order to demonstrate the throughput performance of the proposed scheme at the critical load point, Figure 5 The relationship between normalized throughput and normalized load under different parameter settings is shown. By adopting a distributed intelligent global congestion control optimization solution, the throughput of the present invention decreases slowly after the critical load point 3.0, while the throughput of the traditional DQL-NOMA-SA solution decreases rapidly after the critical load point 2.0. The present invention improves the throughput threshold effect after the critical load point of the time slot ALOHA type protocol, so that the throughput of the system can be slowly reduced after overload, thereby improving the throughput performance of the system after overload.

[0074] It can be seen that in the satellite random access scenario, the present invention establishes a multi-dimensional joint optimization non-orthogonal random access transmission and detection model, constructs a throughput optimization problem, and proposes a decoupled reinforcement learning algorithm to solve the optimization problem, effectively achieving convergence and improving the total throughput of the system.

[0075] The present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and when the computer instructions are executed on a computer, the computer executes Figure 1 The method described.

[0076] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) containing computer-usable program code.

[0077] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0078] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0079] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0080] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

[0081] The contents not described in detail in the specification of the present invention belong to the common knowledge of those skilled in the art.

Claims

1. A satellite random access method based on decoupled reinforcement learning, characterized in that: include: Construct a transmission scenario where multiple users can access uplinks on demand based on a single satellite; According to the random access transmission scenario, a non-orthogonal multiple access random access model of a codebook-power-time slot joint domain is established; Designing a random access strategy space for the non-orthogonal multiple access random access model, and performing decoupling and dimensionality reduction optimization on the strategy space; According to the optimized random access strategy space, the optimization function of the non-orthogonal multiple access random access model is designed, and the optimization problem with the goal of maximizing the system throughput is established; Solve the optimization problem to obtain the optimal codebook, power level, and time slot optimal access solution for each user equipment to achieve multi-user access; A non-orthogonal multiple access random access data packet detection model is established in the codebook-power-time slot joint domain to detect and count the accessed multi-user data packets; According to the detection and statistical results, the evaluation index of the non-orthogonal multiple access random access method in the codebook-power-time slot joint domain is established to perform multi-user access performance analysis.

2. The satellite random access method based on decoupled reinforcement learning according to claim 1, characterized in that: The non-orthogonal multiple access random access model of the codebook-power-time slot joint domain is a multi-dimensional joint optimization access model based on random precoding and power multiplexing, and the multi-dimensional resources include time domain, code domain and power domain.

3. The satellite random access method based on decoupled reinforcement learning according to claim 1, characterized in that: The random access strategy space includes: the user equipment has L power levels to choose from, and the power set is represented by P = [P1, ..., P l ,...,P L ], where P l =Γ(Γ+1) L-l , represents the lth power level, Γ=2 r -1 represents the target signal to noise ratio, r is the transmission rate; there are V codebooks in total, and the sparse codebook set is represented by C = [C1,...,C v ,...,C V ], where codebook C v The length of a frame is M time slots; a frame contains N subframes, and the nth subframe is represented by T n , 1≤n≤N, each user equipment randomly selects a power level and codebook, and randomly accesses in each subframe; the maximum overload factor is ρ=VL / M, that is, the ratio of the maximum number of decodable satellite user equipment to the number of time slots in a subframe, and the maximum normalized throughput of the system is ρ; the strategy space based on the random access model is {P,C,T}, and the dimension size of the strategy space is L×V×N; the strategy space is decoupled into two domains: subframe {T} and power-codebook {P,C}. All users first make random selections in subframe {T}. If there is no conflict, they enter the power-codebook {P,C} for random selection. If there is a conflict, the subframe selection is re-performed.

4. The satellite random access method based on decoupled reinforcement learning according to claim 1, characterized in that: The optimization goal of the optimization problem is to maximize the throughput, and the objective function is the total system throughput, that is, the ratio of the number of successfully decoded users without subframe, codebook and power collision to the number of time domain time slots of the system, and the constraints are the number of users, the number of time subframes, the number of codebooks, and the number of power levels.

5. The satellite random access method based on decoupled reinforcement learning according to claim 1, characterized in that: The solving of the optimization problem comprises: A decoupled distributed reinforcement learning algorithm is constructed, and the access process of each device in the satellite network is modeled as two decoupled Markov decision processes. Each device learns the access time slot, power level, and codebook in a distributed manner according to the feedback information received from the satellite, and updates its two Q tables using the corresponding Q-value function and reward function respectively. After selecting the subframe and codebook-power pair with the highest Q value according to the two Q tables, the device starts to transmit information. This process continues until convergence, until each device can find the optimal subframe, codebook, and power level for its transmission, and finally randomly accesses the satellite in the uplink.

6. The satellite random access method based on decoupled reinforcement learning according to claim 1, characterized in that: The random access data packet detection model is established according to the message transmission joint continuous interference elimination algorithm, and the satellite transparently forwards the IoT device data to the ground gateway station to realize the multi-slot multi-user joint detection algorithm for the code domain and the power domain.

7. The satellite random access method based on decoupled reinforcement learning according to claim 6, characterized in that: In the time domain uplink, multiple user devices are allowed to share the same time-frequency resources by using different transmit powers and codebooks; the satellite receiving end uses message passing combined with serial interference cancellation method for multi-user detection; First, the message passing algorithm detector is used to detect Y n The codebook of the user in the received signal is detected. When there is no codebook conflict, the code domain decoding is successful and removed from the received signal. On the contrary, when a codebook conflict is detected, the power of the collision packet is detected using the continuous interference elimination method. When the target signal-to-interference-noise ratio is greater than the preset threshold, the user equipment data packet message is successfully decoded and deleted from the received data packet. This process is repeated until all packets are decoded.

8. The satellite random access method based on decoupled reinforcement learning according to claim 5, characterized in that: The evaluation indexes of the random access method include normalized throughput of multi-user access, algorithm convergence time and normalized throughput of overload; The performance of the method using codebook and time slot as policy space, the method using power and time slot as policy space, the Q learning method using power domain and code domain as policy space but not using policy space decoupling, and the method using power domain and code domain as policy space but using reinforcement learning for action selection optimization are compared and analyzed; the Q learning method is a decoupled distributed reinforcement learning algorithm.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

10. A satellite random access device based on decoupled reinforcement learning, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

Images