LoRa spreading factor and power joint distribution method based on reinforcement learning

Abstract

The invention discloses a LoRa spreading factor and power joint distribution method based on reinforcement learning, and belongs to the field of low-power wide area network (LPWAN), and the method comprises the steps: constructing a single-gateway LoRa network; introducing a collision parameter to represent whether the reference terminal equipment is interfered by the interference terminal equipment or not; modeling and analyzing the signal to interference plus noise ratio of the LoRa signal at the gateway, and deducing and calculating the signal to interference plus noise ratio; deducing and calculating the transmission success probability by combining the demodulation characteristic of LoRa, and quantifying the influence of the spreading factor and the transmitting power on the transmission success probability; establishing a reinforcement learning algorithm by taking maximization of the overall transmission success probability of the system as a target; and distributing an optimal spread spectrum factor and transmitting power combination for each terminal device by using the algorithm. According to the invention, under the condition that a large number of terminal devices are accessed and channel resources are insufficient, the data packets sent by the terminal devices can be ensured to be correctly received, interference to other terminal devices is minimized, and the overall transmission performance is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of low-power wide-area network (LPWAN), and particularly relates to a method for jointly allocating LoRa spreading factor and power based on reinforcement learning. Background Art

[0002] As an extension and expansion of the Internet, the Internet of Things (IoT) technology can provide services such as data collection, data processing, and transmission, providing important technical support for the intelligent advancement of multiple fields. Considering the application scenarios of the IoT, its data transmission is usually sparse, and there is also a need for long-distance transmission performance. In addition, with the increase in the number of deployed terminal devices, factors such as the cost, power consumption, and service life of IoT terminal devices have become increasingly important.

[0003] To effectively address the low-power and long-distance communication requirements of IoT terminal devices, the low-power wide-area network (LPWAN) technology has emerged. LPWAN is a wireless communication technology designed specifically for IoT applications, capable of achieving long-distance communication with low power consumption. LoRa, as one of the representative technologies of LPWAN, is a modulation technology based on chirp spread spectrum (CSS). By optimizing the communication protocol and reducing the data transmission rate, it significantly reduces the energy consumption of terminal devices, while supporting communication distances of several kilometers or even dozens of kilometers. It has the characteristics of long distance, low power consumption, and low cost, supports two-way communication, and is flexibly deployable. It is being widely applied in various fields and has become an important choice for IoT solutions.

[0004] LoRa adopts a channel access protocol based on ALOHA. Terminal devices can send data packets for uplink access at any time. When multiple terminal devices using the same spreading factor (SF) transmit data simultaneously on the same channel, signal collisions will occur, resulting in packet loss and a decline in communication performance. With the deepening of intelligence in various fields and the advancement of IoT technology applications, it can be foreseen that the number of terminal devices will inevitably increase explosively. In such high-density application scenarios, the LoRaWAN protocol is restricted by the above factors, and the network performance deteriorates severely, making it difficult to be applied in large-scale deployment scenarios.

[0005] In response to a series of challenges faced by LoRa, such as packet loss caused by channel conflicts and interference, congestion problems brought about by network capacity limitations, and the impact of signal attenuation on coverage in complex environments, researchers have conducted extensive research. Considering the approximate orthogonality of signals with different SF modulations in the LoRaWAN protocol, many scholars have studied the SF allocation strategy in order to improve the performance of LoRa. The literature [Zhang C, Wang L, Jiao L, et al. A novel orthogonal LoRa multiple access algorithm for satellite Internet of Things [J]. China Communications, 2022, 19(3): 279-289.] utilizes the approximate orthogonality between different SFs to connect multiple LoRa users with different SFs to the same channel, thereby increasing the network throughput. The literature [Shilpa B, Kumar PR, Jha R K. Spreading Factor Optimization for Interference Mitigation in Dense Indoor LoRa Networks [C] / / 2023 IEEE IAS Global Conference on Emerging Technologies (GlobConET). IEEE, 2023: 1-5.] takes into account that higher SFs are more vulnerable to interference during transmission and that LoRa will default to a higher SF during retransmission. It modifies the retransmission mechanism and selects the SF with the highest success probability for each terminal device during retransmission, improving the network performance. Researchers generally believe that when LoRa signals modulated with the same SF collide, the signals cannot be correctly received due to the interference caused by the collision. For signals modulated with different SFs, researchers generally consider them to be completely orthogonal. However, some studies have pointed out that when the Signal-Interference-Noise Ratio (SINR) of a certain signal among two colliding LoRa signals at the receiver is not lower than the reception threshold, the signal can still be correctly received, and interference also occurs between LoRa signals modulated with different SFs. When the SINR of a certain signal at the receiver is lower than the reception threshold, the signal still cannot be received, which means that different SFs of LoRa are not completely orthogonal. Existing work mainly focuses on allocating the SF of LoRa to reduce the probability of signal collision and improve network performance.However, current researchers have paid little attention to the allocation of the transmission power of terminal devices. Considering that whether the interference between SFs and the interference within SFs will affect the signal reception depends on the SINR of the signal, and the SINR is related to the transmission power of the terminal device, the allocation of the transmission power of the terminal device also helps to improve network performance.

[0006] In the related research in the field of LoRa performance optimization, reinforcement learning is generally used for channel access control. By learning the network state and the behavior of terminal devices, the access strategy of terminal devices is dynamically adjusted to reduce channel conflicts and data loss, thereby improving the overall throughput of the network. In addition, in terms of resource allocation and power control, reinforcement learning is often used to optimize the transmission power, extend the battery usage time of terminal devices, and ensure communication quality at the same time. Finally, reinforcement learning is also used for packet scheduling and priority management, dynamically adjusting the transmission order according to the importance of packets and the network state to ensure the timely transmission of high-priority packets. In the literature [Hong S, Yao F, Zhang F, et al. Reinforcement Learning Approach for SF Allocation in LoRa Network[J]. IEEE Internet of Things Journal, 2023.], reinforcement learning is introduced to assist SF allocation in a single-base-station single-channel multi-user scenario. By effectively allocating SF resources, packet conflicts are reduced and network throughput is improved. The literature [Huang X, Jiang J, Yang S H, et al. A Reinforcement Learning Based Medium Access Control Method for LoRa Networks[J]. IEEE, 2020. DOI: 10.1109 / ICNSC48988.2020.9238127.] proposes a LoRa access method based on reinforcement learning, allowing terminal devices to select when to transmit data according to the environment and reducing the probability of packet collisions. In the related work of using reinforcement learning to optimize LoRa performance, most only consider the interference between signals using the same SF modulation and do not consider the interference between signals using different SF modulations. In addition, for the interference caused by the same SF, researchers usually ignore the impact of the capture effect, resulting in an incorrect estimation of the performance of LoRa in actual application scenarios.

[0007] In summary, due to the existence of the capture effect and the incomplete orthogonality between different SFs of LoRa, the existing SF allocation methods for LoRa performance optimization cannot meet the requirements. Therefore, there is an urgent need to design a joint allocation method of LoRa spreading factor and power based on reinforcement learning for the optimization requirements of LoRa systems. Summary of the Invention

[0008] In view of the deficiencies of the existing technology, considering the capture effect in LoRa and the problem that signals modulated with different SFs are not completely orthogonal, aiming to improve the performance of the LoRa network, the present invention proposes a method for jointly allocating the spreading factor and power of LoRa based on reinforcement learning. When a large number of terminal devices access and the channel resources are insufficient, the present invention can minimize the interference to other terminal devices while ensuring that the data packets sent by the terminal devices can be correctly received, thereby improving the overall transmission performance.

[0009] The technical solution adopted by the present invention to solve the technical problem is as follows:

[0010] A method for jointly allocating the spreading factor and power of LoRa based on reinforcement learning according to the present invention, which jointly considers the capture effect in LoRa and the incomplete orthogonality between signals modulated with different SFs, includes the following steps:

[0011] Step S1: Construct a LoRa network according to different LoRa access scenarios;

[0012] Step S2: Design a collision parameter to represent whether the reference terminal device is interfered by interfering terminal devices in the nth time slot;

[0013] Step S3: Model and analyze the signal-to-interference-plus-noise ratio (SINR) of LoRa signals at the gateway, deduce and calculate the SINR; combined with the demodulation characteristics of LoRa, deduce and calculate the transmission success probability, and quantify the influence of the spreading factor and transmit power on the transmission success probability;

[0014] Terminal device D i The transmission success probability in the nth time slot is:

[0015]

[0016] where θ capture is the threshold value at which the capture effect occurs, r i is the distance from the terminal device to the gateway, α is the path loss exponent, is the noise power, is the transmit power of terminal device D i , is the deterministic loss, f c is the carrier frequency, is the set of terminal devices, s jm c indicates whether the interfering terminal device D j selects the spreading factor m, s jm = 1 indicates that the interfering terminal device D j selects the spreading factor m, sjm = 0 indicates the interfering terminal device D j The spreading factor m is not selected; Indicates the terminal device D i And the interfering terminal device D j The collision situation in the nth time slot, Is the transmitting power of the interfering terminal device D j ; Is the transmitting power of the terminal device D i ; r j Is the distance from the interfering terminal device D j To the gateway, Is the set of spreading factors of the terminal device, Is the interfering terminal device D that selects the spreading factor k j The transmitting power used; the parameter ω n Indicates whether there is interference with the same SF, ω n = 1 indicates the existence of interference with the same SF, ω n = 0 indicates the non-existence of interference with the same SF;

[0017] According to formula (15), the final transmission success probability is only related to the SF selection situation, transmitting power, and the distance of the terminal device from the gateway;

[0018] Step S4: Introduce reinforcement learning to jointly allocate the spreading factor and transmitting power of LoRa to achieve the access and optimization of LoRa;

[0019] With the goal of maximizing the overall system transmission success probability, establish a reinforcement learning algorithm; use the reinforcement learning algorithm to allocate the optimal spreading factor and transmitting power combination for each terminal device.

[0020] Furthermore, the LoRa network includes I terminal devices, and the terminal device set is defined as Let the i th terminal device D i Be the reference terminal device, and its transmitting power is Then its received power is:

[0021]

[0022] Among them, Is the deterministic loss, f c Is the carrier frequency, r i Is the distance from the terminal device to the gateway, and α is the path loss exponent; h i Is the channel gain from the i-th terminal device D i To the gateway. Considering the Rayleigh fading channel condition, Then |h i |2 Follow an exponential distribution with parameter 1.

[0023] Furthermore, the LoRa network includes I terminal devices, and the set of terminal devices is defined as Let the i th terminal device D i be the reference terminal device, and the selection of its spreading factor is represented by the parameter s ij as follows:

[0024]

[0025] When the terminal device D i selects the spreading factor j, the parameter s ij is 1, otherwise the parameter s ij is 0.

[0026] Furthermore, the calculation formula of the collision parameter is as follows:

[0027]

[0028] where is the transmission period of the interfering terminal device D j n m' is the number of time slots occupied by the interfering terminal device D j to send data once, is the transmission period of the reference terminal device D i ; when the condition of formula (3) is met, at the nth time slot, the reference terminal device D i sends data and the interfering terminal device D j is in transmission, resulting in data overlap and thus a collision, and the collision parameter is 1.

[0029] Furthermore, the signal interference in LoRa is divided into in-SF interference and between-SF interference, and the signal-to-interference-plus-noise ratio of the terminal device is calculated in the following two cases:

[0030] (1) In the case of the existence of in-SF interference and in-SF interference being dominant, the signal-to-interference-plus-noise ratio of the terminal device D i selecting the spreading factor m at the nth time slot is:

[0031]

[0032] where is the noise power, NF is the receiver noise figure, BW is the bandwidth, is the received power of the interfering terminal device D j s jm c represents the interfering terminal device Dj Whether to select the spreading factor m, s jm = 1 indicates the interfering terminal device D j Select the spreading factor m, s jm = 0 indicates the interfering terminal device D j The spreading factor m is not selected; considering the case of interference with the same spreading factor, that is, the spreading factors used by the reference terminal device and the interfering terminal device are both m, then here Indicates the terminal device D i And the interfering terminal device D j The collision situation in the nth time slot;

[0033] Dividing both the numerator and denominator of formula (4) by the noise power gives:

[0034]

[0035] Among them, Is the signal-to-noise ratio of the signal sent by the terminal device D at the gateway, denoted as γ i ; h im ; i Is the channel gain from the ith terminal device D i To the gateway, |h i | 2 Follows an exponential distribution. At this time, γ im Follows an exponential distribution with parameters . Then formula (5) is further expressed as:

[0036]

[0037] When there is only interference within the SF, the probability of successful transmission is θ capture Is the threshold value for the capture effect to occur;

[0038] Substituting formula (6) gives:

[0039]

[0040] Denote the signal-to-noise ratio of the data packet sent by the terminal device at the gateway as γ im , then γ 11 ,...γ i-1m-1 ,γ i+1m+1 ,...γ IM Represents the signal-to-noise ratios of all possible combinations except the selected terminal device; Since γ im Follows an exponential distribution, for all γ im Marginalization gives:

[0041]

[0042] Denote the one in formula (8) as A; since γ follows an exponential distribution, according to the probability density function of the exponential distribution, calculating A gives: im

[0043]

[0044] where λ im obeys the parameter λ, x is the formal parameter used for integration; substituting A back into formula (8) gives:

[0045]

[0046] where is the transmit power of the interfering terminal device D j , is the transmit power of the terminal device D i , r j is the distance from the interfering terminal device D j to the gateway;

[0047] (2) In the case of no in-SF interference and only inter-SF interference, the signal-to-interference-plus-noise ratio i of the terminal device D that selects the spreading factor m at the nth time slot is expressed as:

[0048]

[0049] where P im is the transmit power of the terminal device D i , s jk is the spreading factor selection situation of the interfering terminal device D j , s jk =1 indicates that the interfering terminal device D j has selected the spreading factor k, s jk =0 indicates that the interfering terminal device D j has not selected the spreading factor k, P is the collision situation between the terminal device D i and the interfering terminal device D j , P jk is the transmit power of the interfering terminal device D j ;

[0050] Referring to the derivation process of formula (6), formula (11) is further expressed as:

[0051]

[0052] where γ jk + indicates the interfering terminal device D that selects the spreading factor k j ​Signal-to-noise ratio at the gateway;

[0053] At this time, the probability of successful transmission is θ m The decoding SINR threshold corresponding to the spreading factor m; Through a derivation similar to that in case (1), the mathematical expression for the probability of successful transmission is:

[0054]

[0055] Among them, The transmit power of the interfering device that selects the spreading factor k;

[0056] Combining the above two cases, since it is necessary to judge whether there is in-SF interference in the nth time slot, set the parameter ω n To indicate whether there is interference with the same SF, its definition is:

[0057]

[0058] According to formula (14), ω n Equal to 0 means that no interfering device and the reference device select the same SF in the nth time slot, and the judgment condition is: for SF = m, the sum of the collision parameters is 0; otherwise, it means that there is interference with the same SF.

[0059] Furthermore, the reinforcement learning algorithm is implemented using Q-learning, and it continuously updates to obtain the optimal Q-table according to the Bellman equation during the continuous exploration process. The update method of the Q value is:

[0060]

[0061] Among them, γ is the discount factor, β is the learning rate, (s t Is the current state, a t Is the current action, (s t+1 , is the next state, a t+1 Is the next action, (r t+1 Is the reward obtained after executing the action a t )), Q′(s t , a t ) is the target Q value, and Q(s t , a t ) is the current Q value.

[0062] Furthermore, in the reinforcement learning algorithm, the action is defined as a i =(m, P tx ), m is the spreading factor, P tx Is the transmit power; when the terminal device does not send data, its action is (m, P 0 ).

[0063] Further, in the reinforcement learning algorithm, the states of all terminal devices where \(n\) is the current time slot and \(T\) is the least common multiple of the transmission periods of all terminal devices.

[0064] Further, in the reinforcement learning algorithm, the transmission success probability of all terminal devices during the entire transmission process is:

[0065]

[0066] where \(T\) is the least common multiple of the transmission periods of all terminal devices, is the set of terminal devices, is the terminal device \(D\) i The probability of successful transmission in the \(n\)th time slot.

[0067] Further, in the reinforcement learning algorithm, the overall reward of the system is:

[0068]

[0069] where the parameter \(\zeta\) is used to balance the transmission success probability and power consumption, \(T\) is the least common multiple of the transmission periods of all terminal devices, is the set of terminal devices, is the terminal device \(D\) i The probability of successful transmission in the \(n\)th time slot, \(n\) is the current time slot, is the terminal device \(D\) i The transmit power of.

[0070] The beneficial effects of the present invention are:

[0071] On the premise of considering the capture effect and the incomplete orthogonality between SFs, the present invention models and analyzes the signal-to-interference-plus-noise ratio of LoRa signals at the gateway, and derives the system transmission success probability of LoRa. Since the transmission success probability is related to SF and transmit power, therefore, the present invention designs a reinforcement learning algorithm, takes the transmission success probability as the optimization goal, allocates SF and transmit power, and proposes a method for allocating LoRa spreading factor and transmit power based on reinforcement learning, thereby ensuring the performance improvement of the LoRa network.

[0072] Compared with the prior art, the advantages of the present invention are:

[0073] 1. Aiming at the problem of performance degradation of LoRa in large-scale access scenarios, the present invention jointly considers the capture effect existing in LoRa and the incomplete orthogonality between different SF modulated signals, and introduces the joint allocation of transmit power and spreading factor to improve the performance of the LoRa network.

[0074] 2. To study the probability of successful LoRa signal reception, the present invention comprehensively considers the channel gain and the interference situation among terminal devices, models the signal-to-interference-plus-noise ratio (SINR) of LoRa signals at the gateway, and further derives the overall probability of successful reception as the subsequent optimization objective.

[0075] 3. The present invention introduces a reinforcement learning algorithm to maximize the overall probability of successful reception in the LoRa network. Through the above-mentioned modeling and derivation process of the overall probability of successful reception, it is analyzed that the overall probability of successful reception of LoRa is related to the spreading factor and transmission power. The optimization problem of the overall probability of successful reception is further simplified to the allocation problem of the spreading factor and transmission power. By designing rewards, actions, and states, a joint spreading factor and power allocation algorithm for LoRa based on reinforcement learning is proposed. Under the premise of considering the capture effect and the incomplete orthogonality between SFs, the spreading factor and power are jointly allocated to ensure the improvement of LoRa performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] Figure 1 It is a model diagram of the LoRa network. DETAILED DESCRIPTION OF THE INVENTION

[0077] The following further elaborates on the present invention with reference to the accompanying drawings.

[0078] A joint spreading factor and power allocation method for LoRa based on reinforcement learning provided by the present invention. First, for the problem of spreading factor and transmission power allocation of terminal devices in the LoRa network, considering the scenario of single-channel multi-terminal device access, a single-gateway LoRa network model is established. This LoRa network model comprehensively considers the channel gain of terminal devices, the interference situation among terminal devices, and the impact of the allocation of SF and transmission power on communication performance. Secondly, by modeling and analyzing the SINR of LoRa signals at the gateway, the mathematical expression of SINR is derived, and further combined with the demodulation characteristics of LoRa, the formula for the probability of successful reception is derived, quantifying the impact of SF and transmission power on the probability of transmission success. Finally, with the goal of maximizing the overall system transmission success probability, a reinforcement learning algorithm is proposed to use the reinforcement learning algorithm to allocate the optimal combination of SF and transmission power for each terminal device, thereby improving the overall system performance while meeting the communication requirements.

[0079] The specific implementation process of a joint spreading factor and power allocation method for LoRa based on reinforcement learning provided by the present invention is as follows:

[0080] Step S1: Considering the scenario of single-channel multi-terminal device access, construct a single-gateway LoRa network;

[0081] As Figure 1As shown in the figure, consider a LoRa network with a single gateway. There are I end devices in the LoRa network. These end devices collect information from the environment through sensors and periodically send data packets containing the collected information to the gateway.

[0082] Considering common LoRa application scenarios, such as smart agriculture, intelligent water and electricity meters, urban infrastructure monitoring, etc., the positions of end devices are usually relatively fixed. Therefore, it is assumed that the distance from the end device to the gateway is fixed and known. At the same time, considering the scenario of massive access, the number of end devices far exceeds the number of available channels, resulting in multiple end devices inevitably using the same channel. Therefore, this invention studies the situation where the transmissions of multiple end devices are carried out on the same channel.

[0083] The set of end devices is defined as where the i-th end device is D i which is also the reference end device in this invention. The set of spreading factors of end devices is defined as where the i-th end device D i has a corresponding spreading factor of m i ,

[0084] Let the length of a single time slot be τ. The i-th end device D i uses time slots to send data once, denoted as the transmission duration δ i , that is The corresponding transmission period of the i-th end device D i is defined as t i , that is, it sends data once every time slots.

[0085] Let the transmit power of the i-th end device D i be Then its received power is expressed as:

[0086]

[0087] where is the deterministic loss, f c is the carrier frequency, r i is the distance from the end device to the gateway, and α is the path loss exponent; h i is the channel gain from the i-th end device D i to the gateway. Considering the Rayleigh fading channel condition, then |h i | 2 follows an exponential distribution with a parameter of 1.

[0088] Introduce parameter s ij Denote the i-th terminal device D i 's SF selection situation, expressed as:

[0089]

[0090] Specifically, when the i-th terminal device D i selects the spreading factor j, parameter s ij is 1, otherwise parameter s ij is 0.

[0091] Step S2: Design a collision parameter to represent whether the reference terminal device D i is interfered by the interfering terminal device D j in the n-th time slot;

[0092] Since it is necessary to determine which terminal devices cause interference when calculating the signal-to-interference-plus-noise ratio of a terminal device, therefore, the present invention designs a collision parameter to represent whether the reference terminal device D i is interfered by the interfering terminal device D j in the n-th time slot. Among them, the spreading factor selected by the reference terminal device D i is m, and the spreading factor selected by the interfering terminal device D j is m'. Then the specific calculation formula of the collision parameter is as follows:

[0093]

[0094] Among them, represents the collision situation between the terminal device D i selecting the spreading factor m and the interfering terminal device D j selecting the spreading factor m′ in the n-th time slot, n is the current time slot, is the transmission period of the interfering terminal device D j , n m' is the number of time slots occupied by the interfering terminal device D j to send data once, is the transmission period of the reference terminal device D i . When the condition of formula (3) is satisfied, in the n-th time slot, the reference terminal device D i sends data and the interfering terminal device D j is in transmission, resulting in data overlap and thus collision, and the collision parameter is 1.

[0095] Step S3: Model and analyze the signal-to-interference-plus-noise ratio (SINR) of LoRa signals at the gateway, and derive the mathematical expression of SINR; further, in combination with the demodulation characteristics of LoRa, derive the mathematical expression of the transmission success probability, and quantify the impact of the spreading factor (SF) and transmission power on the transmission success probability.

[0096] In the present invention, a reinforcement learning algorithm is subsequently introduced to jointly allocate the spreading factor and transmission power of LoRa. Therefore, it is necessary to study the transmission success probability of the LoRa system under the premise of considering the capture effect and the imperfect orthogonality between SFs.

[0097] Since the signal interference in LoRa is divided into two cases: in-SF interference and between-SF interference, and when in-SF interference exists, it dominates and the between-SF interference can be ignored. Therefore, the signal-to-interference-plus-noise ratio of the terminal device is discussed and calculated separately in the following two cases.

[0098] Case 1: There is interference from signals with the same SF, and in this case, the in-SF interference dominates.

[0099] In this case, the terminal device D that selects the spreading factor m i The signal-to-interference-plus-noise ratio at the nth time slot Is expressed as:

[0100]

[0101] Where, Is the noise power, NF is the receiver noise figure, BW is the bandwidth, Is the received power of the interfering terminal device D j Of, s jm c indicates whether the interfering terminal device D j Selects the spreading factor m, s jm = 1 indicates that the interfering terminal device D j Selects the spreading factor m, s jm = 0 indicates that the interfering terminal device D j Does not select the spreading factor m; considering the case of interference with the same spreading factor, that is, the spreading factors used by the reference terminal device and the interfering terminal device are both m, then the Here represents the collision situation between the terminal device D i And the interfering terminal device D j At the nth time slot.

[0102] Dividing both the numerator and denominator of formula (4) by the noise power, we get:

[0103]

[0104] Where, Is the terminal device Di The signal-to-noise ratio of the transmitted signal at the gateway is denoted as γ im . As mentioned above, |h i | 2 follows an exponential distribution. At this time, γ im follows an exponential distribution with parameter . Then formula (5) can be further expressed as:

[0105]

[0106] When there is only interference within the SF, the probability of successful transmission is θ capture is the threshold value at which the capture effect occurs, and its typical value is 6 dB.

[0107] The following is a further derivation of the probability formula for successful transmission:

[0108] Substituting formula (6) gives:

[0109]

[0110] Among them, the signal-to-noise ratio of the data packet transmitted by the terminal device at the gateway is denoted as γ im , then γ 11 ,...γ i-1m-1 ,γ i+1m+1 ,...γ IM represents the signal-to-noise ratios of all possible combinations except the selected terminal device. Since γ im follows an exponential distribution, for all γ im marginalization gives:

[0111]

[0112] Denote the in formula (8) as A. Since γ im follows an exponential distribution, according to the probability density function of the exponential distribution, calculating A gives:

[0113]

[0114] Among them, λ im is the parameter that follows λ, x is the formal parameter used for integration and has no practical meaning.

[0115] Substitute A back into formula (8) to get:

[0116]

[0117] Among them, is the transmit power of the interfering terminal device D j . is the transmission power of the terminal device D i , and r j is the distance from the interfering terminal device D j to the gateway.

[0118] Case 2: There is no interference within the spreading factor (SF), and only interference between SFs exists.

[0119] In this case, the signal-to-interference-plus-noise ratio (SINR) of the terminal device D i selecting the spreading factor m at the nth time slot is expressed as:

[0120]

[0121] where P im is the transmission power of the terminal device D i , s jk is the spreading factor selection situation of the interfering terminal device D j , s jk =1 indicates that the interfering terminal device D j has selected the spreading factor k, and s jk =0 indicates that the interfering terminal device D j has not selected the spreading factor k. P is the collision situation between the terminal device D i and the interfering terminal device D j , and P jk is the transmission power of the interfering terminal device D j .

[0122] Referring to the derivation process of formula (6), formula (11) can be further expressed as:

[0123]

[0124] where γ jk + represents the signal-to-noise ratio of the interfering terminal device D j selecting the spreading factor k at the gateway.

[0125] At this time, the probability of successful transmission is θ m is the decoding SINR threshold corresponding to the spreading factor m. Through a similar derivation as in Case 1, the mathematical expression of the probability of successful transmission is obtained as:

[0126]

[0127] where is the transmission power of the interfering device selecting the spreading factor k.

[0128] Combining the above two cases, since it is necessary to determine whether there is interference within the SF in the nth time slot, a parameter ω is set n to indicate whether there is interference within the same SF, which is defined as:

[0129]

[0130] According to formula (14), ω n being equal to 0 indicates that no interfering device and the reference device select the same SF in the nth time slot, and the judgment condition is: for SF = m, the sum of the collision parameters is 0. Otherwise, it indicates that there is interference within the same SF.

[0131] Then the probability of successful transmission of the terminal device D i in the nth time slot is:

[0132]

[0133] According to formula (15), the final probability of successful transmission only depends on the SF selection situation of the terminal device, the transmission power, and the distance between the terminal device and the gateway.

[0134] Step S4: With the goal of maximizing the overall system transmission success probability, establish a reinforcement learning algorithm, and use the reinforcement learning algorithm to allocate the optimal SF and transmission power combination for each terminal device, so as to improve the overall system performance while meeting the communication requirements;

[0135] Under the premise of considering the capture effect and the imperfect orthogonality between SFs, the present invention introduces reinforcement learning to jointly allocate the spreading factor and power of LoRa. In the related research on LoRa performance optimization, most studies focus on allocating the spreading factor to improve the performance of LoRa, pay little attention to the power allocation, and most studies assume that there is perfect orthogonality between different SFs. In addition, the existence of the capture effect is usually not concerned by researchers.

[0136] Q-learning is a model-free reinforcement learning algorithm, which is widely used for an agent to learn the optimal policy in an environment. Its goal is to learn a policy so that appropriate actions can be taken in different states, and it is widely used to solve control and resource allocation problems in wireless communication. Therefore, in the algorithm design part, the present invention introduces the Q-learning algorithm to achieve the reasonable allocation of the spreading factor and transmission power of each terminal device. The core of Q-Learning is its Q-table, which is continuously updated according to the Bellman equation during the continuous exploration process to obtain the optimal Q-table, and the update method of the Q value is:

[0137]

[0138] Among them, γ is the discount factor, β is the learning rate, (s t , is the current state, a t is the current action, (s t+1 , is the next state, a t+1 is the next action, (r t+1 is the reward obtained after executing the action a t ), Q′(s t , a t ) is the target Q value, Q(s t , a t ) is the current Q value.

[0139] The design of the algorithm mainly includes: the design of actions, states, and rewards, and the state transition rules. The following elaborates on the design of the algorithm in detail.

[0140] First, design the actions of the algorithm. The purpose of the algorithm is to find reasonable spreading factors and transmission powers for each terminal device. Therefore, the action of the algorithm is defined as a i = (m, P tx ), where m is the spreading factor, P tx is the transmission power, is the set of transmission powers; among them, P i (i = 1, 2, 3, 4) are the transmission powers defined in the LoRaWAN protocol that can be selected by the terminal device, and P 0 indicates that the terminal device does not send data. The unit of the transmission power is dBm. When the terminal device does not send data, its action is (m, P 0 ).

[0141] Secondly, design the states of the algorithm. Let the least common multiple of the transmission cycles of all terminal devices be T, and the state n of all terminal devices is the current time slot. When all terminal devices run for T time slots, it is equivalent to completing one cycle and returning to the initial state.

[0142] Finally, design the rewards of the algorithm. Based on the transmission success probability of a single terminal device in a single time slot derived from formula (15), the transmission success probability of all terminal devices in the entire transmission process can be derived as:

[0143]

[0144] Since the LoRaWAN protocol is a low-power communication protocol, when the terminal device ensures the successful transmission of data packets, it should preferably select a lower transmission power to extend the maximum working time of the terminal device as much as possible. Therefore, the overall reward of the system is designed as:

[0145]

[0146] Among them, the parameter ζ is used to balance the transmission success probability and power consumption.

[0147] In summary, the specific implementation process of the reinforcement learning algorithm established by the present invention is as follows:

[0148] Input: Set of terminal devices Set R of distances from terminal devices to the gateway;

[0149] Output: Actions selected by terminal devices, namely SF and transmission power P tx .

[0150] Initialize the actions a of all terminal devices i = 0, initialize the Q values Q of all terminal devices i = 0;

[0151] Loop: Each terminal device selects the action with the largest Q value with a probability of ε Randomly select an action a with a probability of (1 - ε) i ;

[0152] The terminal device executes the action and obtains the reward r i (s, a i ) and the state s;

[0153] The terminal device updates its corresponding Q value;

[0154] n = n + 1;

[0155] Output the action selection of the terminal device.

[0156] The above is only the preferred embodiment of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the principle of the present invention, several improvements and refinements can be made, and these improvements and refinements should also be regarded as the protection scope of the present invention.

Claims

1. A LoRa spreading factor and power joint allocation method based on reinforcement learning, characterized in that: The method jointly considers the capture effect existing in LoRa and the incomplete orthogonality between different SF modulation signals, and includes the following steps: Step S1: Build a LoRa network according to different LoRa access scenarios; Step S2: designing a collision parameter to indicate whether the reference terminal device is interfered by the interfering terminal device in the nth time slot; Step S3: Model and analyze the signal-to-interference-to-noise ratio of the LoRa signal at the gateway, derive and calculate the signal-to-interference-to-noise ratio; combine the demodulation characteristics of LoRa, derive and calculate the probability of successful transmission, and quantify the impact of the spreading factor and the transmission power on the probability of successful transmission; Terminal device D i The probability of successful transmission in the nth time slot is: Among them, θ capture is the threshold value for the capture effect, r i is the distance from the terminal device to the gateway, α is the path loss index, is the noise power, For terminal device D i The transmission power, A(f c )=(f c 2 ×10 -2.8 ) -1 is the deterministic loss, f c is the carrier frequency, is the terminal device set, s jm Indicates interference terminal device D j Whether to select the spreading factor m, s jm =1 indicates interference with terminal device D j Select spreading factor m, s jm =0 indicates interference terminal device D j The spreading factor m is not selected; Indicates terminal device D i Interference with terminal equipment D j In the case of a collision at the nth time slot, To interfere with terminal equipment D j The transmission power, For terminal device D i The transmission power, r j To interfere with terminal equipment D j Distance to gateway, is the spreading factor set of the terminal equipment, To select the interfering terminal device D with spreading factor k j The transmit power used; parameter ω n Indicates whether there is same SF interference, ω n =1 indicates the presence of same SF interference, ω n =0 means there is no same-SF interference; According to formula (15), the final transmission success probability is only related to the SF selection of the terminal device, the transmission power, and the distance between the terminal device and the gateway; Step S4: Introduce reinforcement learning to jointly allocate the spreading factor and transmit power of LoRa to achieve access and optimization of LoRa; A reinforcement learning algorithm is established with the goal of maximizing the overall transmission success probability of the system; the reinforcement learning algorithm is used to assign the optimal spreading factor and transmission power combination to each terminal device.

2. The LoRa spreading factor and power joint allocation method based on reinforcement learning according to claim 1, characterized in that: The LoRa network includes I terminal devices, and the terminal device set is defined as Assume i Terminal device D i is the reference terminal device, and its transmission power is Then its received power is: Among them, A(f c )=(f c 2 ×10 -2.8 ) -1 is the deterministic loss, f c is the carrier frequency, r i is the distance from the terminal device to the gateway, α is the path loss index; h i For the i-th terminal device D i The channel gain to the gateway, considering the Rayleigh fading channel condition, Then |h i | 2 It follows an exponential distribution with parameter 1.

3. The LoRa spreading factor and power joint allocation method based on reinforcement learning according to claim 1, characterized in that: The LoRa network includes I terminal devices, and the terminal device set is defined as Assume i Terminal device D i The reference terminal equipment, its spreading factor selection is represented by parameter s ij To express: When the terminal device D i The spreading factor j is selected, and the parameter s ij is 1, otherwise the parameter s ij is 0.

4. The LoRa spreading factor and power joint allocation method based on reinforcement learning according to claim 1, characterized in that: The calculation formula of the collision parameter is as follows: in, To interfere with terminal equipment D j The sending cycle, n m' To interfere with terminal equipment D j The number of time slots occupied by sending data once, For reference terminal device D i When the condition of formula (3) is met, the reference terminal device D in the nth time slot i Send data and interfere with terminal device D j Transmission is in progress, data overlap occurs, resulting in a collision, and the collision parameter is 1.

5. The LoRa spreading factor and power joint allocation method based on reinforcement learning according to claim 1, characterized in that: The signal interference in LoRa is divided into intra-SF interference and inter-SF interference. The signal interference-to-noise ratio of the terminal device is calculated in the following two cases: (1) When there is same-SF interference and same-SF interference is dominant, select terminal device D with spreading factor m i Signal-to-interference-noise ratio in the nth time slot for: in, is the noise power, NF is the receiver noise figure, BW is the bandwidth, To interfere with terminal equipment D j The received power, s jm Indicates interference terminal device D j Whether to select the spreading factor m, s jm =1 indicates interference with terminal device D j Select spreading factor m, s jm =0 indicates interference terminal device D j The spreading factor m is not selected; considering the interference with the same spreading factor, that is, the spreading factor used by the reference terminal device and the interference terminal device is m, then Indicates terminal device D i Interference with terminal equipment D j Collision situation at the nth time slot; Dividing both the numerator and denominator in formula (4) by the noise power yields: in, For terminal device D i The signal-to-noise ratio of the transmitted signal at the gateway is denoted by γ im ;h i For the i-th terminal device D i Channel gain to gateway, |h i | 2 Obeying exponential distribution, at this time γ im The obedience parameters are The exponential distribution of , then formula (5) is further expressed as: When there is only interference within the SF, the probability of successful transmission is θ capture is the threshold value for the capture effect to occur; Substituting formula (6) into the expression is: The signal-to-noise ratio of the data packet sent by the terminal device at the gateway is γ im , then γ 11 ,...γ i-1m-1 ,γ i+1m+1 ,...γ IM represents the signal-to-noise ratio of all possible combinations except the selected terminal device; due to γ im Follows exponential distribution, for all γ im Marginalization gets: In formula (8), Denoted as A; due to γ im It obeys exponential distribution. According to the probability density function of exponential distribution, A is calculated as follows: Among them, λ im To obey the parameter λ, x is the formal parameter used for integration; Substituting A back into formula (8) yields: in, To interfere with terminal equipment D j The transmission power, For terminal device D i The transmission power, r j To interfere with terminal equipment D j Distance to the gateway; (2) When there is no intra-SF interference and only inter-SF interference, select terminal device D with spreading factor m i Signal-to-interference-noise ratio in the nth time slot It is expressed as: Among them, P im For terminal device D i The transmission power, s jk To interfere with terminal equipment D j The selection of spreading factor, s jk =1 indicates interference with terminal device D j Select the spreading factor k, s jk =0 indicates interference terminal device D j The spreading factor k is not selected, P is the terminal device D i Interference with terminal equipment D j The collision situation, P jk To interfere with terminal equipment D j The transmission power; Referring to the derivation process of formula (6), formula (11) is further expressed as: Among them, γ jk Denotes the interference terminal device D with a spreading factor k j signal-to-noise ratio at the gateway; At this time, the probability of successful transmission is θ m is the decoding SINR threshold corresponding to the spreading factor m; the mathematical expression of the transmission success probability obtained by similar derivation as in case (1) is: in, The transmission power of the interfering device for selecting the spreading factor k; Combining the above two situations, since it is necessary to determine whether there is intra-SF interference in the nth time slot, the parameter ω is set n To indicate whether there is same SF interference, it is defined as: According to formula (14), ω n Equal to 0 means that no interference device and the reference device select the same SF in the nth time slot. The judgment condition is: for SF=m, the sum of the collision parameters is 0; otherwise, it means that the same SF interference exists.

6. The LoRa spreading factor and power joint allocation method based on reinforcement learning according to claim 1, characterized in that: The reinforcement learning algorithm is implemented using Q-learning, which continuously updates the optimal Q table according to the Bellman equation during the continuous exploration process, where the Q value is updated as follows: Among them, γ is the discount factor, β is the learning rate, (s t is the current state, a t is the current action, (s t+1 , is the next state, a t+1 For the next action, r t+1 To perform action a t The reward obtained after t ,a t ) is the target Q value, Q(s t ,a t ) is the current Q value.

7. The LoRa spreading factor and power joint allocation method based on reinforcement learning according to claim 1, characterized in that: In the reinforcement learning algorithm, the action is defined as a i =(m,P tx ), m is the spreading factor, P tx is the transmission power; when the terminal device does not send data, its action is (m, P0).

8. The LoRa spreading factor and power joint allocation method based on reinforcement learning according to claim 1, characterized in that: In the reinforcement learning algorithm, the status of all terminal devices n is the current time slot, and T is the least common multiple of the transmission periods of all terminal devices.

9. The LoRa spreading factor and power joint allocation method based on reinforcement learning according to claim 1, characterized in that: In the reinforcement learning algorithm, the transmission success probability of all terminal devices in the entire transmission process is: Where T is the least common multiple of the transmission periods of all terminal devices. For terminal equipment collection, For terminal device D i The probability of successful transmission in the nth time slot.

10. The LoRa spreading factor and power joint allocation method based on reinforcement learning according to claim 1, characterized in that: In the reinforcement learning algorithm, the overall reward of the system is: Among them, the parameter ζ is used to balance the probability of successful transmission and power consumption, T is the least common multiple of the transmission period of all terminal devices, For terminal equipment collection, For terminal device D i The probability of successful transmission in the nth time slot, where n is the current time slot, For terminal device D i of transmission power.

Images