Method for optimizing degree distribution of cache-constrained lt codes based on reinforcement learning

Abstract

The application provides a cache-restricted LT code degree distribution optimization method based on reinforcement learning, comprising the following steps: S1, generating a source symbol sequence with a length of k; S2, constructing a reinforcement learning agent for determining a degree distribution of a transmitting end, the reinforcement learning agent is composed of a 2-node input layer, a 2k-node hidden layer and a k-node output layer; S3, updating a current degree value through the reinforcement learning agent, and performing LT coding on the degree value and the source symbol by an encoder to obtain a coded symbol and send the coded symbol to a receiving end decoder; S4, judging the degree value of the received coded symbol according to the currently recovered source symbol by the decoder, and performing a cache or decoding according to the degree value; S5, training the reinforcement learning agent by using the reinforcement learning environment in step S3, and finally obtaining a degree distribution that makes the reward tend to be maximum through iterative training. The application can obtain a degree distribution with smaller redundancy overhead required for full recovery under the condition of cache restriction.

Description

Technical Field

[0001] This invention belongs to the field of channel coding technology, specifically relating to a method for optimizing cache-constrained LT code degree distribution based on reinforcement learning. Background Technology

[0002] Fountain codes differ from traditional fixed-rate coding methods. They are a type of rateless code, meaning the code rate can adaptively change with the channel state without prior acquisition of channel state information. Therefore, they are well-suited for high-capacity data communication scenarios such as broadcasting, satellite communication, and data distribution. At the transmitting end, k source symbols can generate an infinite number of coded symbols through this coding process. The receiving end only needs to receive any n coded symbols to achieve a high probability of successfully recovering the source symbols, where n is slightly greater than k, and n = k × overhead = k(1 + ε), where ε is called redundancy overhead.

[0003] LT codes, as a fountain code encoding method with linear encoding and decoding complexity, rely heavily on their degree distribution, as the success of decoding largely depends on it. A fundamental requirement for a good degree distribution is that the rate at which source symbols add ripples should match the processing speed, similar to the balance between soliton scattering and refraction. The Ideal Soliton Distribution (ISD) corresponds to the scenario where, during iterative decoding after receiving a certain number of encoded symbols, only one new encoded symbol with a degree of 1 appears and is added to the ripple in each iteration. It is clear that ISD represents an ideal scenario, but in reality, this degree distribution is very fragile, and ripples easily disappear, making it impractical. However, its analysis and derivation laid the foundation for the Robust Soliton Distribution (RSD), which can guarantee that ripples do not disappear with a very high probability and is more commonly used in practical situations.

[0004] However, in practical applications of RSD, the probability of full recovery is small when the fixed redundancy overhead ε is small (i.e., the redundancy overhead required for full recovery is large). However, in real-world scenarios, the receiver's buffer is usually limited. How to optimize the degree distribution under buffer-limited conditions is a topic that has not yet been studied in the existing technology. Summary of the Invention

[0005] The purpose of this invention is to propose a cache-constrained LT code degree distribution optimization method based on reinforcement learning, which can obtain a degree distribution that reduces the redundancy overhead required for full recovery under cache-constrained conditions.

[0006] This invention is achieved through the following technical solution:

[0007] The reinforcement learning-based method for optimizing cache-constrained LT code-degree distribution includes the following steps:

[0008] Step S1: Generate a source symbol sequence of length k;

[0009] Step S2: Construct a reinforcement learning agent to determine the transmitter degree distribution. This reinforcement learning agent consists of a 2-node input layer, a 2k-node hidden layer, and a k-node output layer. The 2 nodes of the input layer correspond to a state space of size 2 {k, B}. s}, where B s This is the upper limit of the number of encoded symbols that the receiver buffer can hold;

[0010] Step S3: Model the transmitter encoder and receiver decoder as a reinforcement learning environment. Update the current degree value through the reinforcement learning agent. The encoder performs LT coding based on the degree value and the source symbol to obtain the coded symbol and sends the coded symbol to the receiver decoder.

[0011] Step S4: The decoder determines the degree value of the received encoded symbol based on the currently recovered source symbols. When the degree value is 0, return to step S3. When the degree value is neither 0 nor 1, add the encoded symbol to the buffer. When the degree value is 1, add the encoded symbol to the ripple. Perform BP decoding on the encoded symbols in the ripple until the ripple is empty. If all source symbols are recovered at this time, proceed to step S5; otherwise, proceed to step S3.

[0012] Step S5: Using the reinforcement learning environment in step S3, train the reinforcement learning agent using the REINFORCE with Baseline algorithm. Define the encoding and decoding process of each source symbol as 1 step, define the full recovery process of each source symbol as 1 round, and use the reciprocal of the cost of each full recovery as the reward. Iterate and train until the degree distribution that maximizes the reward is obtained.

[0013] Furthermore, in step S2, the k node of the output layer corresponds to an action space of size k {1,2,…,k}, which is the selectable range of the current degree value updated in step S3.

[0014] Furthermore, in step S4, the decoder performs XOR edge-removal processing on the received encoded symbol based on the currently recovered source symbol to reduce the degree value of the encoded symbol, and then judges the reduced degree value.

[0015] Furthermore, in step S4, when the degree value is neither 0 nor 1, the encoded symbol is added to the cache; however, when the cache is full, an encoded symbol in the cache is randomly replaced with the encoded symbol.

[0016] Furthermore, in step S5, the maximum number of steps for each round is set to 10k.

[0017] Furthermore, in step S5, the state value function in the REINFORCE with Baseline algorithm is constructed as a neural network. This neural network consists of a 2-node input layer, a 2k-node hidden layer, and a 1-node output layer. The 2 nodes of the input layer correspond to a state space of size 2 {k, B}. s The output layer has a node corresponding to the state value. The neural network backpropagates the error with the reinforcement learning agent and iterates through training to finally obtain a degree distribution that maximizes the reward.

[0018] The present invention has the following beneficial effects:

[0019] 1. This invention constructs a reinforcement learning agent that determines the degree distribution at the transmitter, thereby obtaining a degree distribution that minimizes the redundancy overhead required for full recovery under cache-constrained conditions. Compared to the encoding methods in the prior art, this invention can ensure performance without feedback even when the cache is limited, thus eliminating the resource consumption required for feedback. Attached Figure Description

[0020] The present invention will now be described in further detail with reference to the accompanying drawings.

[0021] Figure 1 This is a flowchart of the present invention.

[0022] Figure 2 This is a schematic diagram of the degree distribution obtained in Embodiment 1 of the present invention.

[0023] Figure 3 This is a comparison chart of the redundancy overhead of Embodiment 1 of the present invention with that of LT DRSD and OFC in the prior art. Detailed Implementation

[0024] Example 1:

[0025] Suppose there are two reconnaissance drones, A and B, each carrying a mobile terminal T. A and T B Now T A Need to T B Report target information, i.e., T A To T B Send a message of length k = 100. Considering the limited computing power of the drone, the encoding and decoding process should not consume too many resources, so a feedback-free LT fountain code is used. Furthermore, due to the limited information buffer space of the drone, a buffer capacity limit B is used. s =0.3k=30 LT fountain code.

[0026] In this embodiment, the reinforcement learning-based cache-constrained LT code-degree distribution optimization method includes the following steps:

[0027] Step S1: Generate a source symbol sequence of length k. This step is implemented using existing technology.

[0028] Step S2: Construct a reinforcement learning agent-Policy network to determine the transmitter degree distribution. This reinforcement learning agent consists of a 2-node input layer, a 2k-node hidden layer, and a k-node output layer. The 2 nodes of the input layer correspond to a state space of size 2 {k, B}. s}={100,30}, the k nodes of the output layer correspond to an action space of size k {1,2,…,k}, which is the range of possible values ​​for the current degree value updated in step S3. The sum of the probabilities of each element in this space is 1, representing the degree distribution used in the encoding and decoding process, where B s In this embodiment, k = 100, B is the upper limit of the number of encoded symbols that the receiver buffer can hold. s =0.3k=30;

[0029] Step S3: Model the transmitter encoder and receiver decoder as a reinforcement learning environment. Update the current degree value through the reinforcement learning agent. The encoder performs LT coding based on the degree value and the source symbol to obtain the coded symbol and sends the coded symbol to the receiver decoder.

[0030] Step S4: After receiving the encoded symbol, the decoder performs XOR edge-removal processing on the encoded symbol based on the currently recovered source symbol to reduce the degree value of the encoded symbol, thereby simplifying the decoding process.

[0031] The reduced degree value is judged as follows: when the degree value is 0, it means that the encoded symbol cannot be decoded, return to step S3, update the degree value again and encode to obtain a new encoded symbol; when the degree value is neither 0 nor 1, it means that the encoded symbol cannot be directly decoded yet, so the encoded symbol is added to the receiver's buffer, but when the buffer is full, a random encoded symbol in the buffer is replaced with the encoded symbol; when the degree value is 1, it means that the encoded symbol can be decoded, so the encoded symbol is added to the ripple, and BP decoding is performed on the encoded symbols in the ripple until the ripple is empty. If the source symbols are fully recovered at this time, proceed to step S5, otherwise, proceed to step S3; during the BP decoding of the encoded symbols in the ripple, the degree value of the encoded symbols in the buffer will also decrease as the encoded symbols in the ripple are decoded. When the degree value of an encoded symbol in the buffer decreases to 1, it is put into the ripple for BP decoding.

[0032] Step S5: Using the reinforcement learning environment from Step S3, train the reinforcement learning agent using the REINFORCE with Baseline algorithm. The encoding and decoding process of each source symbol is defined as one step, and the full recovery process of each source symbol is defined as one round. The maximum number of steps in each round is set to 10k, considered sufficient to reach the "full recovery" point. The number of training rounds is set to n_episodes, the learning rate is set to α, and the number of training samples per round is set to batch_size. Here, it is assumed that each encoded symbol has an equal impact on the result, so the discount factor γ = 1 is set. The reciprocal of the overhead of each full recovery is used as the reward. Where k is the number of source symbols, n is the number of coded symbols, and ε is the redundancy overhead;

[0033] In the REINFORCE with Baseline algorithm, the state-value function is constructed as a neural network, with MSE as the loss function. The Policy network uses Cross Entropy as its loss function. This neural network consists of a 2-node input layer, a 2k-node hidden layer, and a 1-node output layer. The two nodes of the input layer correspond to a state space of size 2 {k, B}. s}={100,30}, where 1 node in the output layer corresponds to the state value. This neural network performs backpropagation of error with the reinforcement learning agent, iteratively training to finally obtain the degree distribution that maximizes the reward. The degree distribution obtained by the trained agent is then deployed to the transmitter. The degree distribution diagram λ(k) is shown below. Figure 2 As shown.

[0034] Figure 3 This is a comparison of the redundancy overhead of the present invention with that of the LT code with reduced ripple size and degree distribution (LT DRSD) and the online fountain code (OFC) in the prior art. The horizontal axis represents the ratio B of the receiver buffer capacity to the number of source symbols. s / k, with the ordinate representing the overhead O. It can be seen that when k = 100, ∈ = 0 (∈ represents the erasure probability of the binary erasure channel), this invention achieves the following in any B... s The redundancy overhead of full recovery is significantly reduced compared to LT DRSD. Specifically, the average receiver overhead using the degree distribution obtained in this invention is 1.544, while the average receiver overhead using LT DRSD (c1 = 1.7, c2 = 2.5) is 2.601; at k = 100, ∈ = 0, B s In the case of ≤0.3k, the redundancy overhead of the full recovery of this invention is also reduced compared to OFC, in B sWhile slightly higher than OFC at >0.3k, the difference is small. Specifically, the average receiver overhead using the degree distribution obtained by this invention is 1.544, while the average receiver overhead using OFC is 1.568. This means that the performance of this invention is similar to or even partially comparable to OFC. s This invention is better suited for certain scenarios, and OFC requires an average of 11.7 feedbacks, while this invention requires no feedback, saving the resource consumption required for feedback.

[0035] The above description is merely a preferred embodiment of the present invention and should not be construed as limiting the scope of the present invention. All equivalent changes and modifications made in accordance with the scope of the patent application and the contents of the specification of the present invention should still fall within the scope of the patent of the present invention.

Claims

1. A method for optimizing cache-constrained LT code-degree distribution based on reinforcement learning, characterized in that, Includes the following steps: Step S1: Generate a length of The source symbol sequence; Step S2: Construct a reinforcement learning agent for determining the emitter degree distribution. This reinforcement learning agent consists of a 2-node input layer and a... Node hidden layer and one The output layer consists of nodes, and the two nodes in the input layer correspond to a state space of size 2. ,in, This is the upper limit of the number of encoded symbols that the receiver buffer can hold; Step S3: Model the transmitter encoder and receiver decoder as a reinforcement learning environment. Update the current degree value through the reinforcement learning agent. The encoder performs LT coding based on the degree value and the source symbol to obtain the coded symbol and sends the coded symbol to the receiver decoder. Step S4: The decoder determines the degree value of the received encoded symbol based on the currently recovered source symbols. When the degree value is 0, return to step S3. When the degree value is neither 0 nor 1, add the encoded symbol to the buffer. When the degree value is 1, add the encoded symbol to the ripple. Perform BP decoding on the encoded symbols in the ripple until the ripple is empty. If all source symbols are recovered at this time, proceed to step S5; otherwise, proceed to step S3. Step S5: Using the reinforcement learning environment from Step S3, train the reinforcement learning agent using the REINFORCE with Baseline algorithm. Define the encoding and decoding process of each source symbol as one step, define the full recovery process of each source symbol as one round, and define the maximum number of steps in each round as... The number of training rounds is set to The learning rate is set as The number of training samples each time is set to The cost of each full recovery The reciprocal as a reward The inverse of the cost of each full recovery is used as the reward. Iterative training eventually yields a degree distribution that maximizes the reward. The number of source symbols, For the number of encoded symbols, This is for redundant overhead; Output layer The size of the node is Action space The action space is the selectable range of the current degree value updated in step S3; In step S4, the decoder performs XOR edge-removal processing on the received encoded symbol based on the currently recovered source symbol to reduce the degree value of the encoded symbol, and then judges the reduced degree value. When the degree value is neither 0 nor 1, the encoded symbol is added to the cache. However, when the cache is full, an encoded symbol in the cache is randomly replaced with the encoded symbol. In step S5, the maximum number of steps for each round is set to... ; In step S5, the state value function in the REINFORCE with Baseline algorithm is constructed as a neural network, which consists of a 2-node input layer and a... The input layer consists of a hidden layer with one node and an output layer with one node. The two nodes in the input layer correspond to a state space of size 2. The output layer has a node corresponding to the state value. The neural network backpropagates the error with the reinforcement learning agent and iterates through training to finally obtain a degree distribution that maximizes the reward.

Images