Spatial adaptive transmission method and system based on multi-objective reinforcement learning

Abstract

The present invention discloses a spatial adaptive transmission method and system based on multi-objective reinforcement learning, belonging to the field of space communication technology. The system is composed of a link state monitoring module, a performance evaluation module, a link state prediction module, a reinforcement learning agent module, a parameter adjustment module, a transmission execution module and a data storage module, wherein the reinforcement learning agent module is used as a core decision-making unit to output the optimal transmission parameter configuration according to the link state and performance evaluation results; the transmission method includes the steps of reinforcement learning model construction and training, real-time link state monitoring, future link state prediction, performance evaluation, reinforcement learning decision-making, dynamic parameter adjustment and transmission execution. The present invention realizes the intelligent adaptation of the DTN protocol in a complex space communication environment through multi-objective reinforcement learning, significantly improves the transmission efficiency, reliability and resource utilization, is applicable to various space communication scenarios, and provides a new solution for the design of future space communication systems.

Description

Technical Field

[0001] The present invention relates to the field of space communication technology, and in particular to a space adaptive transmission method and system based on multi-objective reinforcement learning. Background Art

[0002] With the rapid development of aerospace technology and the rise of commercial aerospace, space networks play an increasingly important role in global communications, deep space exploration, earth observation and other fields. This network not only includes geosynchronous orbit and low-orbit satellite communication systems, but also covers the communication needs of deep space exploration missions such as the moon and Mars. However, space networks face communication challenges that are completely different from ground networks. The traditional transmission control protocol (TCP) has poor transmission performance in space networks. The main reasons are: extremely long propagation delay (such as the round-trip delay between the earth and the moon is about 2.6 seconds, and the round-trip delay between the earth and Mars is 3 to 20 minutes), high bit error rate (due to factors such as cosmic rays and solar activity), intermittent connection (due to periodic obstruction caused by celestial movement), and limited bandwidth and energy resources. These factors cause the congestion control mechanism of TCP to fail to work properly, and frequent timeout retransmission and slow start processes seriously affect the transmission efficiency. In order to meet these challenges, the delay-tolerant network (DTN) protocol solution was proposed, and achieved better transmission performance than TCP in space communications. The core of the DTN protocol stack is the Bundle Protocol (BP) and the Licklider Transmission Protocol (LTP). BP allows data to be stored and transmitted for a long time in the network through a store-and-forward mechanism, effectively solving the problem of intermittent connections. LTP provides reliable point-to-point transmission services, and through segmented transmission and selective confirmation mechanisms, it greatly improves the transmission efficiency of DTN in long-delay, high-bit-error-rate links.

[0003] Previously, there have been many studies on DTN protocol parameter optimization and mechanism improvement in the field, aiming to further improve the transmission performance of DTN in space networks. For example, some researchers have explored the optimal size of BP data packets and the optimal number of LTP data blocks for different space link conditions to strike a balance between transmission efficiency and protocol overhead. Other scholars have proposed an efficient transmission method combining LTP protocol with packet-level forward erasure code (PL-FEC) to reduce file delivery delay. However, although these studies have achieved certain performance improvements in specific scenarios, the current DTN protocol still faces some challenges in practical applications. First, the key parameters of BP and LTP (such as bundle size, block size, segment size, retransmission timeout, etc.) are usually fixed, which is difficult to adapt to dynamically changing space link conditions. Second, the existing DTN implementation lacks the ability to perceive and predict the link status in real time, and cannot adjust the transmission strategy in advance to cope with the upcoming link changes. In addition, most of the existing optimization methods are aimed at a single parameter or a specific scenario, and lack comprehensive optimization of multiple transmission parameters. Therefore, how to realize the intelligent adaptive transmission of DTN protocol so that it can predict the future link status to dynamically adjust multiple key transmission parameters has become an important issue to be solved in the current space communication field. Summary of the invention

[0004] In order to overcome the shortcomings of the above-mentioned prior art, the purpose of the present invention is to provide a spatial adaptive transmission method and system based on multi-objective reinforcement learning. By introducing reinforcement learning (RL) technology, the active adaptive capability of the DTN protocol is realized, and the transmission parameters are optimized to better cope with dynamic link changes.

[0005] In order to achieve the above-mentioned purpose, the present invention adopts the following technical solutions:

[0006] The present invention first provides a spatial adaptive transmission system based on multi-objective reinforcement learning, including a link state monitoring module, a performance evaluation module, a link state prediction module, a reinforcement learning agent module, a parameter adjustment module, a transmission execution module and a data storage module, wherein:

[0007] The link status monitoring module is responsible for real-time monitoring of various status parameters of the space link and outputting them to the link status prediction module, including:

[0008] RTT estimation unit: when there is data transmission, it estimates the link round-trip delay based on the time difference between sending the LTP signaling segment and receiving the ACK; when there is no data transmission, it uses the packet-train algorithm (Packet-Train) to actively detect;

[0009] Bandwidth estimation unit: when there is data transmission, it estimates the link available bandwidth based on the time interval between data segment arrivals in each LTP transmission round; when there is no data transmission, it uses the packet queue algorithm to actively detect;

[0010] Packet loss rate and bit error rate estimation unit: when there is data transmission, the link packet loss rate and bit error rate are estimated according to the symbol loss rate of the LTP lower layer FEC matrix; when there is no data transmission, the packet queue algorithm is used for active detection.

[0011] The performance evaluation module is responsible for calculating and evaluating the transmission performance indicators of the system and outputting them to the reinforcement learning agent module, including:

[0012] Bundle delivery delay calculation unit: calculates the average bundle delivery delay at the current moment;

[0013] Effective throughput calculation unit: calculates the effective throughput at the current moment.

[0014] The link state prediction module predicts the link state of a future period of time steps based on the link monitoring data of the link state monitoring module in the previous period of time steps using a long short-term memory network (LSTM) and outputs it to the reinforcement learning agent module.

[0015] include:

[0016] Data preprocessing unit: used to normalize and extract features from the input monitoring data of the first X time steps;

[0017] LSTM prediction unit: realizes the link status prediction for the next B time steps.

[0018] The LSTM network structure is as follows:

[0019] Input layer: corresponding to the monitored link status parameters;

[0020] LSTM layer: contains multiple layers of LSTM units;

[0021] Fully connected layer: one or more fully connected layers;

[0022] Output layer: predict the link status for the next B time steps.

[0023] The reinforcement learning agent module is used to make the best decision based on the link status prediction results and transmission performance evaluation results, and output the optimal transmission parameter configuration to the parameter adjustment module. The deep Q network algorithm is used to implement reinforcement learning, including:

[0024] Deep Q Network: DQN for short, as the main network of reinforcement learning, its structure includes an input layer, multiple hidden layers and an output layer, which is used to approximate the expected cumulative reward for taking each possible action in a given state, referred to as Q value;

[0025] Target network: A copy of the DQN network. It has the same structure as the main network but has a lower parameter update frequency. It is used to calculate the target Q value and update the main network parameters by comparing it with the main network Q value to improve training stability.

[0026] Experience replay buffer: used to store and sample interactive experience;

[0027] Action selection unit: used to implement the ε-greedy strategy;

[0028] Model update unit: adjusts the main network parameters according to the feedback and periodically copies the main network parameters to the target network.

[0029] The parameter adjustment module is used to dynamically adjust the corresponding parameters of the DTN protocol stack according to the output of the reinforcement learning agent module. It includes:

[0030] BP parameter adjustment unit: adjust the bundle size;

[0031] LTP parameter adjustment unit: adjusts the block size, segment size and retransmission timer RTO;

[0032] FEC parameter adjustment unit: adjusts the FEC encoding bit rate.

[0033] The transfer execution module uses the adjusted parameters to perform the actual data transfer.

[0034] The data storage module is used to store historical link status data for offline training and online optimization of reinforcement learning models.

[0035] The present invention also provides a spatial adaptive transmission method based on multi-objective reinforcement learning, which runs on the above-mentioned spatial adaptive transmission system based on multi-objective reinforcement learning, and comprises:

[0036] S1. Build a reinforcement learning model: define the state space, action space, and reward function;

[0037] S1.1. Define the state space S, including but not limited to the following parameters:

[0038] Link round-trip time RTT;

[0039] Link available bandwidth BW;

[0040] Link packet loss rate P e ;

[0041] Bundle average delivery delay T avg ;

[0042] Effective throughput TH.

[0043] The state space can be expressed as:

[0044] S={P e ,RTT,BW,T avg ,TH} (1)

[0045] S1.2. Define the action space A, including but not limited to the following adjustable parameters:

[0046] BP Bundle Size L Bundle ;

[0047] LTP Block size L Block ;

[0048] LTP segment size L Segment ;

[0049] LTP signaling segment retransmission timer RTO;

[0050] FEC code rate R c .

[0051] The action space can be expressed as:

[0052] A={L Bundle ,L Block ,L Segment ,RTO,R c} (2)

[0053] S1.3. Define the reward function R, taking into account the following optimization objectives:

[0054] Transmission delay difference D T : The difference between the average delivery delay of the bundle and the one-way propagation delay of the current link;

[0055]

[0056] Link utilization U: the ratio of effective throughput to available bandwidth;

[0057]

[0058] Reliability Gap D R : The difference between the currently used FEC code rate and the theoretical optimal code rate.

[0059] D R =(1-P e )-R c (5)

[0060] The reward function is designed as follows:

[0061]

[0062] Among them, w1, w2, and w3 are weight coefficients, satisfying w1+w2+w3=1, and can be dynamically adjusted according to specific application scenarios.

[0063] S2. Training the reinforcement learning model: Use the spatial link history data stored in the data storage module to build a training environment, and use the DQN algorithm to train the reinforcement learning model built in S1;

[0064] S2.1. Use the space link history data stored in the data storage module to simulate a space link whose state parameters change over time as a training environment for the reinforcement learning model;

[0065] S2.2. Initialize the DQN network as the main network Q, whose parameter set is represented by θ; at the same time, create a copy of the main network as the target network Q′, whose parameter set is represented by θ′;

[0066] S2.3. Use the ε-greedy strategy to select actions and execute them, observing the rewards and the next state;

[0067] S2.4. Use Experience Replay technology to improve training efficiency and stability. Maintain an experience pool D. After each interaction with the environment, store the combination of state, action, reward and next state (s, a, r, s′) in D. During training, randomly extract a certain number of samples from D for learning.

[0068] S2.5. Perform offline training on the reinforcement learning model in the constructed training environment. The training steps are as follows:

[0069] Initialization state s1

[0070] fort=1toTdo:

[0071] Use the ε-greedy strategy to select action a t

[0072] Execute action a t , observe the reward r t and the new state s t+1

[0073] Will (s t ,a t ,r t ,s t+1 )Deposit into D

[0074] Randomly sample N samples from D

[0075] For each sample (s j ,a j ,r j ,sj+1 ), calculate the target Q value:

[0076] y j =r j +γ·max a′ Q′(s j+1 ,a′;θ′) (7)

[0077] S2.6. Use the mean square error loss function to perform gradient descent and update the parameter set θ of the main network Q:

[0078]

[0079] S2.7. Regularly update the target network parameters: θ′←θ;

[0080] S2.8. Repeat S2.3 to S2.7 until the reinforcement learning model converges or reaches the preset training rounds.

[0081] S3. Real-time link status monitoring: Use a dedicated link status monitoring module to continuously monitor the parameters of the current space link in the following ways:

[0082] S3.1. When data is being transmitted:

[0083] Estimate the current link round-trip delay RTT(t) based on the time difference between the LTP signaling segment sending time and the ACK receiving time;

[0084] Estimate the available bandwidth BW(t) based on the current data transmission situation;

[0085] Estimate the current link packet loss rate P based on the symbol loss rate of the LTP lower layer FEC matrix e (t) and bit error rate;

[0086] S3.2. When the link is idle and no data is being transmitted:

[0087] Use the packet queue algorithm to periodically and actively detect the link status, estimate the link round-trip delay RTT(t), available bandwidth BW(t) and packet loss rate P e (t) and bit error rate.

[0088] S4. Future link status prediction: Based on the link monitoring data of the previous period of time, the LSTM network is used to predict the link status of the next period of time. The prediction process is as follows:

[0089] S4.1. Input the link status monitoring data of the past X time steps;

[0090] S4.2.LSTM network outputs the predicted value for the next B time steps;

[0091] S4.3. When the link state of the next time step is actually observed, the prediction error is calculated and the LSTM model parameters are updated;

[0092] S4.4. Using the sliding window method, repeat steps S4.1 to S4.3.

[0093] S5. Performance evaluation: Calculate the average delivery delay and effective throughput of the current data packet;

[0094] Use a dedicated performance evaluation module to calculate the average delivery delay T of all bundles in the period before the current time t avg (t), and the effective throughput TH(t).

[0095] S6. Reinforcement learning makes the best decision: outputs the optimal transmission parameter configuration;

[0096] Taking the future link state predicted by LSTM and the evaluation results of the current performance evaluation module as input, the reinforcement learning agent outputs the optimal transmission parameter configuration.

[0097] S7. Dynamic parameter adjustment: According to the transmission parameter configuration output by the reinforcement learning agent module, the parameter adjustment module dynamically adjusts the corresponding parameters in the DTN protocol stack;

[0098] According to the transmission parameter configuration output by the reinforcement learning agent, dynamically adjust the corresponding parameters in the DTN protocol stack:

[0099] Adjust the bundle size L in BP Bundle ;

[0100] Adjust the Block size L in LTP Block ;

[0101] Adjust the segment size L in LTP Segment ;

[0102] Adjust the LTP signaling segment retransmission timer RTO;

[0103] Adjust FEC encoding rate R c .

[0104] S8. Transmission execution: The transmission execution module uses the adjusted parameters for data transmission;

[0105] Based on the adjusted parameters, the complete transmission process of data from the sender to the receiver through the space link is realized. On the sender, the application data is encapsulated into the Bundle data packet through BP, LTP is aggregated to form a data block Block, and segmented through FEC encoding, and finally transmitted through the User Datagram Protocol UDP; on the receiver, the underlying UDP delivers data to FEC for decoding, recovers the LTP segment, reconstructs the LTP Block, decompresses the aggregated Bundles, and takes out the application data to deliver to the upper layer application. When the link is idle, the UDP transmission detection packet queue is used to actively detect the link status.

[0106] S9. Repeat S3 to S8 to continuously optimize the transmission strategy. After each iteration, update the historical database for future link state prediction and offline training of the reinforcement learning model.

[0107] The spatial adaptive transmission method and system based on multi-objective reinforcement learning described in the present invention adopt the above technical solution compared with the prior art, and have the following beneficial effects:

[0108] (1) By introducing reinforcement learning technology, the system can actively predict link state changes and make corresponding adjustments compared to traditional fixed parameter configuration, significantly improving the adaptability of the DTN protocol in dynamic spatial links. At the same time, the system has the ability of continuous learning and self-optimization, and can continuously improve performance as the running time increases.

[0109] (2) The present invention takes into account multiple optimization objectives such as transmission delay, reliability and link utilization, and achieves a comprehensive improvement in transmission performance. Through the multi-objective reinforcement learning algorithm, the system can find the best balance between these conflicting objectives. For example, while ensuring reliability, the transmission delay is reduced as much as possible; while improving link utilization, the reliability of transmission is not excessively sacrificed. This multi-objective optimization method enables the system to maintain good performance in various complex space communication environments.

[0110] (3) The present invention has broad application prospects and excellent adaptability. It can be deployed on various nodes of the space communication network, including ground stations, relay satellites and deep space probes, and is suitable for a variety of space communication scenarios such as low-Earth orbit satellite communication, lunar communication, and deep space exploration. This distributed optimization method can not only better adapt to the complex space network topology and dynamically changing link conditions, but also quickly adapt to the performance requirements of different scenarios by adjusting the weight of the reward function. BRIEF DESCRIPTION OF THE DRAWINGS

[0111] Figure 1 This is a diagram of the architecture of the spatial adaptive transmission system based on multi-objective reinforcement learning of the present invention;

[0112] Figure 2The data flow and control flow chart of the spatial adaptive transmission method of the present invention;

[0113] Figure 3 This is a schematic diagram of the LSTM network structure in the link state prediction module of the present invention;

[0114] Figure 4 This is a schematic diagram of the DQN network structure in the reinforcement learning agent module of the present invention;

[0115] Figure 5 This is an example diagram of the dynamic parameter mapping and configuration process of the parameter adjustment module of the present invention. DETAILED DESCRIPTION

[0116] The present invention is further explained below in detail with reference to the accompanying drawings so that those skilled in the art can more deeply understand the present invention and be able to implement it. However, the following reference examples are only used to explain the present invention and are not intended to limit the present invention.

[0117] like Figure 1 As shown, the spatial adaptive transmission system based on multi-objective reinforcement learning of the present invention has a system architecture including the following main modules: link status monitoring module, performance evaluation module, link status prediction module, reinforcement learning agent module, parameter adjustment module, transmission execution module and data storage module.

[0118] like Figure 2 As shown, the data flow of the spatial adaptive transmission method based on multi-objective reinforcement learning of the present invention includes BP layer, LTP layer, FEC layer and UDP layer data processing and transmission, and the control process includes link status monitoring, performance evaluation, link status prediction, reinforcement learning decision and parameter adjustment.

[0119] like Figure 1 and Figure 2 As shown, the link status monitoring module is responsible for RTT estimation, available bandwidth estimation, packet loss rate and bit error rate estimation, and outputs the estimation results to the link status prediction module to predict the state of the link within the next RTT. Specifically:

[0120] When the link is busy, that is, when data is being transmitted, the actual transmission status fed back by LTP and lower-layer FEC is used to estimate the link round-trip delay, available bandwidth, and packet loss rate.

[0121] When the link is in an idle state, i.e. when there is no data transmission, the packet queue algorithm is used for active detection. The sender actively sends a string of length Y = 30 packets with a packet size L every 60 seconds. p=1KB data packet queue. The receiver responds with an ACK for each packet it receives, and records whether each packet in the packet queue has arrived successfully and the arrival time, and feeds back to the sender after the reception is completed. The sender estimates the link round-trip delay, available bandwidth and packet loss rate based on the received ACK and the final feedback information.

[0122] RTT estimation:

[0123] When there is data transmission, the last data segment of each LTP transmission round will be set as the signaling segment, and the sending time t s , when the corresponding ACK is received, record the receiving time t r , use the low-pass filtering algorithm to update the RTT estimate, where the update weight α = 0.1, that is:

[0124] RTT new =α·(t r -t s )+(1-α)·RTT old (1)

[0125] When there is no data transmission, use the packet team algorithm to actively detect and record the sending time t for each packet sent. s , when the corresponding ACK is received, record the receiving time t r , for n packets that successfully reach the receiver and receive the corresponding ACK, take the minimum value as the RTT estimate, that is:

[0126] RTT new =min{(t r,1 -t s,1 ),(t r,2 -t s,2 ),…,(t r,n -t s,n )} (2)

[0127] Available bandwidth estimate:

[0128] When there is data transmission, the link available bandwidth is estimated based on the time interval between data segment arrivals in each LTP transmission round, in bits per second (bps). The estimated available bandwidth sample value is multiplied by the growth factor δ=1.2, and the available bandwidth estimate is updated using a low-pass filtering algorithm, where the update weight α=0.2, that is:

[0129]

[0130] Where M is the number of data segments that successfully reach the receiving end in each transmission round, Δ m Defined as the time interval between the first and last data segments arriving at the receiver, the LTP segment size L segment: Optional range: {200B,400B,600B,800B,1000B,1200B,1400B}.

[0131] When there is no data transmission, the packet queue algorithm is used for active detection to estimate the link available bandwidth based on the arrival time interval, in bits per second (bps), that is:

[0132]

[0133] Where P is the number of packets in the detection packet sequence that successfully reach the receiving end, Δ p It is defined as the time interval between the first and last detection packets arriving at the receiving end, and the packet size is L p .

[0134] Packet loss rate and bit error rate estimation:

[0135] When data is transmitted, the FEC layer receiver under LTP will reply with feedback on the receiving status of each matrix. The sender calculates the matrix symbol loss rate p based on the feedback and uses a low-pass filtering algorithm to update the current link packet loss rate estimate, where the update weight α = 0.1, that is:

[0136] P e,new =α*p+(1-α)*P e,old (5)

[0137] According to the current matrix symbol length L symbol Calculate the bit error rate:

[0138]

[0139] When there is no data transmission, the packet team algorithm is used for active detection. The current link packet loss rate is estimated based on the number of detection packets that arrive, that is:

[0140]

[0141] The bit error rate calculation is the same as above.

[0142] like Figure 1 and Figure 2 As shown, the performance evaluation module is responsible for calculating the average delivery delay and effective throughput, and outputs the calculation results to the reinforcement learning agent module. Specifically:

[0143] Calculate the average bundle delivery delay T avg (t):

[0144] For each successfully delivered Bundle, record its sending time t s and receiving time t r , calculate the delivery delay T = t r -ts . Calculate the average delivery delay of all bundles within the 3 RTTs before the current time t, taking into account the difference in bundle length:

[0145]

[0146] Where T i is the delivery delay of the ith Bundle, L i is the length of the i-th Bundle.

[0147] Calculate the effective throughput TH(t):

[0148] Count the total amount of data L successfully transmitted within 3 RTTs before the current time t data ,but:

[0149]

[0150] like Figure 1 and Figure 2 As shown in Figure 2, the link state prediction module is responsible for predicting the link state and outputting it to the reinforcement learning agent module to make the optimal decision. Figure 3 As shown in the figure, the link status prediction module uses the LSTM network for prediction. The structure includes an input layer, an LSTM layer, a fully connected layer, and an output layer. Specifically:

[0151] Input layer: 90 neurons, corresponding to the link state parameters (RTT, BW, P e );

[0152] LSTM layer: 2 layers, 256 units per layer;

[0153] Fully connected layer: 1 layer, 128 neurons, using ReLU activation function;

[0154] Output layer: 3 neurons, corresponding to the predicted link state parameters within 1 RTT in the future.

[0155] The sliding window method is used in the prediction process, and the window size is 30 time steps. After each prediction, the model is updated with the actual observed link status to maintain the accuracy of the prediction. Specifically:

[0156] a) Input contains the link state parameters within 30 RTTs before the current time t

[0157] b) LSTM network outputs the predicted value of the next RTT, RTT(t+1), BW(t+1), P e (t+1);

[0158] c) When the link status of the next time step is actually observed, the prediction error is calculated and the LSTM model parameters are updated;

[0159] d) Slide the prediction window and repeat steps a) to c).

[0160] like Figure 1 and Figure 2 As shown in , the reinforcement learning agent module is responsible for making the best decision based on the predicted link status within the next RTT and the current transmission performance evaluation results and outputting it to the parameter adjustment module. Figure 4 As shown, the reinforcement learning agent uses a deep Q network (DQN) structure, including an input layer, a hidden layer, and an output layer. In a specific embodiment, a reinforcement learning model is first constructed:

[0161] Define the state space S, including the following parameter types:

[0162] Round Trip Time (RTT) (t+1): indicates the round trip time within the next RTT predicted at the current time t;

[0163] Available bandwidth BW(t+1): indicates the available bandwidth within the next RTT predicted at the current time t;

[0164] Link packet loss rate P e (t+1): represents the link packet loss rate within the next RTT predicted at the current time t;

[0165] Bundle average delivery delay T avg (t): represents the average bundle delivery delay at the current time t;

[0166] Effective throughput TH(t): represents the effective throughput at the current time t.

[0167] Define the action space A, and set the parameter range as follows:

[0168] BP Bundle Size L Bundle : Optional range {1KB,2KB,4KB,8KB,16KB,32KB};

[0169] LTP Block size L Block : Optional range {32KB,64KB,128KB,256KB,512KB};

[0170] LTP segment size L Segment : Optional range {200B,400B,600B,800B,1000B,1200B,1400B};

[0171] LTP signaling segment retransmission timer RTO: optional range {1.1RTT, 1.2RTT, 1.3RTT};

[0172] FEC code rate R c : Optional range {0.66,0.70,0.74,0.78,0.82,0.86,0.90}.

[0173] These parameter types and ranges are set based on typical space communication scenarios and can be adjusted according to specific application requirements.

[0174] Define the reward function R, and a typical setting of the weight coefficients is: w1 = 0.4, w2 = 0.3, w3 = 0.3. This setting can better balance the three goals of transmission delay, link utilization and reliability in most cases. However, in different application scenarios, these weights may need to be adjusted. For example, in deep space exploration missions, reliability may be given more attention, so the value of w3 can be appropriately increased.

[0175] After the reinforcement learning model is built, it is trained using the DQN network. During the training process, the specific parameter settings are listed in Table 1:

[0176] Table 1 Reinforcement learning training parameters

[0177]

[0178]

[0179] These parameter settings work well in experiments, but can be fine-tuned according to specific application scenarios and performance requirements.

[0180] For each training round (episode from 1 to 300), perform the following steps:

[0181] 1. Get the initial state s1, set the current step number t = 1, and initialize the total reward of the current round to 0;

[0182] 2. In each round, when t is not greater than 300 and the current state is not the terminal state, perform the following steps:

[0183] 2.1. Generate a random number. If the random number is less than ε, randomly select action a t ; Otherwise, choose to make Q(s t ,a) The largest action a t ; Execute action a t , observe the reward r t and the new state s t+1 ; Update the total reward of the current round plus r t ;Experience(s t,a t ,r t ,s t+1 ) is stored in the experience playback buffer D;

[0184] 2.2. If the number of samples in the experience replay buffer D is not less than 256, then 256 samples are randomly sampled from D. For each sample j, if s j+1 is the terminal state, then

[0185] y j =r j (10)

[0186] otherwise

[0187] y j =r j +γ·max a′ Q′(s j+1 ,a′;θ′) (11)

[0188] Use the mean square error loss function for gradient descent to update the parameter set θ of the main network Q:

[0189]

[0190] Update the parameters of Q using gradient descent with a learning rate α;

[0191] 2.3. If the current step number t is a multiple of 100, update the target network parameters: θ′←θ;

[0192] 2.4. Update exploration rate ε = max(ε-2×10 -4 ,0.01); Update status and number of steps: s t =s t+1 , t = t + 1;

[0193] 2.5. Repeat steps 2.1 to 2.4 until the maximum number of steps is reached or a termination state is encountered;

[0194] 3. Record the total reward and number of steps in this round; if the current round number is a multiple of 5, evaluate the performance of the current strategy, and end the training early if the performance target is reached;

[0195] 4. Repeat steps 2 to 3 until the reinforcement learning model converges or reaches the preset training round;

[0196] 5. Save the trained model and apply it to actual transmission scenarios.

[0197] like Figure 1 and 2As shown, the parameter adjustment module is responsible for mapping the optimal decision output by the reinforcement learning agent module to the corresponding actual protocol parameter value and adjusting the parameters of BP, LTP and FEC. Figure 5 A specific parameter adjustment example is shown, in which the reinforcement learning agent outputs the current optimal decision action A = {1, 3, 2, 0, 5}, where each number corresponds to a specific parameter adjustment decision. The parameter adjustment module converts these decisions into specific parameter values ​​according to the predefined mapping relationship.

[0198] Specifically:

[0199] BP Bundle Size L Bundle : Action value 1 corresponds to a 2KB Bundle size;

[0200] LTP Block size L Block : Action value 3 corresponds to a block size of 256KB;

[0201] LTP Segment Size L Segment : Action value 2 corresponds to a segment size of 600 bytes;

[0202] LTP retransmission timeout RTO: The action value 0 corresponds to a retransmission timeout of 1.1 times the RTT;

[0203] FEC code rate R c : An action value of 5 corresponds to a coding bit rate of 0.86.

[0204] Configure each layer of the DTN protocol according to specific parameter values: set the BP layer Bundle size to 2KB, the LTP layer Block size to 256KB, the Segment size to 600 bytes, the retransmission timeout to 1.1 times the RTT, and the FEC layer encoding bit rate to 0.86.

[0205] like Figure 1 and Figure 2 As shown, the transmission execution module is responsible for the data processing and transmission process from the sender to the receiver, specifically:

[0206] When the link is busy, at the sender, the BP layer first generates data packets Bundles, which contain application data that needs to be transmitted. Then, the LTP layer aggregates these Bundles into larger data blocks Block, and further divides the data blocks into LTP data segments Segments. Next, the FEC layer performs forward error correction encoding on these data segments and generates a coding matrix to improve the reliability of transmission. The encoded data is placed in the UDP send queue buffer, waiting to be sent through the space link. After passing through the space link that may have long delays and high bit error rates, the data arrives at the receiver's UDP receive queue buffer. At the receiver, the data is first decoded at the FEC layer to restore the original LTP data segments, then the data blocks are reassembled at the LTP layer, and finally the original Bundles are extracted at the BP layer.

[0207] When the link is in an idle state, the transmission execution module will use the detection packet queue algorithm to actively detect the current link status and deliver it to the receiver unreliably through the UDP protocol in a best-effort manner.

[0208] During the whole process, the receiver will provide the data reception status to the sender through the feedback mechanism of each layer for reliable transmission control, link status estimation and transmission performance evaluation. The transmission execution module will perform data transmission according to the adjusted parameters. For example, if the adjusted LTP Block size is 256KB, the module will aggregate the bundles to be transmitted into blocks of 256KB, and then perform segmentation and FEC encoding transmission.

[0209] like Figure 1 As shown in Figure 1, the data storage module uses a time series database to store historical link state data. These data will be used for offline training and online updating of the reinforcement learning model.

[0210] Through the above detailed implementation, the spatial adaptive transmission method and system based on multi-objective reinforcement learning of the present invention realizes the intelligent adaptation of the DTN protocol in a complex and changeable space communication environment through real-time monitoring, prediction, reinforcement learning decision-making and dynamic adjustment. This method not only improves transmission efficiency and reliability, but also optimizes resource utilization, providing new ideas and solutions for the design of future space communication systems.

[0211] The specific implementation scheme described above further describes in detail the purpose, technical scheme and beneficial effects of the present invention. It should be understood that the above is only a specific implementation scheme of the present invention and is not intended to limit the scope of the present invention. Any equivalent changes and modifications made by any technician in the field without departing from the concept and principle of the present invention should fall within the scope of protection of the present invention.

Claims

1. A spatial adaptive transmission method based on multi-objective reinforcement learning, which is applied to a spatial adaptive transmission system based on multi-objective reinforcement learning, characterized in that: The spatial adaptive transmission system based on multi-objective reinforcement learning constructs and trains a reinforcement learning model according to the optimization target of a specific application scenario, and applies it to the actual transmission process; the system controls the link state monitoring module and the link state prediction module to monitor in real time and predict the spatial link state accordingly, while the performance evaluation module evaluates the transmission performance in real time, the reinforcement learning agent module comprehensively considers the predicted link state and the evaluated transmission performance, makes the best decision, and controls the parameter adjustment module to dynamically adjust the protocol parameters in real time, thereby realizing the spatial adaptive transmission process including the construction of the reinforcement learning model, the training of the reinforcement learning model, the real-time link state monitoring and prediction, the reinforcement learning to make the best decision, the dynamic parameter adjustment and the transmission execution, which specifically includes: S1. Build a reinforcement learning model: define the state space, action space, and reward function; S2. Training the reinforcement learning model: Use the spatial link history data stored in the data storage module to build a training environment, and use the DQN algorithm to train the reinforcement learning model built in S1; S3. Real-time link status monitoring: Continuously monitor various status parameters of the current space link; S4. Future link status prediction: Based on the link monitoring data of the previous period of time, the LSTM network is used to predict the link status of the next period of time. The prediction process is as follows: S4.

1. Input the link status monitoring data of the past X time steps; S4.2.LSTM network outputs the predicted value for the next B time steps; S4.

3. When the link state of the next time step is actually observed, the prediction error is calculated and the LSTM model parameters are updated; S4.

4. Using the sliding window method, repeat steps S4.1 to S4.3; S5. Performance evaluation: Calculate the average delivery delay and effective throughput of the current data packet; S6. Reinforcement learning makes the best decision: outputs the optimal transmission parameter configuration; S7. Dynamic parameter adjustment: According to the transmission parameter configuration output by the reinforcement learning agent module, the parameter adjustment module dynamically adjusts the corresponding parameters in the DTN protocol stack; S8. Transmission execution: The transmission execution module uses the adjusted parameters for data transmission; S9. Repeat S3 to S8 and store the space link status parameters monitored at each moment into the data storage module; The specific implementation steps of constructing the reinforcement learning model in step S1 include: S1.

1. Define the state space, including the predicted link round-trip delay RTT, available bandwidth BW and packet loss rate P e , the average delivery delay of the evaluated bundle is T avg and effective throughput TH; S1.

2. Define the action space, including the BP Bundle size L Bundle 、LTP Block size L Block , LTP segment size L Segment , LTP signaling segment retransmission timer RTO and FEC code rate R c ; S1.

3. Define the reward function, comprehensively consider the transmission delay gap, link utilization and reliability gap, and design different weights accordingly; Transmission delay difference D T : The difference between the average delivery delay of the bundle and the one-way propagation delay of the current link; Link utilization U: the ratio of effective throughput to available bandwidth; Reliability Gap D R : The gap between the currently used FEC code rate and the theoretical optimal code rate; D R =(1-P e )-R c The reward function is designed as follows: Where w1, w2, w3 are weight coefficients, satisfying w1+w2+w3=1; The specific implementation steps of training the reinforcement learning model in step S2 include: S2.

1. Use the space link history data stored in the data storage module to simulate a space link whose state parameters change over time as a training environment for the reinforcement learning model; S2.

2. Initialize the DQN network as the main network Q, whose parameter set is represented by θ; at the same time, create a copy of the main network as the target network Q′, whose parameter set is represented by θ′; S2.

3. Use the ε-greedy strategy to select actions and execute them, observing the rewards and the next state; S2.

4. Use experience replay technology to maintain an experience pool D. After each interaction with the environment, store the combination of current state s, action a, reward r and next state s′ (s, a, r, s′) in D; S2.

5. Randomly extract a certain number of samples from D for learning, and for each sampled sample (s j ,a j ,r j ,s j+1 ), calculate the target Q value: y j =r j +γ·max a′ Q′(s j+1 ,a′;θ′) Among them, y j represents the target Q value, γ represents the discount factor, and a′ represents the next state s j+1 The possible actions below, max a′ Q′(s j+1 ,a′;θ′) means in the next state s j+1 In the above equation, select the possible action a′ that maximizes the Q value; S2.

6. Use the mean square error loss function to perform gradient descent and update the parameter set θ of the main network Q: Among them, L represents the mean square error loss, N represents the number of samples taken from the experience replay pool in one parameter update, j represents the index of the sample, and Q(s j ,a j ; θ) represents the Q value predicted by the main network based on the current parameter set θ, that is, in state s j Take action a j expected benefits; S2.

7. Regularly update the target network parameters: θ′←θ; S2.

8. Repeat S2.3 to S2.7 until the reinforcement learning model converges or reaches the preset training rounds; The real-time link status monitoring described in step S3 continuously monitors various status parameters of the current space link, including: S3.

1. When data is being transmitted, the round-trip delay of the current link is estimated based on the time difference between sending the LTP signaling segment and receiving the ACK, the available bandwidth of the current link is estimated based on the time interval between the arrival of the data segments in each LTP transmission round, and the packet loss rate and bit error rate of the current link are estimated based on the symbol loss rate of the FEC matrix under the LTP layer; S3.

2. When the link is idle and no data is being transmitted, the packet queue algorithm is used to actively detect the link status; The dynamic parameter adjustment described in step S7 maps the optimal decision output by the reinforcement learning agent to each protocol parameter value, and immediately configures the protocol stack transmission parameters accordingly, including BP Bundle size, LTP Block size, LTP segment size, LTP signaling segment retransmission timer and FEC code rate; The transmission execution described in step S8 implements the complete transmission process of data from the sender through the space link to the receiver. At the sender, the application data is encapsulated into the Bundle data packet through BP, LTP is aggregated to form a data block Block, and segmented through FEC encoding, and finally transmitted through UDP; at the receiver, the underlying UDP delivers data to FEC for decoding, restores the LTP segment, reconstructs the LTP Block, decompresses the aggregated Bundles, takes out the application data and delivers it to the upper-layer application. When the link is idle, the UDP transmission detection packet queue is used to actively detect the link status.

2. A spatial adaptive transmission system based on multi-objective reinforcement learning, the system is equipped with the spatial adaptive transmission method based on multi-objective reinforcement learning according to claim 1, characterized in that: The system includes a link status monitoring module, a performance evaluation module, a link status prediction module, a reinforcement learning agent module, a parameter adjustment module, a transmission execution module and a data storage module, wherein: The link status monitoring module is used to monitor various status parameters of the space link in real time; The link state prediction module is used to predict various state parameters of the future space link according to various state parameters output by the link state monitoring module; The performance evaluation module is used to calculate and evaluate the transmission performance indicators of the system; The reinforcement learning agent module is used to make the best decision and output the optimal transmission parameter configuration according to the link state prediction result output by the link state prediction module and the transmission performance evaluation result output by the performance evaluation module; The parameter adjustment module is used to dynamically adjust the corresponding parameters of the DTN protocol stack according to the output of the reinforcement learning agent module; The transmission execution module is used to perform the actual data transmission using the adjusted parameters; The data storage module is used to store historical link status.

3. A spatial adaptive transmission system based on multi-objective reinforcement learning according to claim 2, characterized in that: The link status monitoring module monitors various status parameters of the space link in real time, including link round-trip delay, available bandwidth and packet loss rate; The link status prediction module uses a long short-term memory network, LSTM for short, to predict the link status of a period of time in the future based on the link monitoring data of the link status monitoring module in the previous period of time; The performance evaluation module calculates and evaluates the transmission performance indicators of the system, including the average delivery delay and effective throughput of data packets.

4. The spatial adaptive transmission system based on multi-objective reinforcement learning according to claim 2 is characterized in that: The reinforcement learning agent module uses a deep Q network algorithm to implement reinforcement learning, including: Deep Q Network: DQN for short, as the main network of reinforcement learning, its structure includes an input layer, multiple hidden layers and an output layer, which is used to approximate the expected cumulative reward for taking each possible action in a given state, referred to as Q value; Target network: A copy of the DQN network. It has the same structure as the main network but has a lower parameter update frequency. It is used to calculate the target Q value. By comparing the calculated target Q value with the Q value of the main network, the main network parameters are updated to improve training stability. Experience replay buffer: used to store and sample interactive experience; Action selection unit: used to implement the ε-greedy strategy; Model update unit: adjusts the main network parameters according to the feedback and periodically copies the main network parameters to the target network.

5. The spatial adaptive transmission system based on multi-objective reinforcement learning according to claim 2 is characterized in that: The parameter adjustment module comprises: BP parameter adjustment unit: used to adjust the size of the data packet Bundle; LTP parameter adjustment unit: used to adjust the data block size, data segment size and retransmission timer RTO; FEC parameter adjustment unit: used to adjust the FEC encoding bit rate.

Images