Spatial adaptive transmission method and system based on multi-target reinforcement learning

By introducing multi-objective reinforcement learning technology, the spatial link status is monitored and predicted in real time, and the DTN protocol parameters are dynamically adjusted, which solves the problem of insufficient adaptability of the DTN protocol in space communication and achieves a comprehensive improvement in transmission performance.

CN118971946AActive Publication Date: 2024-11-15NANJING UNIV

Patent Information

Application Number
CN202411399670.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-10-09
Publication Date
2024-11-15
Estimated Expiration
2044-10-09

AI Technical Summary

Technical Problem

Existing DTN protocols are ill-suited to adapting to dynamically changing link conditions in space communication, lacking real-time sensing and prediction capabilities, resulting in low transmission efficiency.

Method used

An adaptive transmission method based on multi-objective reinforcement learning is adopted. By monitoring, predicting and adjusting the link state, reinforcement learning technology is used to optimize the transmission parameters of the DTN protocol, thereby achieving proactive adaptation to the future link state.

Benefits of technology

It significantly improves the adaptability of the DTN protocol in dynamic space links, achieves a comprehensive improvement in transmission performance, and can maintain good performance in complex space communication environments.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118971946A_ABST
    Figure CN118971946A_ABST
Patent Text Reader

Abstract

The invention discloses a space adaptive transmission method and system based on multi-target reinforcement learning, and belongs to the technical field of space communication. The system is composed of a link state monitoring module, a performance evaluation module, a link state prediction module, a reinforcement learning agent module, a parameter adjustment module, a transmission execution module and a data storage module, and the reinforcement learning agent module serves as a core decision-making unit and outputs optimal transmission parameter configuration according to the link state and the performance evaluation result; the transmission method comprises the steps of reinforcement learning model construction and training, real-time link state monitoring, future link state prediction, performance evaluation, reinforcement learning decision making, dynamic parameter adjustment, transmission execution and the like. According to the method, intelligent self-adaption of the DTN protocol in a complex space communication environment is realized through multi-target reinforcement learning, the transmission efficiency, the reliability and the resource utilization rate are remarkably improved, the method is suitable for various space communication scenes, and a new solution is provided for future space communication system design.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of space communication technology, and in particular to a space adaptive transmission method and system based on multi-objective reinforcement learning. Background Art

[0002] With the rapid development of aerospace technology and the rise of commercial aerospace, space networks play an increasingly important role in global communications, deep space exploration, earth observation and other fields. This network not only includes geosynchronous orbit and low-orbit satellite communication systems, but also covers the communication needs of deep space exploration missions such as the moon and Mars. However, space networks face communication challenges that are completely different from ground networks. The traditional transmission control protocol (TCP) has poor transmission performance in space networks. The main reasons are: extremely long propagation delays (such as a round-trip delay of about 2.6 seconds between the earth and the moon, and a round-trip delay of 3 to 20 minutes between the earth and Mars), high bit error rates (due to factors such as cosmic rays and solar activity), intermittent connections (due to periodic obstructions caused by celestial movement), and limited bandwidth and energy resources. These factors cause the TCP congestion control mechanism to fail to work properly, and frequent timeout retransmissions and slow start processes seriously affect transmission efficiency. In order to meet these challenges, the delay-tolerant network (DTN) protocol solution was proposed and achieved better transmission performance than TCP in space communications. The core of the DTN protocol stack is the Bundle Protocol (BP) and the Licklider Transmission Protocol (LTP). BP allows data to be stored and transmitted for a long time in the network through a store-and-forward mechanism, effectively solving the problem of intermittent connection. LTP provides reliable point-to-point transmission services, and greatly improves the transmission efficiency of DTN in long-delay, high-bit-error-rate links through segmented transmission and selective confirmation mechanisms.

[0003] Previously, there have been many studies on DTN protocol parameter optimization and mechanism improvement in the field, aiming to further improve the transmission performance of DTN in space networks. For example, some researchers have explored the optimal size of BP data packets and the optimal number of LTP data blocks for different space link conditions to strike a balance between transmission efficiency and protocol overhead. Other scholars have proposed an efficient transmission method combining the LTP protocol with packet-level forward erasure code (PL-FEC) to reduce file delivery delay. However, although these studies have achieved certain performance improvements in specific scenarios, the current DTN protocol still faces some challenges in practical applications. First, the key parameters of BP and LTP (such as bundle size, block size, segment size, retransmission timeout, etc.) are usually fixed, which is difficult to adapt to dynamically changing space link conditions. Second, the existing DTN implementation lacks the ability to perceive and predict the link status in real time, and cannot adjust the transmission strategy in advance to cope with the upcoming link changes. In addition, most of the existing optimization methods are aimed at a single parameter or a specific scenario, and lack comprehensive optimization of multiple transmission parameters. Therefore, how to realize the intelligent adaptive transmission of DTN protocol so that it can predict the future link status to dynamically adjust multiple key transmission parameters has become an important issue to be solved in the current space communication field. Summary of the invention

[0004] In order to overcome the shortcomings of the above-mentioned prior art, the purpose of the present invention is to provide a spatial adaptive transmission method and system based on multi-objective reinforcement learning. By introducing reinforcement learning (RL) technology, the active adaptive capability of the DTN protocol is realized, and the transmission parameters are optimized to better cope with dynamic link changes.

[0005] In order to achieve the above-mentioned purpose, the present invention adopts the following technical solutions: The present invention first provides a spatial adaptive transmission system based on multi-objective reinforcement learning, including a link state monitoring module, a performance evaluation module, a link state prediction module, a reinforcement learning agent module, a parameter adjustment module, a transmission execution module and a data storage module, wherein: The link status monitoring module is responsible for real-time monitoring of various status parameters of the space link and outputting them to the link status prediction module, including: RTT estimation unit: when there is data transmission, it estimates the link round-trip delay based on the time difference between sending the LTP signaling segment and receiving the ACK; when there is no data transmission, it uses the packet-train algorithm (Packet-Train) to actively detect; Bandwidth estimation unit: when there is data transmission, it estimates the link available bandwidth based on the time interval between data segment arrivals in each LTP transmission round; when there is no data transmission, it uses the packet queue algorithm to actively detect; Packet loss rate and bit error rate estimation unit: when there is data transmission, the link packet loss rate and bit error rate are estimated according to the symbol loss rate of the LTP lower layer FEC matrix; when there is no data transmission, the packet queue algorithm is used for active detection.

[0006] The performance evaluation module is responsible for calculating and evaluating the transmission performance indicators of the system and outputting them to the reinforcement learning agent module, including: Bundle delivery delay calculation unit: calculates the average bundle delivery delay at the current moment; Effective throughput calculation unit: calculates the effective throughput at the current moment.

[0007] The link status prediction module uses the long short-term memory network (LSTM) to predict the link status of the next period of time according to the link monitoring data of the previous period of time of the link status monitoring module, and outputs it to the reinforcement learning agent module. It includes: Data preprocessing unit: used to normalize and extract features from the input monitoring data of the first M time steps; LSTM prediction unit: realizes the link status prediction for the next N time steps.

[0008] The LSTM network structure is as follows: Input layer: corresponding to the monitored link status parameters; LSTM layer: contains multiple layers of LSTM units; Fully connected layer: one or more fully connected layers; Output layer: predicts the link status for the next N time steps.

[0009] The reinforcement learning agent module is used to make the best decision based on the link status prediction results and transmission performance evaluation results, and output the optimal transmission parameter configuration to the parameter adjustment module. The deep Q network algorithm is used to implement reinforcement learning, including: Deep Q Network: DQN for short, as the main network of reinforcement learning, its structure includes an input layer, multiple hidden layers and an output layer, which is used to approximate the expected cumulative reward for taking each possible action in a given state, referred to as Q value; Target network: A copy of the DQN network. It has the same structure as the main network but has a lower parameter update frequency. It is used to calculate the target Q value and update the main network parameters by comparing it with the main network Q value to improve training stability. Experience replay buffer: used to store and sample interactive experience; Action selection unit: used to implement -greedy strategy; Model update unit: adjusts the main network parameters according to the feedback and periodically copies the main network parameters to the target network.

[0010] The parameter adjustment module is used to dynamically adjust the corresponding parameters of the DTN protocol stack according to the output of the reinforcement learning agent module. It includes: BP parameter adjustment unit: adjust the bundle size; LTP parameter adjustment unit: adjusts the block size, segment size and retransmission timer RTO; FEC parameter adjustment unit: adjusts the FEC encoding bit rate.

[0011] The transfer execution module uses the adjusted parameters to perform the actual data transfer.

[0012] The data storage module is used to store historical link status data for offline training and online optimization of reinforcement learning models.

[0013] The present invention also provides a spatial adaptive transmission method based on multi-objective reinforcement learning, which runs on the above-mentioned spatial adaptive transmission system based on multi-objective reinforcement learning, and comprises: S1. Build a reinforcement learning model: define the state space, action space, and reward function; S1.1. Defining the state space , including but not limited to the following parameters: Link round trip delay ; Link available bandwidth ; Link packet loss rate ; Average Bundle Delivery Latency ; Effective throughput .

[0014] The state space can be expressed as: .

[0015] S1.2. Defining the action space , including but not limited to the following adjustable parameters: BP Bundle Size ; LTP Block Size ; LTP segment size ; LTP signaling segment retransmission timer ; FEC code rate .

[0016] The action space can be expressed as: .

[0017] S1.3. Define the reward function , taking into account the following optimization objectives: Transmission delay gap : The difference between the average delivery delay of the bundle and the one-way propagation delay of the current link; , Link Utilization : The ratio of effective throughput to available bandwidth; , Reliability gap : The difference between the currently used FEC code rate and the theoretical optimal code rate is expressed as follows: .

[0018] The reward function is designed as follows: , in , , is the weight coefficient, satisfying , which can be dynamically adjusted according to specific application scenarios.

[0019] S2. Training the reinforcement learning model: Use the spatial link history data stored in the data storage module to build a training environment, and use the DQN algorithm to train the reinforcement learning model built in S1; S2.1. Use the space link history data stored in the data storage module to simulate a space link whose state parameters change over time as a training environment for the reinforcement learning model; S2.2. Initialize the DQN network as the main network , whose parameter set is Indicates; at the same time, create a copy of the main network as the target network , whose parameter set is express; S2.3. Use -The greedy strategy selects an action and executes it, observing the reward and the next state.

[0020] S2.4. Use Experience Replay technology to improve training efficiency and stability. Maintain an experience pool D. After each interaction with the environment, the combination of state, action, reward and next state is Store it in D, and randomly extract a certain number of samples from D for learning during training.

[0021] S2.5. Perform offline training on the reinforcement learning model in the constructed training environment. The training steps are as follows: Initialization state ; for = 1 to do: use -greedy strategy selection action ; Execute an action , observe the reward and the new state ; Will Deposit into D; Randomly sample a certain number of samples from D; For each sample , calculate the target Q value: .

[0022] S2.6. Use the mean square error loss function to perform gradient descent and update the parameter set of the main network Q : .

[0023] S2.7. Update target network parameters regularly: .

[0024] S2.8. Repeat S2.3 to S2.7 until the reinforcement learning model converges or reaches the preset training rounds.

[0025] S3. Real-time link status monitoring: Use a dedicated link status monitoring module to continuously monitor the parameters of the current space link in the following ways: S3.1. When data is being transmitted: Estimate the current link round-trip delay based on the time difference between the LTP signaling segment sending time and the ACK receiving time ; Estimate available bandwidth based on current data transfer ; Estimate the current link packet loss rate based on the LTP lower layer FEC matrix symbol loss rate and bit error rate.

[0026] S3.2. When the link is idle and no data is being transmitted: Use the packet queue algorithm to periodically and actively detect the link status and estimate the link round-trip delay , available bandwidth And packet loss rate and bit error rate.

[0027] S4. Future link status prediction: Based on the link monitoring data of the previous period of time, the LSTM network is used to predict the link status of the next period of time. The prediction process is as follows: S4.1. Input the link status monitoring data of the past M time steps; S4.2.LSTM network outputs the predicted value of the next N time steps; S4.3. When the link state of the next time step is actually observed, the prediction error is calculated and the LSTM model parameters are updated; S4.4. Using the sliding window method, repeat steps S4.1 to S4.3.

[0028] S5. Performance evaluation: Calculate the average delivery delay and effective throughput of the current data packet; Use a dedicated performance evaluation module to calculate the current moment Average delivery delay of all bundles in the previous period , and the effective throughput .

[0029] S6. Reinforcement learning makes the best decision: outputs the optimal transmission parameter configuration; Taking the future link state predicted by LSTM and the evaluation results of the current performance evaluation module as input, the reinforcement learning agent outputs the optimal transmission parameter configuration.

[0030] S7. Dynamic parameter adjustment: According to the transmission parameter configuration output by the reinforcement learning agent module, the parameter adjustment module dynamically adjusts the corresponding parameters in the DTN protocol stack; According to the transmission parameter configuration output by the reinforcement learning agent, dynamically adjust the corresponding parameters in the DTN protocol stack: Adjust the Bundle size in BP ; Adjust the Block size in LTP ; Adjust segment size in LTP ; Adjust the LTP signaling segment retransmission timer ; Adjust FEC encoding bitrate .

[0031] S8. Transmission execution: The transmission execution module uses the adjusted parameters for data transmission; Based on the adjusted parameters, the complete transmission process of data from the sender to the receiver through the space link is realized. On the sender, the application data is encapsulated into the Bundle data packet through BP, LTP is aggregated to form a data block Block, and segmented through FEC encoding, and finally transmitted through the User Datagram Protocol UDP; on the receiver, the underlying UDP delivers data to FEC for decoding, recovers the LTP segment, reconstructs the LTP Block, decompresses the aggregated Bundles, and takes out the application data to deliver to the upper layer application. When the link is idle, the UDP transmission detection packet queue is used to actively detect the link status.

[0032] S9. Repeat S3 to S8 to continuously optimize the transmission strategy. After each iteration, update the historical database for future link state prediction and offline training of the reinforcement learning model.

[0033] The spatial adaptive transmission method and system based on multi-objective reinforcement learning described in the present invention adopt the above technical solution compared with the prior art, and have the following beneficial effects: (1) By introducing reinforcement learning technology, the system can actively predict link status changes and make corresponding adjustments compared to traditional fixed parameter configuration, significantly improving the adaptability of the DTN protocol in dynamic spatial links. At the same time, the system has the ability of continuous learning and self-optimization, and can continuously improve performance as the running time increases.

[0034] (2) The present invention takes into account multiple optimization objectives such as transmission delay, reliability and link utilization, and achieves a comprehensive improvement in transmission performance. Through the multi-objective reinforcement learning algorithm, the system can find the best balance between these conflicting objectives. For example, while ensuring reliability, the transmission delay is reduced as much as possible; while improving link utilization, the reliability of transmission is not excessively sacrificed. This multi-objective optimization method enables the system to maintain good performance in various complex space communication environments.

[0035] (3) The present invention has broad application prospects and excellent adaptability. It can be deployed on various nodes of the space communication network, including ground stations, relay satellites and deep space probes, and is suitable for a variety of space communication scenarios such as low-Earth orbit satellite communications, lunar communications, and deep space exploration. This distributed optimization method can not only better adapt to complex space network topologies and dynamically changing link conditions, but also quickly adapt to the performance requirements of different scenarios by adjusting the weight of the reward function. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 This is a diagram of the architecture of the spatial adaptive transmission system based on multi-objective reinforcement learning of the present invention; Figure 2 The data flow and control flow chart of the spatial adaptive transmission method of the present invention; Figure 3 This is a schematic diagram of the LSTM network structure in the link state prediction module of the present invention; Figure 4 This is a schematic diagram of the DQN network structure in the reinforcement learning agent module of the present invention; Figure 5 This is an example diagram of the dynamic parameter mapping and configuration process of the parameter adjustment module of the present invention. DETAILED DESCRIPTION

[0037] The present invention is further explained below in detail with reference to the accompanying drawings so that those skilled in the art can more deeply understand the present invention and be able to implement it. However, the following reference examples are only used to explain the present invention and are not intended to limit the present invention.

[0038] like Figure 1 As shown, the spatial adaptive transmission system based on multi-objective reinforcement learning of the present invention has a system architecture including the following main modules: link status monitoring module, performance evaluation module, link status prediction module, reinforcement learning agent module, parameter adjustment module, transmission execution module and data storage module.

[0039] like Figure 2 As shown, the data flow of the spatial adaptive transmission method based on multi-objective reinforcement learning of the present invention includes BP layer, LTP layer, FEC layer and UDP layer data processing and transmission, and the control process includes link status monitoring, performance evaluation, link status prediction, reinforcement learning decision and parameter adjustment.

[0040] like Figure 1 and Figure 2 As shown, the link status monitoring module is responsible for RTT estimation, available bandwidth estimation, packet loss rate and bit error rate estimation, and outputs the estimation results to the link status prediction module to predict the state of the link within the next RTT. Specifically: When the link is busy, that is, when data is being transmitted, the actual transmission status fed back by LTP and lower-layer FEC is used to estimate the link round-trip delay, available bandwidth, and packet loss rate.

[0041] When the link is idle, i.e. when there is no data transmission, the packet queue algorithm is used for active detection. The sender actively sends a string of length =30, packet size =1KB data packet queue. The receiver responds with an ACK for each packet it receives, and records whether each packet in the packet queue has arrived successfully and the arrival time, and feeds back to the sender after the reception is completed. The sender estimates the link round-trip delay, available bandwidth and packet loss rate based on the received ACK and the final feedback information.

[0042] RTT estimation: When there is data transmission, the last data segment of each LTP transmission round will be set as the signaling segment to record the sending time. , when the corresponding ACK is received, record the receiving time , use the low-pass filtering algorithm to update the RTT estimate, where the weight is updated =0.1, that is: , When there is no data transmission, use the packet team algorithm to actively detect and record the sending time for each packet sent. , when the corresponding ACK is received, record the receiving time ,right The minimum value of the packets that successfully reach the receiver and receive the corresponding ACK is taken as the RTT estimate, that is: ; Available bandwidth estimate: When data is being transmitted, the link available bandwidth is estimated based on the time interval between data segment arrivals in each LTP transmission round, in bits per second (bps). The estimated available bandwidth sample value is multiplied by the growth factor = 1.2, and use a low-pass filtering algorithm to update the available bandwidth estimate, where the weights are updated =0.2, that is: ; in is the number of data segments that successfully reach the receiving end in each transmission round, Defined as the time interval between the first and last data segments arriving at the receiver.

[0043] When there is no data transmission, the packet queue algorithm is used for active detection, and the available link bandwidth is estimated based on the arrival time interval, in bits per second (bps), that is: , in is the number of packets in the detection packet sequence that successfully reach the receiving end, Defined as the time interval between the first and last probe packets arriving at the receiver.

[0044] Packet loss rate and bit error rate estimation: When data is transmitted, the FEC layer receiver under LTP will reply with feedback on the receiving status of each matrix, and the sender will calculate the matrix symbol loss rate based on the feedback. , use the low-pass filtering algorithm to update the current link packet loss rate estimate, where the weight is updated =0.1, that is: ; According to the current matrix symbol length Calculate the bit error rate: ; When there is no data transmission, the packet team algorithm is used for active detection. The current link packet loss rate is estimated based on the number of detection packets that arrive, that is: , The bit error rate calculation is the same as above.

[0045] like Figure 1 and Figure 2 As shown, the performance evaluation module is responsible for calculating the average delivery delay and effective throughput, and outputs the calculation results to the reinforcement learning agent module. Specifically: Calculate the average bundle delivery delay : For each successfully delivered Bundle, record the time it was sent and receiving time , calculate the delivery delay . Statistics of the current time Average delivery latency of all bundles within the first 3 RTTs, taking into account differences in bundle length: , in For the The delivery delay of each bundle, For the The length of the Bundle.

[0046] Calculating Effective Throughput : Statistics of current time The total amount of data successfully transmitted in the previous 3 RTTs ,but: .

[0047] like Figure 1 and Figure 2 As shown in Figure 2, the link state prediction module is responsible for predicting the link state and outputting it to the reinforcement learning agent module to make the optimal decision. Figure 3 As shown in the figure, the link status prediction module uses the LSTM network for prediction. The structure includes an input layer, an LSTM layer, a fully connected layer, and an output layer. Specifically: Input layer: 90 neurons, corresponding to the link state parameters within 30 RTTs before the current time ( ); LSTM layer: 2 layers, 256 units per layer; Fully connected layer: 1 layer, 128 neurons, using ReLU activation function; Output layer: 3 neurons, corresponding to the predicted link state parameters within 1 RTT in the future.

[0048] The sliding window method is used in the prediction process, and the window size is 30 time steps. After each prediction, the model is updated with the actual observed link status to maintain the accuracy of the prediction. Specifically: a) Input contains the current time Link status parameters within the first 30 RTTs ; b) The LSTM network outputs the predicted value of the next RTT ; c) When the link status of the next time step is actually observed, the prediction error is calculated and the LSTM model parameters are updated; d) Slide the prediction window and repeat steps a) to c).

[0049] like Figure 1 and Figure 2 As shown in , the reinforcement learning agent module is responsible for making the best decision based on the predicted link status within the next RTT and the current transmission performance evaluation results and outputting it to the parameter adjustment module. Figure 4 As shown, the reinforcement learning agent uses a deep Q network (DQN) structure, including an input layer, a hidden layer, and an output layer. In a specific embodiment, a reinforcement learning model is first constructed: Defining the state space , including the following parameter types: Round Trip Delay : Indicates the current time The predicted round trip delay within the next RTT; Available bandwidth : Indicates the current time The predicted available bandwidth within the next RTT; Link packet loss rate : Indicates the current time The predicted packet loss rate of the link within the next RTT; Average Bundle Delivery Latency : Indicates the current time Average bundle delivery latency; Effective throughput : Indicates the current time effective throughput.

[0050] Defining the action space , the parameter range is set as follows: BP Bundle Size : Optional range {1KB, 2KB, 4KB, 8KB, 16KB, 32KB}; LTP Block Size : Optional range {32KB, 64KB, 128KB, 256KB, 512KB}; LTP segment size : Optional range: {200B, 400B, 600B, 800B, 1000B, 1200B,1400B}; LTP signaling segment retransmission timer : Optional range {1.1RTT, 1.2RTT, 1.3RTT}; FEC code rate : The optional range is {0.66, 0.70, 0.74, 0.78, 0.82, 0.86, 0.90}.

[0051] These parameter types and ranges are set based on typical space communication scenarios and can be adjusted according to specific application requirements.

[0052] Defining the reward function , a typical setting for the weight coefficient is: = 0.4, = 0.3, = 0.3. This setting can better balance the three goals of transmission delay, link utilization and reliability in most cases. However, in different application scenarios, these weights may need to be adjusted. For example, in deep space exploration missions, reliability may be given more attention, so The value of .

[0053] After the reinforcement learning model is built, it is trained using the DQN network. During the training process, the specific parameter settings are listed in Table 1: Table 1 Reinforcement learning training parameters

[0054] These parameter settings work well in experiments, but can be fine-tuned according to specific application scenarios and performance requirements.

[0055] For each training round (episode from 1 to 300), perform the following steps: 1. Get the initial state , set the current step number =1, initialize the total reward of the current round to 0; 2. In each round, when If it is not greater than 300 and the current state is not the terminal state, perform the following steps: 2.1. Generate a random number. If the random number is less than , then randomly select an action Otherwise, select The biggest move ; Execute action , observe the reward and the new state ; Update the total reward of the current round plus ; Experience Store in experience replay buffer D; 2.2. If the number of samples in the experience playback buffer D is not less than 256, then randomly sample 256 samples from D. For each sample j, if is a terminal state, then: ; otherwise: ; Use the mean square error loss function for gradient descent to update the parameter set of the main network Q : ; Using learning rate The gradient descent update Parameters; 2.3. If the current step number If it is a multiple of 100, update the target network parameters: ; 2.4. Update exploration rate ; Update status and steps: , ; 2.5. Repeat steps 2.1 to 2.4 until the maximum number of steps is reached or a termination state is encountered; 3. Record the total reward and number of steps in this round; if the current round number is a multiple of 5, evaluate the performance of the current strategy, and end the training early if the performance target is reached; 4. Repeat steps 2 to 3 until the reinforcement learning model converges or reaches the preset training round; 5. Save the trained model and apply it to actual transmission scenarios.

[0056] like Figure 1 and 2 As shown, the parameter adjustment module is responsible for mapping the optimal decision output by the reinforcement learning agent module to the corresponding actual protocol parameter value and adjusting the parameters of BP, LTP and FEC. Figure 5Shows a specific example of parameter adjustment, in which the reinforcement learning agent outputs the current optimal decision action ={1, 3, 2, 0, 5}, each number corresponds to a specific parameter adjustment decision. The parameter adjustment module converts these decisions into specific parameter values ​​according to the predefined mapping relationship. Specifically: BP Bundle Size : Action value 1 corresponds to a 2KB Bundle size; LTP Block Size : Action value 3 corresponds to a block size of 256KB; LTP Segment Size : Action value 2 corresponds to a segment size of 600 bytes; LTP retransmission timeout : Action value 0 corresponds to a retransmission timeout of 1.1 times the RTT; FEC code rate : An action value of 5 corresponds to a coding bit rate of 0.86.

[0057] Configure each layer of the DTN protocol according to specific parameter values: set the BP layer Bundle size to 2KB, the LTP layer Block size to 256KB, the Segment size to 600 bytes, the retransmission timeout to 1.1 times the RTT, and the FEC layer encoding bit rate to 0.86.

[0058] like Figure 1 and Figure 2 As shown, the transmission execution module is responsible for the data processing and transmission process from the sender to the receiver, specifically: When the link is busy, at the sender, the BP layer first generates data packets Bundles, which contain application data that needs to be transmitted. Then, the LTP layer aggregates these Bundles into larger data blocks Block, and further divides the data blocks into LTP data segments Segments. Next, the FEC layer performs forward error correction encoding on these data segments and generates a coding matrix to improve the reliability of transmission. The encoded data is placed in the UDP send queue buffer, waiting to be sent through the space link. After passing through the space link that may have long delays and high bit error rates, the data arrives at the receiver's UDP receive queue buffer. At the receiver, the data is first decoded at the FEC layer to restore the original LTP data segments, then the data blocks are reassembled at the LTP layer, and finally the original Bundles are extracted at the BP layer.

[0059] When the link is in an idle state, the transmission execution module will use the detection packet queue algorithm to actively detect the current link status and deliver it to the receiver unreliably through the UDP protocol in a best-effort manner.

[0060] During the whole process, the receiver will provide the data reception status to the sender through the feedback mechanism of each layer for reliable transmission control, link status estimation and transmission performance evaluation. The transmission execution module will perform data transmission according to the adjusted parameters. For example, if the adjusted LTP Block size is 256KB, the module will aggregate the bundles to be transmitted into blocks of 256KB, and then perform segmentation and FEC encoding transmission.

[0061] like Figure 1 As shown in Figure 1, the data storage module uses a time series database to store historical link state data. These data will be used for offline training and online updating of the reinforcement learning model.

[0062] Through the above detailed implementation, the spatial adaptive transmission method and system based on multi-objective reinforcement learning of the present invention realizes the intelligent adaptation of the DTN protocol in a complex and changeable space communication environment through real-time monitoring, prediction, reinforcement learning decision-making and dynamic adjustment. This method not only improves transmission efficiency and reliability, but also optimizes resource utilization, providing new ideas and solutions for the design of future space communication systems.

[0063] The specific implementation scheme described above further describes in detail the purpose, technical scheme and beneficial effects of the present invention. It should be understood that the above is only a specific implementation scheme of the present invention and is not intended to limit the scope of the present invention. Any equivalent changes and modifications made by any technician in the field without departing from the concept and principle of the present invention should fall within the scope of protection of the present invention.

Claims

1. A spatial adaptive transmission system based on multi-objective reinforcement learning, characterized in that: The system includes a link status monitoring module, a performance evaluation module, a link status prediction module, a reinforcement learning agent module, a parameter adjustment module, a transmission execution module and a data storage module, wherein: The link status monitoring module is used to monitor various status parameters of the space link in real time; The link state prediction module is used to predict various state parameters of the future space link according to various state parameters output by the link state monitoring module; The performance evaluation module is used to calculate and evaluate the transmission performance indicators of the system; The reinforcement learning agent module is used to make the best decision and output the optimal transmission parameter configuration according to the link state prediction result output by the link state prediction module and the transmission performance evaluation result output by the performance evaluation module; The parameter adjustment module is used to dynamically adjust the corresponding parameters of the DTN protocol stack according to the output of the reinforcement learning agent module; The transmission execution module is used to perform the actual data transmission using the adjusted parameters; The data storage module is used to store historical link status.

2. According to claim 1, a spatial adaptive transmission system based on multi-objective reinforcement learning is characterized in that: The link status monitoring module monitors various status parameters of the space link in real time, including link round-trip delay, available bandwidth and packet loss rate; The link status prediction module uses a long short-term memory network, LSTM for short, to predict the link status of a period of time in the future based on the link monitoring data of the link status monitoring module in the previous period of time; The performance evaluation module calculates and evaluates the transmission performance indicators of the system, including the average delivery delay and effective throughput of data packets.

3. The spatial adaptive transmission system based on multi-objective reinforcement learning according to claim 1, characterized in that: The reinforcement learning agent module uses a deep Q network algorithm to implement reinforcement learning, including: Deep Q Network: DQN for short, as the main network of reinforcement learning, its structure includes an input layer, multiple hidden layers and an output layer, which is used to approximate the expected cumulative reward for taking each possible action in a given state, referred to as Q value; Target network: A copy of the DQN network. It has the same structure as the main network but has a lower parameter update frequency. It is used to calculate the target Q value. By comparing the calculated target Q value with the Q value of the main network, the main network parameters are updated to improve training stability. Experience replay buffer: used to store and sample interactive experience; Action selection unit: used to implement -greedy strategy; Model update unit: adjusts the main network parameters according to the feedback and periodically copies the main network parameters to the target network.

4. The spatial adaptive transmission system based on multi-objective reinforcement learning according to claim 1, characterized in that: The parameter adjustment module comprises: BP parameter adjustment unit: used to adjust the size of the data packet Bundle; LTP parameter adjustment unit: used to adjust the data block size, data segment size and retransmission timer RTO; FEC parameter adjustment unit: used to adjust the FEC encoding bit rate.

5. A spatial adaptive transmission method based on multi-objective reinforcement learning, the method running on the spatial adaptive transmission system based on multi-objective reinforcement learning according to any one of claims 1 to 4, characterized in that: The spatial adaptive transmission system based on multi-objective reinforcement learning constructs and trains a reinforcement learning model according to the optimization target of a specific application scenario, and applies it to the actual transmission process; the system controls the link state monitoring module and the link state prediction module to monitor in real time and predict the spatial link state accordingly, while the performance evaluation module evaluates the transmission performance in real time, the reinforcement learning agent module comprehensively considers the predicted link state and the evaluated transmission performance, makes the best decision, and controls the parameter adjustment module to dynamically adjust the protocol parameters in real time, thereby realizing the spatial adaptive transmission process including the construction of the reinforcement learning model, the training of the reinforcement learning model, the real-time link state monitoring and prediction, the reinforcement learning to make the best decision, the dynamic parameter adjustment and the transmission execution, which specifically includes: S1. Build a reinforcement learning model: define the state space, action space, and reward function; S2. Training the reinforcement learning model: Use the spatial link history data stored in the data storage module to build a training environment, and use the DQN algorithm to train the reinforcement learning model built in S1; S3. Real-time link status monitoring: Continuously monitor various status parameters of the current space link; S4. Future link status prediction: Based on the link monitoring data of the previous period of time, the LSTM network is used to predict the link status of the next period of time; S5. Performance evaluation: Calculate the average delivery delay and effective throughput of the current data packet; S6. Reinforcement learning makes the best decision: outputs the optimal transmission parameter configuration; S7. Dynamic parameter adjustment: According to the transmission parameter configuration output by the reinforcement learning agent module, the parameter adjustment module dynamically adjusts the corresponding parameters in the DTN protocol stack; S8. Transmission execution: The transmission execution module uses the adjusted parameters for data transmission; S9. Repeat S3 to S8, and store the space link status parameters monitored at each moment into the data storage module.

6. The spatial adaptive transmission method based on multi-objective reinforcement learning according to claim 5, characterized in that: The specific implementation steps of constructing the reinforcement learning model in step S1 include: S1.

1. Define the state space, including the predicted link round-trip delay, available bandwidth and packet loss rate, and the estimated average bundle delivery delay and effective throughput; S1.

2. Define the action space, including BP Bundle size, LTP Block size, LTP segment size, LTP signaling segment retransmission timer and FEC code rate; S1.

3. Define the reward function, comprehensively consider the transmission delay gap, link utilization and reliability gap, and design different weights accordingly.

7. The spatial adaptive transmission method based on multi-objective reinforcement learning according to claim 5 is characterized in that: The specific implementation steps of training the reinforcement learning model in step S2 include: S2.

1. Use the space link history data stored in the data storage module to simulate a space link whose state parameters change over time as a training environment for the reinforcement learning model; S2.

2. Initialize the DQN network as the main network , whose parameter set is Indicates; at the same time, create a copy of the main network as the target network , whose parameter set is express; S2.

3. Use -greedy strategy selects actions and executes them, observing rewards and next states; S2.

4. Use experience replay technology to maintain an experience pool D. After each interaction with the environment, the current state ,action ,award and the next state Combination of Deposit into D; S2.

5. Randomly extract a certain number of samples from D for learning, and for each sample , calculate the target value: , in, represents the target Q value, represents the discount factor, Indicates the next step status The possible actions below, Indicates the next state , select possible actions The value that maximizes the Q value; S2.

6. Use the mean square error loss function to perform gradient descent and update the main network The parameter set : , in, represents the mean square error loss, Indicates the number of samples taken from the experience replay pool in one parameter update, represents the index of the sample, Indicates that the main network is based on the current parameter set The predicted Q value, that is, in state Take action expected benefits; S2.

7. Update target network parameters regularly: ; S2.

8. Repeat S2.3 to S2.7 until the reinforcement learning model converges or reaches the preset training rounds.

8. The spatial adaptive transmission method based on multi-objective reinforcement learning according to claim 5 is characterized in that: The real-time link status monitoring described in step S3 continuously monitors various status parameters of the current space link, including: S3.

1. When data is being transmitted, the round-trip delay of the current link is estimated based on the time difference between sending the LTP signaling segment and receiving the ACK, the available bandwidth of the current link is estimated based on the time interval between the arrival of the data segments in each LTP transmission round, and the packet loss rate and bit error rate of the current link are estimated based on the symbol loss rate of the FEC matrix under the LTP layer; S3.

2. When the link is idle and no data is being transmitted, the packet-train algorithm (Packet-Train) is used to actively detect the link status.

9. The spatial adaptive transmission method based on multi-objective reinforcement learning according to claim 5, characterized in that: The dynamic parameter adjustment described in step S7 maps the optimal decision output by the reinforcement learning agent to each protocol parameter value, and immediately configures the protocol stack transmission parameters accordingly, including BP Bundle size, LTP Block size, LTP segment size, LTP signaling segment retransmission timer and FEC code rate.

10. The spatial adaptive transmission method based on multi-objective reinforcement learning according to claim 5, characterized in that: The transmission execution described in step S8 implements the complete transmission process of data from the sender through the space link to the receiver. At the sender, the application data is encapsulated into the Bundle data packet through BP, LTP is aggregated to form a data block Block, and segmented through FEC encoding, and finally transmitted through UDP; at the receiver, the underlying UDP delivers data to FEC for decoding, restores the LTP segment, reconstructs the LTP Block, decompresses the aggregated Bundles, takes out the application data and delivers it to the upper-layer application. When the link is idle, the UDP transmission detection packet queue is used to actively detect the link status.

Citation Information

Patent Citations

  • Power grid cross-region transmission communication system and method

    CN110022229A

  • Underwater acoustic communication adaptive modulation method and device based on deep reinforcement learning

    CN116996147A

  • Adjusting anomaly detection operations based on network resources

    US20170279685A1

  • Method and system for selecting a learning model from among a plurality of learning models

    US20210201209A1

  • Prediction system, information processing device, and information processing program

    WO2022270056A1

Cited By

  • Carrier rocket power supply data transmission and monitoring method and device and storage medium

    CN119341698A

  • State sensing method of data transmission link and electronic equipment

    CN120433866A

  • Method and electronic device for state awareness of a data transmission link

    CN120433866B

  • Monitoring data optimization method and system for ship fuel supply system

    CN121690343A